Ketos-metrics

This merge request adds ketos-metrics, a new command-line utility within the Ketos Commands suite designed to evaluate the performance of audio detectors and classifiers. It seamlessly integrates with the output of 'ketos-run'.

The utility operates on two essential CSV files:

1. Detection or Classification File: Contains the model's predictions.
2. Annotation or Ground Truth File: Provides the actual labels.

ketos-metrics offers two primary modes of operation, catering to various evaluation needs:

Short Clip Evaluation (clips)

This mode is tailored for evaluating shorter audio clips, assuming a one-to-one relationship between each clip and its corresponding annotation (snapshot models).

• detections (Positional, String): Path to the Ketos model file (with a *.kt extension). This is where the model detections are stored.
• annotations (Positional, String): Path to either a folder with audio files or an HDF5 database file containing the data to process.
• --output_folder (Optional, String): Specifies the location where the performance results will be output. The default is the current working directory.
• --add_background_annot (Optional, List of 2 elements): Creates background annotations for audio files based on existing annotations. You need to pass two parameters in this order: [path_to_audio_folder, label].
• --average_type (Optional, String): Specifies the type of averaging to perform when calculating metrics. Accepts either micro or macro as values. Default is macro.

Continuous File Evaluation (continuous)

This mode is intended for evaluating the output of continuous models.

• detections (Positional, String): Path to the Ketos model file (with a *.kt extension). This is where the model detections are stored.
• annotations (Positional, String): Path to either a folder with audio files or an HDF5 database file containing the data to process.
• --output_folder (Optional, String): Specifies the location where the performance results will be output. The default is the current working directory.
• --average_type (Optional, String): Specifies the type of averaging to perform when calculating metrics. Accepts either micro or macro as values. Default is macro.

Output Metrics

Depending on the choice of evaluation method, ketos-metrics generates a range of metrics:

• Confusion Matrix
• True Positives (TP)
• False Positives (FP)
• False Negatives (FN)
• True Negatives (TN)
• Precision
• Recall
• And more...

Note: Some metrics may not be calculated, depending on the evaluation option selected.

Examples and Testing Data

annotations_clips.csv

annotations_continuous.csv

classification.csv

detections.csv

Below you can find a couple of examples and data to test:

Clips:

python -m ketos.ketos_commands.ketos_metrics clips classification.csv annotations_clips.csv (make sure to assign proper paths)

Continuous:

python -m ketos.ketos_commands.ketos_metrics continuous detections.csv annotations_continuous.csv