The tasks covered in this directory / manuscript are defined as the following:
- Detection
- Go from Scene Image to Cropped image
- Validation & Meta Tasks
- Produce an output of [Domain Relevance | Identifiability | Side]
- Identification
- Produce an output of [Individual label]
- Evaluation
- Produce a score for a given scene image
A demo dataset has been made available
This dataset includes all the various files and options to test the methods demonstrated here.
The dataset contains one "source" of images (the Images folder), which contain the original, un-cropped images.
These un-cropped versions are crucial when performing the image-quality step, as they contain information regarding
image brightness and sharpness. Additionally, these images have been manually annotated
with bounding boxes, and the bounding box labels are in the yolo_labels folder.
These images and bounding box labels are available to anyone wishing to train their own object detection model.
The dataset also contains a cropped_images folder, which contains, as one may infer, the cropped, fin-containing
segments of the original images found in the Images folder. These, however, are further separated
by a unique identifier, which has been anonymized from the original killer whale.
These cropped images are already ready for the identification step, as detailed below. As each of these images contains one individual, and there should therefore be one bounding box and one cropped image representing one individual, there are no extraneous bounding boxes / crops available for training of the two quality evaluation steps (domain relevance and identifiability). However, when training a model on this data, and evaluating on a set of your choice, you will most definitely arrive at detections which are neither relevant to your domain or identifiable. These cropped images form the negative examples for both domain relevance and identifiability, and these two datasets are left to the experts in the various domains for creation.
This repository also includes a convert_demo.py script to convert this demonstration dataset to the folder and file
structure recommended for use in this pipeline setup. It can be used after downloading the above-mentioned
dataset like this:
python convert_demo.py --demo_dir /path/to/demo
Configuration is handled by OmegaConf. Each task has an associated configuration script or directory.
In the cases where multiple files are used for configuration instead of a single one (for all tasks except
detection), all files will be combined during training / evaluation, so there is no need to combine them manually.
There are separate configuration directories for validation & meta tasks, and identification. But the directory structure is the same
- data.yaml [validation | identification]: defines directories for datasets, as well as augmentation options, split parameters, etc.
- distance.yaml [validation | identification]: defines which distance metrics should be used
- general.yaml [validation | identification]: defines some general parameters, largely unused
- loss.yaml [validation | identification]: defines loss function parameters
- mining.yaml [validation | identification]: defines how tuple mining is done (only for contrastive learning)
- model.yaml [validation | identification]: parameters for the model backbone, classification head, etc.
- optimization.yaml [validation | identification]: parameters for optimizer, learning rate, etc.
- reducing.yaml [validation | identification]: parameters for reduction
- training.yaml [validation | identification]: parameters for training, such as batch size, gradient accumulation, etc.
Each module of the pipeline has its own method of usage after training. The output of the training procedure
is different depending on the module. For detection, it is a *.pt file, for identification it is a *.ncc and
a *.ckpt file, and for quality evaluation and validation the output is a *.ckpt file. How the end user integrates
these outputs is up to them, but there are python scripts for detection,
validation, and identification.
The deployment scripts for TorchServe are made available, with their handlers for use in that environment.
The pipeline configuration file contains the structure necessary to run the pipeline with all available modules.
The paths necessary for this file are the results of model trainings as described below.
Running the pipeline, in its entirety, without creating any environment-specific files (ONNX, ...), can be done by running run.sh, as long as the pre-requisite models have been trained and the artefacts created by those processes (the trained model files) are available.
Specifying variables such as triage will forego any use of the identification module / model
and only focus on providing a numeric score for each image. This can then be used as a threshold for
assessing large data volumes, for example.
Fin (object of interest (OOI)) detection is done by using the YOLOv8 object detection mechanism. For more details on YOLOv8 from ultralytics, see their website.
For the purposes of downstream identification (of killer whales), bounding boxes should circumscribe the dorsal fin and saddle patches with minimal additional background / environmental features.
For purposes of demonstration, this dataset has already been labelled with YOLOv8-style bounding boxes. Therefore, training a fin detection model is as simple as running train_detector.sh. All that needs to be done is setting the following variables in the script
source_dir="/path/to/FinPrintv2"
save_dir="/path/to/model/output"
env_dir="/path/to/pip/environment"
and the path variable in the fin_detect.yaml file (and fin_detect-test.yaml)
It is important to note that YOLO expects a 'labels' folder, so you may want to move / rename the yolo_labels folder
to simply labels.
If the train/val/test split files do not exist, they will be created with an 80/20/20 split fraction.
All methods here, whether using contrastive learning or not, support class distinction by folder name. That means, if you have a file structure like:
├── 05a9a14eed
│ ├── 05a9a14eed_fin0_IMG_0000.jpg
├── 0925f5a4e2
│ ├── 0925f5a4e2_fin0_IMG_0025.jpg
├── 1d18ab6f29
│ ├── 1d18ab6f29_fin0_IMG_0050.jpg
├── 212fce08d8
│ ├── 212fce08d8_fin0_IMG_0075.jpg
├── 302fe3047c
│ ├── 302fe3047c_fin0_IMG_0100.jpg
├── 5896882672
│ ├── 5896882672_fin0_IMG_0125.jpg
├── 5d3a5308e1
│ ├── 5d3a5308e1_fin0_IMG_0150.jpg
├── 6f628d03b2
│ ├── 6f628d03b2_fin0_IMG_0175.jpgthen the classes will probably be 05a9a14eed, 0925f5a4e2, 1d18ab6f29, etc. This applies to all classification tasks
like identification, validation, etc.
The train_identifier.sh script is provided for a SLURM environment for model training, but the script can be used outside of that environment as well.
Before running the script, you must update the following placeholders to match your environment:
| Placeholder | Description |
|---|---|
IDENTIFICATION_JOB |
A descriptive name for your SLURM job (e.g., fin-id-training) |
NODE1,NODE2,... |
(Optional) Comma-separated list of compute nodes to exclude from scheduling |
/path/to/data |
Absolute or relative path to your dataset directory |
/path/to/training/output |
Directory where model checkpoints and logs will be saved |
/path/to/your/code/${method}/configuration |
Path to your experiment configuration files |
/path/to/your/code/${method} |
Path to the directory containing main.py and associated code |
/path/to/your/envs |
Base directory to create virtual environments (or update to use a conda environment if preferred) |
http://your.proxy.server:port |
(Optional) HTTP proxy address if required for package installation (remove if unused) |
dataset |
Name of the dataset identifier used internally in your training code |
suffix |
Unique identifier for the experiment (used in logging and output directory naming) |
overrides |
JSON string with configuration overrides (optional) |
Tip: You can also pass many of these values using command-line arguments:
bash run_training.sh -s experiment_name -d MyDataset -r /data/dir -t /training/outputThe train_validator.sh script provides methods to train the two quality classifiers.
The required folder structure for training the domain relevance and identifiability classifiers is the following:
bash run_validation.sh -s exp1-valid -d MyDataset -o '{"lr":0.001}' -m standardDomain Relevance
├── in_domain
│ ├── <relevant images...>
├── not_in_domain
│ ├── <irrelevant images...>Identifiability
├── identifiable
│ ├── <identifiable images...>
├── not_identifiable
│ ├── <un-identifiable images...>Again, the important thing here are the folder names (in_domain, not_in_domain...)
Before running the script, you must update the following placeholders to match your environment:
| Variable | Description |
|---|---|
suffix |
A short name used to identify the run (e.g., exp1-valid) |
dataset |
Dataset identifier to be passed to the training/validation code |
overrides |
(Optional) JSON string to override configuration settings |
method |
Subdirectory under your source directory where main.py is located (e.g., standard, experimental) |
config_dir |
Full path to your YAML configuration files |
source_dir |
Full path to the directory containing main.py and source code |
data_directory |
Path to the dataset used for validation |
https_proxy |
(Optional) Proxy required for package installation; comment out if unused |
cluster_root |
Root directory for environment and logs setup (typically your home on the cluster) |
The train_evaluator.sh script provides methods to train the quality assessment module. This requires the training of the domain relevance and identifiability modules, as described above.
| Variable | Description | User Action |
|---|---|---|
yolo_1280, yolo_640, yolo_416 |
Paths to various YOLOv8 detection models trained on different image resolutions | Choose one to assign to YOLO_MODEL_PATH |
DOMAIN_MODEL_PATH |
Path to a domain classification model | Must point to a trained .ckpt file |
IDENTIFIABLE_MODEL_PATH |
Path to a model trained to assess identifiability | Must point to a trained .ckpt file |
DATASET_ROOT |
Root directory of the dataset used for triage training | Update with the correct dataset path |
TRAINING_SCRIPT |
Path to the Python training entry point | Set to the correct main.py script |
TRAINING_DIRECTORY |
Where model checkpoints and logs will be saved | Define an appropriate output location |
PROJECT_NAME |
Name used in experiment tracking (e.g., wandb or filenames) | Customize for clarity |
ENV_DIR |
Path to the Python virtual environment | Ensure this points to a valid environment with PyTorch, etc. installed |
https_proxy |
Optional proxy if required by your cluster | Leave as-is or update as needed |
@article{Barnhill2025,
title = {Advances in Deep Learning-Driven Photo Identification and Meta Analysis of Cetaceans in Large Data Repositories},
url = {http://dx.doi.org/10.2139/ssrn.5280016},
DOI = {10.2139/ssrn.5280016},
publisher = {Elsevier BV},
author = {Barnhill, Alexander and Towers, Jared R. and Shaw, Tasli J. H. and Arias, Magdalena and Bécares, Adrián and Doniol-Valcroze, Thomás and von Fersen, Lorenzo and Genoves, Rodrigo and R\"{o}rup, Tim and Sutton, Gary J. and Thornton, Sheila and Weiss, Michael and Maier, Andreas and N\"{o}th, Elmar and Bergler, Christian},
year = {2025}
}