Abstract: Automated analysis of bird behavior from video has the potential to enable large-scale ecological monitoring. Existing methods for bird behavior recognition rely almost exclusively on RGB information, overlooking complementary modalities that have proven effective in other domains. In this work, we present ViP-BiRd, a multimodal framework that jointly exploits visual appearance from RGB videos and explicitly models bird pose dynamics for subject-level bird behavior recognition. Given the lack of a standardized bird pose skeleton, we introduce a 10-node avian skeleton representation and an extraction pipeline based on off-the-shelf animal pose keypoint detectors. Experiments on the Visual WetlandBirds dataset demonstrate that multimodal RGB–pose fusion consistently outperforms unimodal baselines.
- 10-Node Avian Graph: A robust skeletal representation optimized for wetland birds, invariant to the specific tracking algorithm.
- Multimodal Fusion: Effectively combines appearance (V-JEPA/RGB) and structural motion (ST-GCN++/Pose).
- State-of-the-Art Baseline: Record performance of 55.91% on the Visual WetlandBirds benchmark compared to unimodal 49.46% (RGB) and 47.31% (Pose).
src/: Core Python modules and scripts.scripts/: Training, evaluation, and data processing shell scripts.models/: PyTorch definitions for Visual-Pose fusion modules.configs/: Training configuration parameters.logs/: Checkpoints and experiment logs.
Important: All scripts and models in this repository must be executed using the official Docker container.
The project requires a specific PyTorch+CUDA ecosystem (e.g. mi_proyecto/pytorch_gpu_openface).
docker run --gpus all -it --rm -v $(pwd):/workspace mi_proyecto/pytorch_gpu_openface /bin/bashIf you find this code or our paper useful for your research, please consider citing:
@inproceedings{sancho2024vipbird,
title={},
author={},
year={}
}