Audio Understanding Benchmarks — BirdCLEF 2024 + CoughBench

Overview

• This repository evaluates a multimodal audio model across two domains:
  • BirdCLEF-2024 bird sound identification (India-focused subset)
  • CoughBench respiratory audio tasks (COUGHVID, Coswara, NoCoCoDa)
• Benchmarks include naming, forced-choice recognition, and pairwise discrimination. Results are saved to outputs/ and summarized in notebooks for presentation.

Benchmarks At A Glance

• BirdCLEF Identification
  • Unconstrained: predict common + scientific name (JSON). Baseline ≈ 1/N species in sample.
  • 4-Choice: choose among 4 options. Baseline 25%.
  • Pairwise (Same/Different): are two clips of the same species? Baseline 50%.
• CoughBench
  • Worrying vs Not (COUGHVID): binary triage proxy. Baseline 50%.
  • Pairwise (Same-Type): do two coughs imply the same type/severity? Baseline 50%.

Methodology

• Sampling (BirdCLEF)
  • scripts/sampling.py:sample_stratified builds up to 1000 clips, capped at ≤5 per species to reduce long-tail dominance. The effective N species in the current sample is shown in the reporting notebook.
• Preprocessing
  • Bird clips: OGG → WAV when needed, mono 16 kHz, base64-encoded.
  • CoughBench: all audio normalized, mono 16 kHz WAV. Pair audio concatenated with 2 s silence.
• Prompting
  • Prompts are concise, JSON-only instructions to keep parsing deterministic:
    • Birds: scripts/bench_audio_understanding.py (unconstrained, 4-choice) and scripts/pairs_benchmark.py (pairwise).
    • Coughs: scripts/coughbench_model_io.py (worrying, type-classification, same-type pairs). Pair prompt focuses on “same clinical worrying severity/intensity”.
• Rate-limit handling
  • Shared helper scripts/retry_utils.py retries indefinitely on HTTP 429 with exponential backoff (birds). CoughBench uses internal backoff. Jobs resume on rerun; already processed items are skipped.
• Scoring
  • Birds: scripts/scoring.py resolves predictions to species codes (via canonical name maps) and computes accuracy and Wilson CIs. Pairwise scores same/different correctness.
  • CoughBench: scripts/coughbench_scoring.py computes accuracy, precision/recall/F1 (worrying), per-class accuracies (type), and pairwise accuracy.

How To Run

• Prerequisites
  • Python 3.10+, ffmpeg in PATH.
  • Install deps: pip install -r requirements_coughbench.txt (plus standard libs for notebooks: pandas, matplotlib, seaborn optional).
  • Env var: set BOSON_API_KEY. Optional: BOSON_API_BASE.
• BirdCLEF identification (unconstrained + 4-choice)
  • python -m scripts.bench_audio_understanding --full --workers 4 --limit 1000
  • Sampling CSV regenerates if --limit changes; raw JSONL in outputs/raw/, scored CSV in outputs/scored/, metrics JSON in outputs/metrics/.
• Bird pairwise (same/different)
  • python -m scripts.pairs_benchmark --full --workers 4
  • Appends to outputs/pairs/raw.jsonl and scores to outputs/metrics/pairs_summary.json.
• CoughBench end‑to‑end
  • python -m scripts.coughbench_runner full --normalize peak --seed 2026 --api-delay 0.1
  • Produces raw under outputs/coughbench/raw, scored CSVs and metrics in outputs/coughbench/scored|metrics.

Outputs

• BirdCLEF
  • Raw: outputs/raw/unconstrained.jsonl, outputs/raw/4choice.jsonl
  • Scored: outputs/scored/unconstrained.csv, outputs/scored/4choice.csv
  • Metrics: outputs/metrics/unconstrained_summary.json, outputs/metrics/4choice_summary.json, outputs/metrics/pairs_summary.json
• CoughBench
  • Raw: outputs/coughbench/raw/worrying.jsonl, .../type_unconstrained.jsonl, .../type_constrained.jsonl, .../pairs.jsonl
  • Scored: outputs/coughbench/scored/*.csv
  • Metrics: outputs/coughbench/metrics/*.json

Results & Notebooks

• Bird Report: notebooks/benchmark_report.ipynb
  • Shows overall accuracies with chance baselines, species coverage, pair balance, latency, and prompt sanity checks.
• Cough Report: notebooks/coughbench_report.ipynb
  • Explains dataset provenance (COUGHVID worrying flag, Coswara labels), class distributions, accuracy vs chance, confidence plots, and latency.
• Tip: After running benchmarks, “Restart & Run All” to refresh charts.

Run Configuration & Seeds

• BirdCLEF identification seed: 2025
• Bird pairwise seed: 2026
• CoughBench seed: 2026 (normalize peak, API delay 0.1 s)

Relevance

• The three BirdCLEF tasks probe naming, forced-choice disambiguation, and pure discrimination—complementary capabilities for biodiversity monitoring.
• The three CoughBench tasks mirror clinical use-cases: triage risk (worrying), phenotype labeling (type), and longitudinal comparison (same-type).

Troubleshooting

• HTTP 429 / rate limits: birds use retry_utils.py to wait and resume; coughbench has exponential backoff per call. Re-running the same command safely resumes.
• Partial outputs: some runners flush raw JSONL at task completion (e.g., coughbench), so you won’t see line-by-line writes mid-loop.

Data Sources

• COUGHVID (ETH Zürich): https://zenodo.org/records/7024894 (worrying labels via expert review)
• BirdCLEF 2024: https://www.kaggle.com/competitions/birdclef-2024/data