audiocurate

A small command-line tool and library for curating large audio/speech datasets. It walks a directory tree, probes each file, computes cheap quality signals, drops the obviously-broken clips, removes duplicates, and emits a reproducible train/dev/test manifest plus a summary report.

It is built for the boring-but-necessary step before training: turning a folder of mixed-quality audio into a clean, documented, seed-stable manifest you can hand to a data loader.

Install

pip install -e .
# optional fallback decoder for mp3/m4a:
pip install -e ".[torch]"

Requires Python 3.9+. Core dependencies are numpy, soundfile, pandas, pyyaml, click and tqdm.

How it works

The pipeline is four stages, each reading and writing a manifest so you can stop, inspect, and resume between them:

  scan   -> probe headers + compute quality signals  -> raw.jsonl
  filter -> apply threshold rules from a config       -> filtered.jsonl
  dedup  -> drop exact + near-duplicate audio         -> deduped.jsonl
  split  -> deterministic train/dev/test + sharding   -> manifest.jsonl
  report -> summary statistics over a manifest        -> (stdout / json)

A manifest is newline-delimited JSON (one record per line). TSV output is also supported for spreadsheets; pass a .tsv path and the quality block is flattened into columns.

Quality signals are plain numpy DSP, computed on a bounded window of each file (30s by default): clipping ratio, RMS/peak loudness in dBFS, silence ratio, an energy-VAD ratio, and a crude SNR estimate from the spread between speech-level and noise-level frame energies.

CLI reference

audiocurate scan   <root> -o <manifest> [--no-quality] [--probe-seconds N]
audiocurate filter <manifest> -o <out> [-c config.yaml]
audiocurate dedup  <manifest> -o <out> [--exact-only] [--max-distance N]
audiocurate split  <manifest> -o <out> [-c config.yaml] [--seed N] [--group-by KEY]
audiocurate report <manifest> [--json]

Add -v (info) or -vv (debug) before the subcommand for logging.

Scan

$ audiocurate -v scan ./corpus -o raw.jsonl --probe-seconds 30
scan: 100%|████████████| 8421/8421 [02:13<00:00, 63.0file/s]
scanned -> 8409 records written to raw.jsonl

A single record looks like:

{"id": "corpus/spk07/utt_0142.wav", "path": "corpus/spk07/utt_0142.wav",
 "duration_sec": 4.31, "sample_rate": 16000, "channels": 1,
 "quality": {"clipping_ratio": 0.0, "rms_db": -23.118, "peak_db": -6.02,
             "silence_ratio": 0.214, "est_snr_db": 27.84, "vad_ratio": 0.61},
 "content_hash": "8f1c…", "fingerprint": "00a3…", "split": null, "shard": null}

Filter

$ audiocurate filter raw.jsonl -o filtered.jsonl -c configs/default.yaml
{
  "kept": 7813,
  "dropped": 596,
  "by_reason": {
    "max_silence_ratio": 311,
    "min_snr_db": 142,
    "min_rms_db": 88,
    "max_clipping_ratio": 41,
    "min_duration_sec": 14
  }
}

Split + report

$ audiocurate split deduped.jsonl -o manifest.jsonl -c configs/default.yaml
{"counts": {"train": 6731, "dev": 372, "test": 358}, "shards": {"train": 4, "dev": 1, "test": 1}}

$ audiocurate report manifest.jsonl
files        : 7461
total hours  : 11.83

split        files      hours
----------------------------------
train          6731      10.67
dev             372       0.59
test            358       0.57

sample rates : 16000Hz x7461

quality signal        p5       p50       p95
----------------------------------------------
duration_sec        1.402     4.180    14.905
rms_db            -34.221   -23.004   -14.880
clipping_ratio      0.000     0.000     0.002
silence_ratio       0.041     0.198     0.512
est_snr_db         14.910    26.880    38.220

Config reference

Filter thresholds and split settings live in one YAML file. Every filter key is optional — omit a key to disable that rule.

filter:
  min_duration_sec: 1.0
  max_duration_sec: 30.0
  min_sample_rate: 16000
  allowed_channels: [1]
  max_clipping_ratio: 0.01
  min_rms_db: -45.0
  max_silence_ratio: 0.6
  min_snr_db: 10.0
  min_vad_ratio: 0.2

split:
  train: 0.90
  dev: 0.05
  test: 0.05
  seed: 1234
  shard_size: 2000
  # group_by: speaker   # keep all clips of a speaker in one split

See configs/default.yaml and configs/strict.yaml.

Reproducibility

Splits are deterministic in the record id and the seed, not in iteration order or worker count. The bucket for a file is hash(seed:id) mapped into [0,1), so re-running with the same seed reproduces the split even after new files are added — existing files keep their bucket. Use --group-by speaker (or any meta key) to guarantee a group never leaks across train/test.

Changelog

• v0.5 (2026-06) — strict config profile, TSV manifest round-trip, per-split hours in the report.
• v0.4 (2026-02) — near-duplicate grouping via perceptual fingerprint; dedup subcommand.
• v0.3 (2025-11) — deterministic seeded splitting + sharding, --group-by.
• v0.2 (2025-08) — config-driven filtering with drop-reason summary; pandas report.
• v0.1 (2025-06) — first cut: directory scan, header probe, numpy quality signals, JSONL manifest.

License

BSD-3-Clause. Copyright (c) 2025 Wu Yunling.