Evaluation harness & golden set

[thorwhalen]

Scope

Know it's working — a layered harness from cheap deterministic QC to cost-gated fit-judging.

Tasks

• Tier-1: ranx retrieval metrics (R@k / mAP@10 / nDCG@10) over a frozen golden set; PR gate on Δ nDCG@10 ≥ −0.02
• [~] Build the ~150-item golden (6-item Ring-0 seed shipped in 10a; the ~150-item captioned corpus is a deferred scale-up) — original: Build the ~150-item golden (context → expected-sound) fixture (dog-foods decompose_context)
• Tier-2: audio-LM / LLM fit-judge + generation fidelity (PAM, Audiobox-PQ, FAD-P / KAD) + Krippendorff-α human calibration

Grounded in

Research reports 08, 11 — see misc/docs/research/ (authoritative synthesis in report 10).

Depends on

Index, Corpora, Select agent

---

Part of the foley build epic (see the tracking epic issue).

[thorwhalen]

Evaluation harness landed (Tier-0 / Tier-1 / Tier-2)

Tier-1 (10a, PR #21): pure-numpy ranx-equivalent retrieval metrics (nDCG@10 / R@k / mAP / MRR) + the Δ nDCG@10 ≥ −0.02 per-PR gate over a frozen Ring-0 golden seed.

Tier-2 (10b, PR #28): the judge-based fit harness (foley.eval.fit → fit-precision over a seeded stratified sample; the fit-judge binds at #7's verify_match seam), pure-numpy FAD/KAD generation fidelity (foley.eval.fidelity), and Krippendorff-α inter-rater reliability + judge-vs-human calibration (foley.eval.reliability). foley.evaluate_fit() + a report-only foley eval-fit CLI. Tier-2 is nightly/pre-release, never touches retrieval ranking, and is never a per-PR gate.

Deferred follow-ups (seams shipped; heavy externals wired later — consistent with the epic's discipline)

• Scale the golden fixture from the 6-item Ring-0 seed to the ~150-item captioned corpus (FSD50K/Clotho slice).
• Wire the real Qwen2-Audio weights into AudioLMJudge (class + injectable fake-pipeline seam shipped; _build_pipeline deferred behind foley[fit]).
• Real fadtk/kadtk reference toolkits + PAM / Audiobox-PQ per-sample generation gates.
• A committed fit-baseline stamp (once real judges produce stable numbers) and MUSHRA/2AFC human-panel infrastructure (Bradley–Terry) to feed the Krippendorff-α calibration.

Closing the harness-capability work; the deferrals above are enhancement follow-ups.

[thorwhalen]

The deferred golden-set scale-up checkbox is done: #34 grew the frozen retrieval-eval set from 6 → 156 items (a decoupled caption-only eval corpus + a regenerated gld-v2 nDCG baseline), for a tighter, broadly-covering gate. Landed as part of the post-v1 pass (main @ v0.0.20).