Human-curated / real-clip golden set (higher-signal eval)
Follow-up to #10. The frozen retrieval-eval set was scaled 6 → 156 items in #34, but those
items are synthetic: a caption-only Ring-0 corpus with keyword queries derived from the
captions. With the deterministic bag-of-words embedder that yields a clean, diverse regression
gate (nDCG@10 = 1.0 by design) — good for catching pipeline breakage, but it does not
measure real semantic-retrieval quality (synonyms, paraphrase, cross-modal fit).
Goal
A smaller human-curated golden set over real clips (a licensed subset — FSD50K / Sonniss /
BBC / Freesound-CC0) with natural-language queries and graded relevance (0/1/2), so the eval
measures the real CLAP + hybrid-search ranking, not just word overlap.
Scope
- Curate ~50–100
(context → expected sounds) items over real, license-clean clips; graded
relevance with 1–2 acceptable alternates per event (exercise nDCG's graded discrimination).
- A second baseline computed with the real CLAP embedder (not the hashing BoW) — likely
< 1.0, so a meaningful floor; keep the synthetic Ring-0 gate as the fast, deterministic,
no-download CI gate.
- Optional: inter-rater reliability (Krippendorff α — the harness already exists) over a
double-labeled subset.
Acceptance
foley eval reports both the Ring-0 (synthetic, CI) and the real-clip (curated) nDCG.- The real-clip baseline is committed + sha-stamped like the synthetic one.
Refs #10. Companion to the synthetic scale-up in #34.
Human-curated / real-clip golden set (higher-signal eval)
Follow-up to #10. The frozen retrieval-eval set was scaled 6 → 156 items in #34, but those
items are synthetic: a caption-only Ring-0 corpus with keyword queries derived from the
captions. With the deterministic bag-of-words embedder that yields a clean, diverse regression
gate (nDCG@10 = 1.0 by design) — good for catching pipeline breakage, but it does not
measure real semantic-retrieval quality (synonyms, paraphrase, cross-modal fit).
Goal
A smaller human-curated golden set over real clips (a licensed subset — FSD50K / Sonniss /
BBC / Freesound-CC0) with natural-language queries and graded relevance (0/1/2), so the eval
measures the real CLAP + hybrid-search ranking, not just word overlap.
Scope
(context → expected sounds)items over real, license-clean clips; gradedrelevance with 1–2 acceptable alternates per event (exercise nDCG's graded discrimination).
< 1.0, so a meaningful floor; keep the synthetic Ring-0 gate as the fast, deterministic,
no-download CI gate.
double-labeled subset.
Acceptance
foley evalreports both the Ring-0 (synthetic, CI) and the real-clip (curated) nDCG.Refs #10. Companion to the synthetic scale-up in #34.