Context
The full 1,200-cell run (bench/FULL-RUN-FINDINGS.md) reports κ = N/A — no human labels were collected. Every surprising verdict lives in LLM-judge-graded categories (cross-service, call-trace, absence, semantic): e.g. “all conditions ≈0.8 when finished” could be judge leniency, not real quality. The judge was spot-checked on one cell (bc-cs-01_D) but never validated against humans at scale.
Why it blocks
C1 (partial) and C3 (not supported) both hinge on judge scores. Until the judge is human-validated, those verdicts are provisional and not publishable.
Work
- Emit blinded transcripts for a stratified sample (~50–100 cells across categories × conditions, heavy on the judged categories).
- A human labels each blind transcript correct/incorrect against its oracle.
- Compute Cohen’s κ vs the judge (binarized at
JUDGE_CORRECT_THRESHOLD = 0.5).
- Accept at κ ≥ ~0.6; if below, revise the rubric / threshold and re-run.
Acceptance
bench/results/<run>/human_labels.json committed (or referenced).- κ reported in
report.md; verdicts in FULL-RUN-FINDINGS.md upgraded from “provisional” to “validated” (or the findings revised if κ is poor).
Context
The full 1,200-cell run (
bench/FULL-RUN-FINDINGS.md) reports κ = N/A — no human labels were collected. Every surprising verdict lives in LLM-judge-graded categories (cross-service, call-trace, absence, semantic): e.g. “all conditions ≈0.8 when finished” could be judge leniency, not real quality. The judge was spot-checked on one cell (bc-cs-01_D) but never validated against humans at scale.Why it blocks
C1 (partial) and C3 (not supported) both hinge on judge scores. Until the judge is human-validated, those verdicts are provisional and not publishable.
Work
JUDGE_CORRECT_THRESHOLD = 0.5).Acceptance
bench/results/<run>/human_labels.jsoncommitted (or referenced).report.md; verdicts inFULL-RUN-FINDINGS.mdupgraded from “provisional” to “validated” (or the findings revised if κ is poor).