Trace: PRD prd-judge-panel-layer-methodology-scoring-infrastructure (stub, active)
Status: stub — full requirements develop in Phase 3 (T+2..T+4 weeks) after smoke run published.
Scope summary:
- Build on Inspect AI
multi_scorer(model_graded_qa(model=[...])) per EVID-004
- Median reducer override (Inspect default mean; POLLMEVALS uses median per EVID-001 HELM divergence)
- Blind labels + anonymisation
- Calibration suite + Krippendorff α target ≥ 0.70
Blocked by: PRD-001 (smoke must work first)
Next: at Phase 3 start, deepen PRD-002 to Standard depth, create SPEC-002 (judge contracts), RFC-002 (impl plan), ADR-004 (median vs mean rationale).
Trace: PRD prd-judge-panel-layer-methodology-scoring-infrastructure (stub, active)
Status: stub — full requirements develop in Phase 3 (T+2..T+4 weeks) after smoke run published.
Scope summary:
multi_scorer(model_graded_qa(model=[...]))per EVID-004Blocked by: PRD-001 (smoke must work first)
Next: at Phase 3 start, deepen PRD-002 to Standard depth, create SPEC-002 (judge contracts), RFC-002 (impl plan), ADR-004 (median vs mean rationale).