Skip to content

[EPIC PRD-002] judge panel layer — methodology + scoring infrastructure #12

Description

@explosivebit

Trace: PRD prd-judge-panel-layer-methodology-scoring-infrastructure (stub, active)

Status: stub — full requirements develop in Phase 3 (T+2..T+4 weeks) after smoke run published.

Scope summary:

  • Build on Inspect AI multi_scorer(model_graded_qa(model=[...])) per EVID-004
  • Median reducer override (Inspect default mean; POLLMEVALS uses median per EVID-001 HELM divergence)
  • Blind labels + anonymisation
  • Calibration suite + Krippendorff α target ≥ 0.70

Blocked by: PRD-001 (smoke must work first)

Next: at Phase 3 start, deepen PRD-002 to Standard depth, create SPEC-002 (judge contracts), RFC-002 (impl plan), ADR-004 (median vs mean rationale).

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/prdProduct Requirements Documentphase/3-judgesPhase 3 — Judge panel layerpriority/p2Medium — next Phasetype/featNew feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions