Skip to content

calibration: partition verifier false-positive rates by the new grader-independence fact #1865

Description

@macanderson

Problem

#1795's fix (see the PR that closes it) stamps LadderSnapshot::verifier_independent
on every model verdict: Some(false) = the verdict call resolved to the worker's
own model. The 46%-agreement measurement cited in
crates/stella-pipeline/src/verify.rs was made under exactly that self-graded
condition, but the calibration fold cannot see the distinction: CalibrationReport
(crates/stella-pipeline/src/replay.rs) tallies verifier passes/false-positives
as one cohort.

What to build

Extend the calibration fold to partition the verifier cohort by
snapshot.verifier_independent:

Where to look

  • crates/stella-pipeline/src/replay.rsCalibrationReport, calibration,
    render_calibration, and replay/late_reconciliation_tests.rs.
  • crates/stella-protocol/src/ladder.rsLadderSnapshot::verifier_independent.
  • Turn on the independent checker and measure whether it helps #1284 (measure the independent checker) — this is the reading side of that
    measurement.

Done when

The report renders the two cohorts' false-positive rates side by side (with
unknown excluded, never folded into either), pinned by a fold test over a
scripted stream carrying both polarities.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Important — next in linearea:pipelinestella-pipeline — triage→…→judge orchestration

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions