You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#1795's fix (see the PR that closes it) stamps LadderSnapshot::verifier_independent
on every model verdict: Some(false) = the verdict call resolved to the worker's
own model. The 46%-agreement measurement cited in crates/stella-pipeline/src/verify.rs was made under exactly that self-graded
condition, but the calibration fold cannot see the distinction: CalibrationReport
(crates/stella-pipeline/src/replay.rs) tallies verifier passes/false-positives
as one cohort.
What to build
Extend the calibration fold to partition the verifier cohort by snapshot.verifier_independent:
The same split for reconciled/false-positive counts, so stella verify calibration (or whatever renders render_calibration) can
answer THE question the field exists for: is a self-graded PASS measurably
less trustworthy than an independent one?
Where to look
crates/stella-pipeline/src/replay.rs — CalibrationReport, calibration, render_calibration, and replay/late_reconciliation_tests.rs.
The report renders the two cohorts' false-positive rates side by side (with unknown excluded, never folded into either), pinned by a fold test over a
scripted stream carrying both polarities.
Problem
#1795's fix (see the PR that closes it) stamps
LadderSnapshot::verifier_independenton every model verdict:
Some(false)= the verdict call resolved to the worker'sown model. The 46%-agreement measurement cited in
crates/stella-pipeline/src/verify.rswas made under exactly that self-gradedcondition, but the calibration fold cannot see the distinction:
CalibrationReport(
crates/stella-pipeline/src/replay.rs) tallies verifier passes/false-positivesas one cohort.
What to build
Extend the
calibrationfold to partition the verifier cohort bysnapshot.verifier_independent:verifier_passes_self_graded/_independent/_unknown(pre-Nothing enforces verifier != worker for the verdict call — one configured provider means the worker grades itself with a prose warning #1795snapshots and worker-unresolvable runs stay
unknown— never assumed).stella verify calibration(or whatever rendersrender_calibration) cananswer THE question the field exists for: is a self-graded PASS measurably
less trustworthy than an independent one?
Where to look
crates/stella-pipeline/src/replay.rs—CalibrationReport,calibration,render_calibration, andreplay/late_reconciliation_tests.rs.crates/stella-protocol/src/ladder.rs—LadderSnapshot::verifier_independent.measurement.
Done when
The report renders the two cohorts' false-positive rates side by side (with
unknownexcluded, never folded into either), pinned by a fold test over ascripted stream carrying both polarities.