Why
caliper run reports a score and always exits 0 — the only nonzero exit is a backend configuration error (caliper/commands/run.py:171). There is nothing to put in CI, so a score can inform a model change but cannot block one.
See docs/adr/0013-a-blocking-score-is-pre-registered.md: a score is only blocking if the task set, k, and the bar were fixed before the run produced a number. The gate is the artifact that makes that enforceable rather than advisory.
Shape
- Declare the bar next to the spec (committed, so it is pre-registered by construction) rather than as an ad-hoc flag that can be tuned after seeing the result.
- Default rule = the existing non-inferiority bar (
docs/CONTEXT.md): at k>=5, within 5% of control and still beating --baseline.
- Nonzero exit when the run misses the bar, distinct from the existing exit 2 for configuration errors.
- Unusable attempts must not silently satisfy or fail the gate — the denominator rule from
docs/adr/0001-attempt-outcome-taxonomy.md applies unchanged.
Open
- Where the bar lives: spec field vs. a separate committed policy file. A spec field conflicts with
docs/adr/0004 only if the bar is engine-dependent, which it should not be.
- Whether
compare gets the same gate, or stays purely a reporting surface.
Why
caliper runreports a score and always exits 0 — the only nonzero exit is a backend configuration error (caliper/commands/run.py:171). There is nothing to put in CI, so a score can inform a model change but cannot block one.See
docs/adr/0013-a-blocking-score-is-pre-registered.md: a score is only blocking if the task set,k, and the bar were fixed before the run produced a number. The gate is the artifact that makes that enforceable rather than advisory.Shape
docs/CONTEXT.md): atk>=5, within 5% of control and still beating--baseline.docs/adr/0001-attempt-outcome-taxonomy.mdapplies unchanged.Open
docs/adr/0004only if the bar is engine-dependent, which it should not be.comparegets the same gate, or stays purely a reporting surface.