Skip to content

Add a pre-registered gate: declared bar + nonzero exit on failure #73

Description

@edonadei

Why

caliper run reports a score and always exits 0 — the only nonzero exit is a backend configuration error (caliper/commands/run.py:171). There is nothing to put in CI, so a score can inform a model change but cannot block one.

See docs/adr/0013-a-blocking-score-is-pre-registered.md: a score is only blocking if the task set, k, and the bar were fixed before the run produced a number. The gate is the artifact that makes that enforceable rather than advisory.

Shape

  • Declare the bar next to the spec (committed, so it is pre-registered by construction) rather than as an ad-hoc flag that can be tuned after seeing the result.
  • Default rule = the existing non-inferiority bar (docs/CONTEXT.md): at k>=5, within 5% of control and still beating --baseline.
  • Nonzero exit when the run misses the bar, distinct from the existing exit 2 for configuration errors.
  • Unusable attempts must not silently satisfy or fail the gate — the denominator rule from docs/adr/0001-attempt-outcome-taxonomy.md applies unchanged.

Open

  • Where the bar lives: spec field vs. a separate committed policy file. A spec field conflicts with docs/adr/0004 only if the bar is engine-dependent, which it should not be.
  • Whether compare gets the same gate, or stays purely a reporting surface.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions