Skip to content

Report uncertainty: per-task confidence intervals on the score #74

Description

@edonadei

Why

Caliper reports point estimates with no uncertainty. At the default --k 3, the difference between 3/3 and 2/3 is noise, but compare will flag it as a regression under the any-below rule (docs/CONTEXT.md -> Regression). So Caliper can currently report a confident regression that is nothing at all.

This is the deeper half of the trustworthiness question in docs/adr/0013-a-blocking-score-is-pre-registered.md — the flat 5% non-inferiority margin is a stand-in for an interval that should widen honestly at low k.

Shape

  • Per-task interval on TaskResult.score (Wilson is the obvious default for a small-n binomial; the score is successes / usable).
  • n is the usable count, not k — consistent with the score denominator.
  • Surface in the run table and in compare, so a delta can be read against the noise floor.
  • Longer term: let the gate in Add a pre-registered gate: declared bar + nonzero exit on failure #73 block on the interval rather than the point estimate.

Non-goals

  • Not a significance test between two runs. Start by making the noise floor visible; deciding a comparison is a separate step.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions