You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Caliper reports point estimates with no uncertainty. At the default --k 3, the difference between 3/3 and 2/3 is noise, but compare will flag it as a regression under the any-below rule (docs/CONTEXT.md -> Regression). So Caliper can currently report a confident regression that is nothing at all.
This is the deeper half of the trustworthiness question in docs/adr/0013-a-blocking-score-is-pre-registered.md — the flat 5% non-inferiority margin is a stand-in for an interval that should widen honestly at low k.
Shape
Per-task interval on TaskResult.score (Wilson is the obvious default for a small-n binomial; the score is successes / usable).
n is the usable count, not k — consistent with the score denominator.
Surface in the run table and in compare, so a delta can be read against the noise floor.
Why
Caliper reports point estimates with no uncertainty. At the default
--k 3, the difference between 3/3 and 2/3 is noise, butcomparewill flag it as a regression under the any-below rule (docs/CONTEXT.md-> Regression). So Caliper can currently report a confident regression that is nothing at all.This is the deeper half of the trustworthiness question in
docs/adr/0013-a-blocking-score-is-pre-registered.md— the flat 5% non-inferiority margin is a stand-in for an interval that should widen honestly at lowk.Shape
TaskResult.score(Wilson is the obvious default for a small-n binomial; the score issuccesses / usable).nis the usable count, notk— consistent with the score denominator.compare, so a delta can be read against the noise floor.Non-goals