Bidiff leads with an execution-unstable classification code and ships no false-alarm profile

[jszmajda]

Context / Problem

Bidiff's user-facing headline is a per-EARS classification code. EXP-BIDIFF-015:

the system SHALL produce a user summary with per-arrow classification counts, the top-priority drift findings across the audited scope, and the recommended reconciliation actions.

B10 measured those codes and found them unreliable at exactly that unit. RESULTS.md, finding 7:

Classification codes remain descriptive, not discriminative

and in Confounds and limits:

Unit-level classification codes are execution-unstable (BD-DRIFT ↔ B-ONLY-DRIFT flip observed between duplicate T2 executions of one unit); per-finding scoring is unaffected.

So the summary leads with the unit that flips between identical re-runs, and subordinates the per-finding scoring that held up — including through 22 human-adjudicated disputes where "recall was robust to the entire dispute."

Second half: no error profile travels with a finding. B10 measured one and it is reassuring, which is precisely why it is worth shipping:

False alarms on untouched specs | 0–7.5% per arm; most rejected "noise" is missing cross-references between sibling specs, not fabrication

plus prominence — when the instrument catches a seeded gap it leads with it (matched finding almost always #0). None of this reaches a user reading an audit record. They get a code and a Default Action.

The asymmetry that makes this more than a port. B10 could compute recall because it had a sealed answer key. A user running /differential-audit on their own repo has no key and never will, so recall and precision are uncomputable in the field. "Error matrix rather than a score" therefore cannot mean the same thing in both places. What is portable is calibration — known false-alarm rate, known dominant false-alarm shape, known prominence — attached to findings, so a user reads a drift finding with the instrument's measured error profile rather than a bare verdict.

Cold readers repeatedly asked for an error matrix rather than a score for bidiff results. At the research layer B10 already delivers that (recall by class × arm, false alarms per arm, prominence, stability, cost). The gap is in what the shipped instrument hands a user.

Proposed change

A direction, not a spec.

• Lead the user summary and audit record with per-finding results (realness / severity), and subordinate the per-EARS classification code.
• If the codes are retained in the presentation, mark them descriptive rather than discriminative, so nobody reads a code as a verdict.
• Attach a calibration note to findings, sourced from measured false-alarm data, rather than a computed matrix — including the dominant false-alarm shape (missing cross-references between sibling specs, not fabrication).
• State plainly that recall and precision are not computable in a field audit, so no one later tries to ship them.
• Research-layer residue while in here: B10 CRITIQUE B6 notes per-unit false-alarm rates by tier are "derivable but not tabulated." Tabulate or explicitly defer.

Acceptance criteria

• The headline unit of the user summary is decided — per-finding vs. per-EARS code — and the rationale recorded in Decisions & Alternatives
• Classification codes, wherever presented, are labeled descriptive rather than discriminative if retained
• Findings carry a calibration/error profile sourced from measured data, or its absence is explicit
• The "no ground truth in a field audit" asymmetry is stated, so recall/precision are not attempted downstream
• EXP-BIDIFF-014 and EXP-BIDIFF-015 updated; references/audit-report-template.md cascades
• B10 CRITIQUE B6 residue (per-unit false-alarm rates by tier) tabulated or explicitly deferred
• Docs read cold (fresh-author test)

Scope & non-goals

• Not the correlated-priors / verifier-independence problem — its own issue.
• Not a re-run of B10, and not new measurement. This consumes numbers B10 already produced.
• Not a new command or report surface. Reshape the output that EXP-BIDIFF-014/-015 already produce.

Intent touchpoints (LID)

• Existing EARS: EXP-BIDIFF-014 (audit record contents), EXP-BIDIFF-015 (user summary)
• docs/intent/lid-experimental/bidirectional-differential/bidirectional-differential-design.md — Output shape, Classification codes, Decisions & Alternatives
• Skill assets: references/audit-report-template.md, references/classification-codes.md
• New EARS if a calibration requirement lands: EXP-BIDIFF prefix, number unassigned

Related

• B10: RESULTS.md finding 7 + Confounds and limits; CRITIQUE.md B6 (on the b10-differential-catch-rate bookmark, unmerged).

• Bidiff's two directions share priors by design: correlated blind spots classify as BD-COHERENT #71 — the correlated-priors problem in the same instrument. Independent of this one: the reporting shape would matter even if the two directions were perfectly independent.