FIX-0: T20 remediation programme, tracking issue

[mateuszwalo]

Tracking issue for the remediation programme that comes out of the T20 findings (#389) and
the statistics defects in #390. Plan document: plans/FIX-PLAN-T20-2026-08-23.md.

What this is fixing

Three facts drive the whole thing.

1. selection_mode="xai" never uses saliency for arbitration. It is a greedy argmax on
   selection-validation accuracy, and the benchmarked contrast was accuracy-argmax vs random.
   It is null at n=10 across two datasets and three protocols.
2. That criterion is close to orthogonal to the objective. On Waterbirds it sits at ~97% while
   the objective sits at 59%, and candidate accuracies differ by fractions of a point.
3. The equal-compute protocol costs the deployed model two thirds of its budget. Matching
   deployed epochs closes 4.46 pp to 0.09 pp on Imagewoof.

Plus a family of silent-failure defects found at source.

Phase 0, independent fixes, all parallel

• FIX-0-1: augmentation_runner silently destroys normalised batches #394 augmentation_runner silently destroys normalised batches
• FIX-0-2: XAI cache is dead when the loader yields no sample indices #395 XAI cache is dead when the loader yields no sample indices
• FIX-0-3: ChurchNoise runs a different transform on CPU and GPU #396 ChurchNoise runs a different transform on CPU and GPU
• FIX-0-4: correct the ICD/AICD description in docs and state when each applies #397 correct the ICD/AICD description in docs, docs only, not a fix
• FIX-0-5: extract benchmarks/stats.py, fix rank-biserial and the paired bootstrap #398 extract benchmarks/stats.py, fix rank-biserial and the paired bootstrap ([Bug]: - rank-biserial, paired Δ/CI, and tie handling are incorrect in benchmarks/summarize_grand.py #390)
• FIX-0-6: compute the EBPG mask-coverage baseline #399 EBPG mask-coverage baseline, no GPU
• FIX-0-7: faithfulness noise floor from forced-independent OptiCAM passes #400 faithfulness noise floor from forced-independent OptiCAM passes
• FIX-0-8: fix group-balanced subsampling in the DFR baseline #401 group-balanced subsampling in the DFR baseline

Phase 1, turn saliency into measurements

• FIX-1-1: saliency_stats.py, concentration, gini, border_mass, perturbation_shift #402 analysis/saliency_stats.py
• FIX-1-2: diagnosis.py, AttentionRegime and the decision rule #403 analysis/diagnosis.py, AttentionRegime and the decision rule
• FIX-1-3: hard_quantile_acc and robustness_gap #404 hard_quantile_acc and robustness_gap
• FIX-1-4: diagnosis thresholds as config with no defaults #405 thresholds as config with no defaults, library refuses until supplied

Phase 2, selection refactor

• FIX-2-1: training/selection.py, CandidateSelector protocol and SELECTORS registry #406 training/selection.py, CandidateSelector and SELECTORS
• FIX-2-2: replace min-max normalisation in branching.py with a noise-scaled unit #407 replace min-max normalisation with a noise-scaled unit
• FIX-2-3: indistinguishability test for candidate selection #408 indistinguishability test, reason="indistinguishable"
• FIX-2-4: deprecate the guessed XAI knobs and fix the double-counted accuracy #409 deprecate the guessed XAI knobs, fix the double-counted accuracy

Phase 3, record and observe

• FIX-3-1: mandatory run-record fields for selection and compute #410 mandatory run-record fields
• FIX-3-2: shadow mode, record the diagnosis without acting on it #411 shadow mode, record the diagnosis without acting on it
• FIX-3-3: diagnosis as a first-class report artifact, plus ECE in the headline table #412 diagnosis as a report artifact, ECE in the headline table

Phase 4, search policy

• FIX-4-1: search_policy, exhaustive, diagnosis_single, successive_halving #413 search_policy: exhaustive, diagnosis_single, successive_halving
• FIX-4-2: confidence fallback to metric_argmax #414 confidence fallback to metric_argmax
• FIX-4-3: benchmark conditions for the new search policies #415 benchmark conditions for the new policies

Block 7, calibration and falsification

• FIX-7-1: pre-registration for the threshold calibration study #416 pre-registration
• FIX-7-2: calibrate the diagnosis thresholds #417 calibrate the thresholds
• FIX-7-3: calibration findings, defaults decision and docs #418 findings, defaults decision, docs

Sequencing that matters

Phase 3 before Phase 4. Shadow mode (#411) records the diagnosis on every run
without letting it decide anything. It is what makes #417 possible: from the moment
it lands, every run that was going to happen anyway becomes a calibration sample at zero extra
GPU cost. Without it the only way to get thresholds is to guess, which is the mistake this
whole programme exists to undo.

Thresholds stay None. #405 makes the library refuse diagnostic mode until they
are supplied, and #417 is the only issue allowed to supply them.

Defaults do not move until Block 7 says so. #413 ships the policies with
exhaustive still default.

Kill criterion, stated up front. If the calibrated rule does not beat metric_argmax on
held-out seeds, diagnosis_single does not become the default and the result is published as a
negative result, the same way T20's null was.

Schedule

Week	Dates	Content
CW9	Aug 25-31	#394 to #398 in parallel, #399 alongside, #402 starts
CW10	Sep 1-7	#399 to #401 finish, Phase 1, #406 starts
CW11	Sep 8-14	Phase 2, Phase 3, shadow runs start collecting
CW12	Sep 15-21	Phase 4, #416, #417 running
CW13	Sep 22-30	Buffer. #418, defaults decision, docs, retro

Roughly 200h of estimates against roughly 195h of capacity, so the plan is full. If it slips,
#412 drops first, then Block 7 becomes a post-cohort study with the shadow data
already collected.

Out of scope for the cohort, deliberately: augmentation_schedule (sequential ICD then AICD,
stochastic mix, Proposal B). It opens a new untested axis and is not needed to close either
defect the benchmarks identified.

Decisions already taken

• Issues live in their own FIX- namespace, separate from the cohort T ladder.
• FIX-0-1: augmentation_runner silently destroys normalised batches #394: denormalise and renormalise around the augmentation when stats are available,
  raise when the range looks normalised and they are not.
• FIX-2-3: indistinguishability test for candidate selection #408: when the paired CI contains zero, keep the baseline and do not switch candidate.
• FIX-2-4: deprecate the guessed XAI knobs and fix the double-counted accuracy #409: zero the preset selection weights now, 0.x minor with release notes.

---

Status, 2026-08-30

Phase 0 dev items, Phase 1 and Phase 2 are complete. Phase 3 is half done.

phase	state
Phase 0 (dev)	#394 #395 #396 #397 done. #398–#401 open, taken by others
Phase 1	complete: #402 #403 #404 #405
Phase 2	complete: #406 #407 #408 #409
Phase 3	#410 #411 done, #412 open
Phase 4	#413 #414 #415 open
Block 7	#416 #417 #418 open

Shadow mode is live. Since #411 landed, every run with XAI enabled writes shadow_records.jsonl: the saliency statistics and robustness metrics per candidate, with the arm that was kept flagged. That is the calibration set #417 needs, accumulating at no extra GPU cost from runs that were going to happen anyway. The sooner the team's benchmark runs go through a current build, the larger that set is when #417 starts.

Two decisions taken while implementing, both flagged in their PRs and reversible:

• FIX-1-2: diagnosis.py, AttentionRegime and the decision rule #403 gave the decision rule three points the issue underspecified: the robustness gap acts as a shared precondition rather than a discriminator; UNSTRUCTURED confidence is 1 - max(other two); a tie goes to the regime whose weakest clause has the larger margin.
• FIX-1-4: diagnosis thresholds as config with no defaults #405 kept hard_quantile_q on BNNRConfig rather than moving it into DiagnosisConfig, because the metrics it produces are worth watching with no diagnosis configured.

One deviation from an issue's steps: #408 step 5 asks to reuse the paired estimator from benchmarks/stats.py (#398). That file does not exist yet, so the estimator lives in bnnr/training/paired.py and the convention difference (sample-level mean vs seed-level median, and why) is documented. Aligning them is a follow-up once #398 lands.

Also fixed, filed off the back of #396: #421 (DifPresets/ProCAM CPU/GPU parity) is implemented in #429 but not merged — it changes numpy-path colour output and is waiting on a call about which channel order to keep.