Track: research · Level: core · Effort: ~6h · Phase: 0 · Depends on: none
Source: T20 findings III.6 (#389)
Why this matters
Our DFR-style baseline reaches 83.4 worst-group accuracy against 92.9 published. The findings
identify four reasons, one of which is an implementation defect in the group-balanced
subsampling, the other three being no l1 regularisation, no averaging over retrains, and a
weaker base extractor. A baseline that is wrong for an avoidable reason makes every
comparison against it softer than it should be.
Steps
- Fix the group-balanced subsampling in
benchmarks/spurious_repair.py.
- State the three remaining known deviations from published DFR in the docstring, rather than
leaving them implicit.
- Re-run the
dfr condition and update findings_waterbirds.md.
Done when
The DFR arm is correct as far as group balancing goes, and every remaining deviation from the
published method is written down.
How to take this: comment "taking this" and wait to be assigned. Branch fix-0-8-dfr-group-balancing from upstream/main, and put Closes #<this issue number> in your PR. Full workflow: the Cohort Handbook (pinned in Discord).
Plan: plans/FIX-PLAN-T20-2026-08-23.md, section 4.
Track: research · Level: core · Effort: ~6h · Phase: 0 · Depends on: none
Source: T20 findings III.6 (#389)
Why this matters
Our DFR-style baseline reaches 83.4 worst-group accuracy against 92.9 published. The findings
identify four reasons, one of which is an implementation defect in the group-balanced
subsampling, the other three being no l1 regularisation, no averaging over retrains, and a
weaker base extractor. A baseline that is wrong for an avoidable reason makes every
comparison against it softer than it should be.
Steps
benchmarks/spurious_repair.py.leaving them implicit.
dfrcondition and updatefindings_waterbirds.md.Done when
The DFR arm is correct as far as group balancing goes, and every remaining deviation from the
published method is written down.
How to take this: comment "taking this" and wait to be assigned. Branch
fix-0-8-dfr-group-balancingfromupstream/main, and putCloses #<this issue number>in your PR. Full workflow: the Cohort Handbook (pinned in Discord).Plan:
plans/FIX-PLAN-T20-2026-08-23.md, section 4.