[v0.26] Validation hardening — purged chronological splits + the leakage-vs-regime diagnosis

[ieatyoursushi]

v0.26 — prep for v0.3/v0.4 (GPT-thread Phase A adoption)

First milestone after v0.25. Builds honest temporal validation and uses it to answer a question the random-split numbers couldn't: was the v0.1–v0.25 supervised skill real, or partly look-ahead leakage from overlapping 30-day forward labels?

Infrastructure

• Splits/TemporalSplit.cs — chronological TrainTest with a purge/embargo gap (train t ≤ T*−E ⇒ label window ends strictly before the test period) and PurgedFolds (contiguous k-fold, both-side embargo — López de Prado purged k-fold).
• Splits/SplitPolicy.cs — process-wide mode from the CLI (--split=temporal, --embargo=N default 30, --testfrac=F). Default stays stratified-random, so existing commands reproduce v0.25 bit-for-bit. Temporal artifacts write to data/artifacts-mlnet-temporal/.
• Splits/DataSplit.cs — facade dispatching on the policy; the 14 trainer/pipeline call sites renamed mechanically, so partition semantics live in one place.
• 5 TemporalSplitTests; Program.cs also warns when --oracle/--ctrade are passed to non-simulate modes (review finding — silently-ignored flag was a two-arm-workflow footgun).

The finding (the point of the milestone)

Canonical scalarized arm, 80/20 temporal, embargo = 30d:

target · split	ROC-AUC	PR-AUC
soft · random	0.9935	0.8126
soft · temporal	0.9970	0.4585
oracle · random	0.9986	0.9951
oracle · temporal	1.0000	0.9507

ROC-AUC held (rose); PR-AUC collapsed. ROC-AUC is prevalence-insensitive (pure ranking) — its holding means the champions transfer across time with no material ranking leakage. PR-AUC's no-skill floor is the positive rate, and the temporal test period is the cost-basis-aged tail where soft+ prevalence falls 3.93% → 0.22% (~18×). Identical ranking on an 18×-rarer class yields lower AUPRC by construction — in lift over no-skill the temporal model is better (~25× → ~210×).

So the old random-split PR-AUC was not leakage-inflated in its ranking content, but it was a regime-averaged number dominated by the harvest-rich early years. Full derivation, CV table, and prevalence breakdown in DataMemo/ValidationHardening_v026.md.

Gate: pass, with a mandate

Champions survive honest validation on the skill metric. The prevalence crash it exposes is exactly the v0.3 P0 — contributions/rebalancing/the sell-winner trim restore harvestable supply → recover prevalence → recover PR-AUC without touching ranking. Standing methodology change: report ROC-AUC + PR-AUC + test-prevalence together; prefer --split=temporal for any forward-deployment claim.

Decade walk-forward (--testfrac=0.5, test ~2016–2026, oracle+ 0.018%) is running as confirmation and will be appended.

🤖 Generated with Claude Code

[ieatyoursushi]

Decade walk-forward — confirmed, with a sharper conclusion

The --testfrac=0.5 run (train ~2006–2016, test ~2016–2026, 924k test rows) finished, and it lets the analysis do something the 80/20 split couldn't: separate leakage from genuine drift.

GBT test	random	temporal 80/20	decade
soft ROC-AUC	0.9935	0.9970	0.9609
soft PR-AUC	0.8126	0.4585	0.4173
oracle ROC-AUC	0.9986	1.0000	1.0000
oracle PR-AUC	0.9951	0.9507	0.9953

The oracle target is the clean leakage control — it's a deterministic function of current-timestep features, so leakage would show up as its ROC-AUC being inflated under random and dropping under temporal. Instead it's 0.9986 → 1.0000 → 1.0000, flat and perfect. Leakage is ruled out.

The soft ROC-AUC does dip to 0.9609 under the full-decade shift (from ~0.997 under the milder 80/20) — but that's genuine regime-generalization decay, not leakage: a model that never saw 2016–2026 ranks its harvest propensity slightly less sharply. 0.96 is still strong. The decade split's value is precisely that it distinguishes this mild honest drift from the leakage the 80/20 alone couldn't isolate.

Net: leakage ruled out, mild decade-horizon drift quantified, and the PR-AUC collapse remains the prevalence crash (decade test soft+ = 0.33%). Memo and PR body updated; the gate result stands — pass with the v0.3 mandate.