Walk-Forward Leakage Auditor

A Codex/Claude Code skill for reviewing time-based ML evaluations before trusting their results.

Use it for:

• walk-forward backtests
• rolling retraining
• time-series forecasting
• chronological train/test splits
• trading, demand, weather, operations, recommender, and other past-to-future ML workflows

This is intentionally just a skill/checklist. It gives Codex or Claude Code a structured review process for reading the relevant code, mapping the temporal contract, identifying leakage/faulty logic risks, and recommending concrete guards or fixes. There is no separate command-line auditor.

Contents

SKILL.md                      Skill instructions and full audit checklist
agents/openai.yaml            Optional Codex UI metadata
examples/                     Runnable public-data demo workloads
EXAMPLE.md                    Worked examples showing skill-guided review
requirements.txt              Example dependencies

Install

For the runnable examples:

pip install -r requirements.txt

To install as a Codex skill, copy this directory into your Codex skills folder or invoke it directly by path. In Claude Code, point Claude at SKILL.md and ask it to use the file as the audit checklist for a walk-forward leakage review.

Skill Workflow

The skill asks the coding agent to:

1. State the prediction-time contract.
2. Trace data from raw inputs through features, labels, splits, transforms, models, predictions, and metrics.
3. Inspect split logic, feature timing, fitted-transform scope, label construction, simulation logic, and experiment comparability.
4. Produce concrete findings with file/line references.
5. Require a failable guard or corrected rerun before trusting the metric.

What It Checks

High-risk patterns include:

• random train/test splits in temporal work
• shuffle=True
• standard K-fold CV without time-aware blocking
• negative shifts exposed as features
• forward as-of joins
• backfill imputation
• random row sampling
• selecting best settings from test/OOS results

Other common audit targets include:

• fitted preprocessing before splitting
• rolling/expanding features that are not lagged
• target-like names in feature lists
• global normalization statistics
• mismatched dataset/date range/capital/cost settings across experiments

Worked Examples

See EXAMPLE.md. The examples download public datasets at runtime and show:

1. A random split that looks acceptable until the skill checklist catches the split policy.
2. A future lead feature that improves holdout accuracy but is unavailable at prediction time.
3. A future backfill imputation step that looks like ordinary cleaning but violates point-in-time rules.

Design Notes

The workflow is:

claim -> skill-guided audit -> finding -> corrected rerun or guard -> conclusion

The skill cannot prove every evaluation is valid. Some leakage depends on publication lag, entity membership, execution assumptions, or domain-specific availability rules. The point is to force those assumptions into the review before results are shipped.