fix(backtest): freeze cost book, rank on train, refuse live-go

[Trujillofa]

Summary

Surgical quality-bar pass on this repo's existing backtest / WFO / sweep stack. No shared engine, no new strategies, no retunes, no live-go.

Hypothesis was mostly right: src/backtest/engine.py is already the structured simulator. The leaks were around it — mutable costs, search ranking on holdout, inconsistent fees, a broken param sweep, and no adversarial tests for peek / mutation / live flags.

Gaps found

1. Cost book was not frozen. CostProfile was frozen, but BacktestConfig was mutable. test_backtest_atr.py even swapped strategy_classes after engine construction. Search scripts hard-coded fee_rate=0.001 (legacy) while the factory default is 0.0004. run_backtest.py silently used 0.001 on spot when --fee was omitted.
2. Search ranking used the test window. run_config_search.py and run_mtf_search.py sorted by WFO OOS + full-period return. run_wfo_sweep.py never applied param_grid and ranked on test Sharpe.
3. No refuse path for live-go. Backtest scripts already did not place orders, but nothing failed if --live / live_go / promote was smuggled in.
4. Clock drift. Metrics treated unknown timeframes as 1 minute (_TIMEFRAME_MINUTES.get(..., 1)), which would inflate Sharpe. Cost funding used a second map.
5. Manifests dumped every trade into the JSON result payload.
6. Missing adversarial tests for peek, cost mutation, and holdout-swap ranking.

What changed

• Frozen BacktestConfig + engine CostBook snapshot (fee, slip, funding, size). Forced mutation after init does not change fills. Units documented (fractions of notional/price; slip is spread+slip).
• Shared src/backtest/timeframes.py clock contract: bar time is open; unknown TF raises.
• rank_by_selection_score: ranking uses first-train-window metrics only. Test asserts swapping holdout scores does not change order. Wired into config/MTF search printers.
• refuse_live_go on canonical research CLIs. run_wfo_sweep.py now errors: it is not a selection tool.
• Manifests omit the trades list (keep counts/metrics).
• run_backtest.py uses factory cost defaults unless --fee is explicit.
• Docs: docs/BACKTEST_AND_WFO.md (how to run; not a live-go). Pointers in EXPERIMENT_AUTOPILOT.md and RESEARCH_FRAMEWORK.md.
• Tests in tests/test_backtest_quality_bar.py.

What was left alone (and why)

• execution_parity_v2 causality — already closed-bar / next-open; existing tests are solid. Did not change run_backtest.py's default legacy_v1 (reproducibility of old CLI runs). Autopilot already defaults to v2.
• experiment_autopilot gates — single pre-specified config; OOS Sharpe/return is the point of validation. Full-period drawdown/bootstrap left as-is to avoid retuning gate outcomes.
• run_wfo.py — OOS eval of a frozen config, not a ranker. Only added live-go refuse.
• Legacy research scripts (autoresearch.py, mtf_*.py, run_full_backtest.py fee default) — not the canonical path; not rewritten.
• Synthetic TF subset — already raises on unsupported labels.
• No new strategies, no param retunes, no paper→live.

Type of Change

• Bug fix
• New feature
• Refactoring
• Documentation update
• Tests

Testing

• Targeted pytest: backtest / WFO / quality-bar / synthetic (131 passed)
• ruff check + ruff format --check on touched files

uv run pytest tests/test_backtest_quality_bar.py tests/test_backtest.py \
  tests/test_backtest_atr.py tests/test_backtest_cost_defaults.py \
  tests/test_backtest_execution_parity_v2.py tests/test_backtest_foundation.py \
  tests/test_experiment_autopilot.py tests/test_backtest_slippage.py \
  tests/test_backtest_futures.py tests/test_backtest_executor_exit_model.py \
  tests/test_synthetic_eval.py tests/test_mtf_integration.py \
  tests/test_session_liquidity_backtest.py tests/test_basis_premium_backtest.py \
  tests/test_trend_filter_audit.py tests/test_portfolio_backtest.py \
  tests/test_cross_venue_dislocation.py tests/test_short_side_parity_audit.py \
  tests/test_cost_realism_rerun.py -q

  

[Trujillofa]

Retest after merge

Do not retest frozen autopilot / execution_parity_v2 configs. Fills were already next-open; OOS gates were left alone.

A locked run_wfo.py of an already-chosen config is still valid.

Do re-run run_config_search / run_mtf_search if that is how a winner was picked. Those ranked on WFO OOS plus full-period return, and they billed the old 0.001 fee.

Leave autoresearch.py / run_full_backtest.py unless those are still the canonical path.

[Trujillofa]

Review: approve with nits

Verified locally on 446e9ae (fresh worktree): 1342 passed in 33s, ruff check / ruff format --check clean, CI green on all three jobs, mergeable_state: clean.

The core holds up. I traced every cost call site in the engine — _calculate_entry_qty, _cap_fixed_notional, _apply_quantity_step, _open_long / _open_short, the exit path, and funding all read self._cost_book, with no self._config.fee_rate / slippage_pct reads left behind. Frozen BacktestConfig has no remaining mutation site anywhere in src/, scripts/, or tests/ except the adversarial one at tests/test_backtest_quality_bar.py:211. The shared timeframes.py genuinely kills the Sharpe-inflation bug — _TIMEFRAME_MINUTES.get(config.timeframe, 1) was annualizing an unknown label at 525,600 periods/year, and run() now validates the label (plus both MTF legs) before any bars are read. fee_rate=None correctly falls through to REALISTIC_FEE_RATE in the factory, so dropping the 0.001 default in run_backtest.py routes through the single source rather than a second one.

The diagnosis in the description matches what's in the diff — the leaks were around the simulator, not in it.

Findings

1. The live-go refusal is unreachable on 5 of the 6 wired CLIs (scripts/run_backtest.py:94). refuse_live_go runs after parse_args(), so argparse exits 2 first; I reproduced it. Only the env-var arm can actually fire. The safety outcome is unchanged, but the guard and its message are dead code as wired, and this doc's "Passing one is refused" is true only by accident. Inline comment has the fix.
2. The ranking key isn't printed (scripts/run_config_search.py:903, scripts/run_mtf_search.py:570). Both printers sort on selection_sharpe and show every column except that one.
3. passes_gates dropped out of the sort order — same thread. Probably correct (gates read OOS), but it's a triage regression worth making deliberate.
4. One-window selector (docs/BACKTEST_AND_WFO.md:52). Ranking on windows[0] train only trades the leak for a variance problem; averaging train metrics across windows is equally leak-free.
5. scripts/run_wfo_sweep.py should be deleted, not stubbed — nothing imports wfo_sweep or this module's parse_backtest_output. Step 2 of the framework.
6. Two source-text-grep tests are maintenance traps (tests/test_backtest_quality_bar.py:422) — banning timedelta(days= across two large scripts, and asserting exact call-site strings that ruff format could rewrap.
7. Nit: the two search scripts build the same ranking two different ways, and one silently dedupes by name.

Only #1 is worth fixing before merge; the rest are follow-ups.

Cost/comparability note

Backing your own "Retest after merge" comment: spot run_backtest.py moves 0.001 → 0.0004 and both search scripts drop their hard-coded 0.001. Any spot number produced before this lands is not comparable to one produced after — your re-run guidance for run_config_search / run_mtf_search is the right scope.

Also worth knowing before the first search run: _evaluate_candidate now runs an extra backtest per candidate for the selection window, so a sweep of N candidates goes from N×(1+W) runs to N×(2+W). Correct and worth paying — just not free on wall-clock.

---

Generated by Claude Code

[Trujillofa]

Pushed two commits addressing findings 1 and 2 from my review (446e9ae..dbdae12).

3f70958 — make the live-go refusal reachable

Guard raw argv before parse_args(), then re-check parsed flags, across all six CLIs.

While testing this I found a second, unrelated regression on the branch: scripts/run_wfo.py could not start at all. This branch added the first from src... import to that script, but unlike run_backtest.py / run_config_search.py / run_mtf_search.py / run_full_backtest.py, it has no sys.path.append(os.getcwd()) — so it died at import:

$ uv run python scripts/run_wfo.py --help
ModuleNotFoundError: No module named 'src'

It works on main (where it shells out to run_backtest.py and imports nothing from src), so the PR introduced it. Nothing caught it because no test executes these scripts and CI never invokes them. Fixed by appending cwd the way the sibling scripts do; --help and a real run both work now.

Worth knowing this was live for the two days the PR has been open — anyone who ran the canonical WFO entry point off this branch got an import traceback.

New test: test_research_cli_refuses_live_flag_when_actually_invoked runs each of the six CLIs in a real subprocess and asserts the refusal message appears (and that argparse's "unrecognized arguments" does not). I verified it fails on the old ordering and passes on the new one, so it's a real regression test rather than a restatement:

# with the old ordering restored on run_backtest.py
FAILED test_research_cli_refuses_live_flag_when_actually_invoked[scripts/run_backtest.py]
1 failed, 5 passed

Adds ~5s to the suite. That felt worth paying given the existing unit test passed the entire time the guard was dead.

dbdae12 — print the selection score

sel_sharpe= / sel_return= added to both search printers, so the column the rows are sorted by is visible.

Validation

1348 passed in 48s (1342 + 6 new), ruff check . and ruff format --check . clean.

Left for you

Findings 3–7 are untouched, since they're judgement calls rather than defects: whether dropping passes_gates from the sort was intended, the one-window selector, deleting run_wfo_sweep.py outright, the two source-text-grep tests, and the dedupe nit. Happy to take any of them if you want.

---

Generated by Claude Code