Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -547,7 +547,7 @@ loss_limits:
|--------|---------|
| `scripts/run_backtest.py` | Run single backtest |
| `scripts/run_full_backtest.py` | Full parameter backtest |
| `scripts/run_wfo.py` | Walk-forward optimization |
| `scripts/run_wfo.py` | Fixed-config WFO OOS (same clock as `experiment_autopilot`; not optimization) |
| `scripts/smoke_test.py` | Quick connectivity check |
| `scripts/migrate.py` | Apply database migrations |
| `scripts/config_doctor.py` | Validate configuration |
Expand Down
10 changes: 7 additions & 3 deletions docs/BACKTEST_AND_WFO.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,13 @@ Strategies see completed OHLCV for that bar. v2 queues the fill for the next
open. Unknown timeframe labels raise; they are not treated as 1 minute.

WFO test windows are `[start, end)`. `fetch_range` uses `time >= start AND
time <= end`, so canonical autopilot (and the search CLIs) translate `end`
with `wfo_inclusive_fetch_bounds()` before the fetch. Frozen historical
artifacts stay as written; do not rerun them to “fix” old gates.
time <= end`, so canonical autopilot, the search CLIs, `scripts/run_wfo.py`,
`scripts/run_wfo_short_comparison.py`, and `scripts/analyze_entry_overlap.py`
translate `end` with `wfo_inclusive_fetch_bounds()` before the fetch. Frozen
historical artifacts stay as written; do not rerun them to “fix” old gates.

`scripts/run_wfo.py` is a thin fixed-config OOS runner on that clock. It does
not optimize parameters. Gated WFO is `scripts/experiment_autopilot.py`.

## Cost book

Expand Down
2 changes: 1 addition & 1 deletion docs/MATH_MODELS_ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ surface is still unauthorized work.

| Piece | What is already true |
|-------|----------------------|
| Validation | WFO + bootstrap + concentration gates live in `src/backtest/experiment_autopilot.py` (`GateConfig`, `WfoWindow`). Drive with `scripts/experiment_autopilot.py` / `scripts/run_wfo.py`. |
| Validation | WFO + bootstrap + concentration gates live in `src/backtest/experiment_autopilot.py` (`GateConfig`, `WfoWindow`). Drive gated WFO with `scripts/experiment_autopilot.py`. `scripts/run_wfo.py` is a thin fixed-config OOS runner on the same calendar / half-open clock — not parameter optimization. |
| Primary technical stack | `TrendPullbackStrategy` plus MTF (`mtf_breakout`, `mtf_continuation`, `multi_timeframe_regime`, `regime_router`). Production agents on this stack are **paper / disarmed**; several configs emit zero fills at current aggregator thresholds. |
| Engine | `execution_parity_v2` evaluates at bar close and fills at next-bar open (`fill_source="next_bar_open"`). `legacy_v1` remains the signal-close compatibility path. The `scripts/experiment_autopilot.py` CLI defaults to `execution_parity_v2`. Futures qty is step-truncated in `src/backtest/sizing.py`. |
| Costs | `src/backtest/cost_overrides.py` (`CostProfile`: `legacy` / `realistic` / `corrected`). Corrected book: fee `0.0004`, slip `0.0002`, `scaled_8h` funding. Model work must not edit cost overrides. |
Expand Down
11 changes: 9 additions & 2 deletions docs/RESEARCH_FRAMEWORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,17 @@ python scripts/run_config_search.py # gated search; rank on train only
WFO is the primary validation tool. Fixed-window backtests are insufficient alone.

```bash
python scripts/run_wfo.py BTCUSDT 1h 2021-01-01 2022-01-01 --config <config>
python scripts/experiment_autopilot.py \
--config <config> --symbol BTCUSDT --timeframe 1h \
--start 2021-01-01 --end 2022-01-01 \
--train-months 6 --test-months 3 \
--execution-profile execution_parity_v2
# Thin OOS-only runner (same calendar + half-open fetch; no gates):
# python scripts/run_wfo.py BTCUSDT 1h 2021-01-01 2022-01-01 --config <config>
```

Acceptable result: OOS Sharpe ≥ 0.6 across ≥ 3 folds.
Acceptable result: OOS Sharpe ≥ 0.6 across ≥ 3 folds. Use `experiment_autopilot`
for gated WFO. `run_wfo.py` is not parameter optimization.

#### 3b. Parameter stability check

Expand Down
14 changes: 11 additions & 3 deletions scripts/analyze_entry_overlap.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,11 @@
_run_backtest,
)
from scripts.run_autoresearch import _deep_merge, _read_yaml # noqa: E402
from src.backtest.experiment_autopilot import build_wfo_windows # noqa: E402
from src.backtest.experiment_autopilot import ( # noqa: E402
build_wfo_windows,
wfo_inclusive_fetch_bounds,
)
from src.backtest.research_safety import refuse_live_go # noqa: E402
from src.db import close_pool, get_pool, init_pool # noqa: E402
from src.features.reader import IndicatorReader # noqa: E402
from src.main import _resolve_strategy_config, load_settings # noqa: E402
Expand Down Expand Up @@ -237,20 +241,22 @@ async def _collect_oos_entries(
entries: list[datetime] = []
async with reader:
for window in windows:
_, _, test_start, test_end = wfo_inclusive_fetch_bounds(window)
window_config = _build_backtest_config(
settings=settings,
raw_config=raw_config,
symbol=run_symbol,
timeframe=run_timeframe,
start=window.test_start,
end=window.test_end,
start=test_start,
end=test_end,
strategy_classes=strategy_classes,
strategy_configs=strategy_configs,
aggregator_config=aggregator_config,
initial_capital=10000.0,
disable_trend_filter=False,
replay_sentiment_path=spec.replay_sentiment_log,
replay_sentiment_max_age_hours=spec.replay_sentiment_max_age_hours,
execution_profile="execution_parity_v2",
)
result_bt = await _run_backtest(reader, window_config)
for trade in result_bt.trades:
Expand Down Expand Up @@ -388,7 +394,9 @@ def _interpret(report: dict[str, Any]) -> str:


async def main() -> None:
refuse_live_go(argv=sys.argv[1:])
args = parse_args()
refuse_live_go(flags=vars(args))
configure_logger("INFO")
manifest = _load_manifest(Path(args.manifest))
specs = _parse_agents(manifest)
Expand Down
112 changes: 75 additions & 37 deletions scripts/run_wfo.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,66 @@
#!/usr/bin/env python3
"""Walk-Forward Optimization runner."""
"""Fixed-config walk-forward OOS runner.

This is not parameter optimization. Calendar windows, half-open inclusive
fetch bounds, and ``execution_parity_v2`` match
``scripts/experiment_autopilot.py``. Use that script for gated WFO.
Not a live-go.
"""

from __future__ import annotations

import argparse
import asyncio
import csv
import os
import subprocess
import sys
from datetime import datetime, timedelta
from pathlib import Path
from statistics import mean

sys.path.append(os.getcwd())

from src.backtest.research_safety import refuse_live_go
from src.backtest.experiment_autopilot import ( # noqa: E402
WfoWindow,
build_wfo_windows,
wfo_inclusive_fetch_bounds,
)
from src.backtest.research_safety import refuse_live_go # noqa: E402

ExecutionProfile = str


def parse_run_backtest_stdout(stdout: str) -> dict[str, float]:
"""Parse metrics from ``scripts/run_backtest.py`` stdout labels.

``run_backtest.py`` emits ``Sharpe Ratio:``, not ``Sharpe:``. Matching the
short label silently drops Sharpe and later fills 0.0 into every WFO fold.
"""
metrics: dict[str, float] = {}
for line in stdout.splitlines():
stripped = line.strip()
if stripped.startswith("Total Trades:"):
metrics["trades"] = float(int(stripped.split(":", 1)[1].strip()))
elif stripped.startswith("Win Rate:"):
pct = stripped.split(":", 1)[1].strip().rstrip("%")
metrics["win_rate"] = float(pct) / 100.0
elif stripped.startswith("Sharpe Ratio:"):
metrics["sharpe"] = float(stripped.split(":", 1)[1].strip())
return metrics


def oos_fetch_windows(
start: str,
end: str,
train_months: int,
test_months: int,
) -> list[tuple[WfoWindow, str, str]]:
"""Calendar WFO folds with inclusive reader bounds for the OOS test only."""
rows: list[tuple[WfoWindow, str, str]] = []
for window in build_wfo_windows(start, end, train_months, test_months):
_, _, test_start, test_end = wfo_inclusive_fetch_bounds(window)
rows.append((window, test_start, test_end))
return rows


def run_backtest(
Expand All @@ -24,6 +71,7 @@ def run_backtest(
config_path: str,
replay_sentiment_log: str | None = None,
replay_sentiment_max_age_hours: float | None = None,
execution_profile: ExecutionProfile = "execution_parity_v2",
) -> dict[str, float] | None:
cmd = [
sys.executable,
Expand All @@ -38,6 +86,8 @@ def run_backtest(
end,
"--config",
config_path,
"--execution-profile",
execution_profile,
]
if replay_sentiment_log:
cmd.extend(["--replay-sentiment-log", replay_sentiment_log])
Expand All @@ -47,16 +97,7 @@ def run_backtest(
if result.returncode != 0:
print(f"Backtest failed: {result.stderr}")
return None
lines = result.stdout.splitlines()
metrics: dict[str, float] = {}
for line in lines:
if "Total Trades:" in line:
metrics["trades"] = float(int(line.split(":")[1]))
elif "Win Rate:" in line:
metrics["win_rate"] = float(line.split(":")[1].strip("%")) / 100.0
elif "Sharpe:" in line:
metrics["sharpe"] = float(line.split(":")[1])
return metrics
return parse_run_backtest_stdout(result.stdout)


async def wfo(
Expand All @@ -70,53 +111,41 @@ async def wfo(
config_path: str = "config/settings.yaml",
replay_sentiment_log: str | None = None,
replay_sentiment_max_age_hours: float | None = None,
execution_profile: ExecutionProfile = "execution_parity_v2",
) -> list[dict[str, str | float]]:
start_dt = datetime.fromisoformat(start)
end_dt = datetime.fromisoformat(end)
results: list[dict[str, str | float]] = []

current = start_dt
while current + timedelta(days=test_months * 30 + 1) < end_dt:
train_end = current + timedelta(days=train_months * 30)
test_start = train_end
test_end = test_start + timedelta(days=test_months * 30)

if test_end > end_dt:
break

train_str = train_end.strftime("%Y-%m-%d")
test_str = test_end.strftime("%Y-%m-%d")

for window, test_start, test_end in oos_fetch_windows(start, end, train_months, test_months):
print(
f"Train: {current.strftime('%Y-%m')} - {train_str} | Test: {train_end.strftime('%Y-%m')} - {test_str}"
f"Train: {window.train_start} - {window.train_end} | "
f"Test: {window.test_start} - {window.test_end}"
)

metrics = run_backtest(
symbol,
timeframe,
train_str,
test_str,
test_start,
test_end,
config_path,
replay_sentiment_log=replay_sentiment_log,
replay_sentiment_max_age_hours=replay_sentiment_max_age_hours,
execution_profile=execution_profile,
)
if metrics:
results.append(
{
"symbol": symbol,
"timeframe": timeframe,
"train_start_month": current.strftime("%Y-%m"),
"train_end_date": train_str,
"test_start_month": train_end.strftime("%Y-%m"),
"test_end_date": test_str,
"train_start_month": window.train_start[:7],
"train_end_date": window.train_end[:10],
"test_start_month": window.test_start[:7],
"test_end_date": window.test_end[:10],
"trades": metrics.get("trades", 0.0),
"win_rate": metrics.get("win_rate", 0.0),
"sharpe": metrics.get("sharpe", 0.0),
}
)

current = train_end

if results:
sharpe_mean = mean(float(r["sharpe"]) for r in results)
win_rate_mean = mean(float(r["win_rate"]) for r in results)
Expand Down Expand Up @@ -146,7 +175,9 @@ async def wfo(


def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run walk-forward optimization")
parser = argparse.ArgumentParser(
description="Fixed-config walk-forward OOS (same clock as experiment_autopilot)"
)
parser.add_argument("symbol", nargs="?", default="ETHUSDT")
parser.add_argument("timeframe", nargs="?", default="5m")
parser.add_argument("start", nargs="?", default="2023-01-01")
Expand All @@ -158,6 +189,12 @@ def parse_args() -> argparse.Namespace:
"--config",
default=os.getenv("SETTINGS_PATH", "config/settings.yaml"),
)
parser.add_argument(
"--execution-profile",
choices=("legacy_v1", "execution_parity_v2"),
default="execution_parity_v2",
help="Execution semantics; v2 is the canonical WFO default",
)
parser.add_argument(
"--replay-sentiment-log",
type=str,
Expand Down Expand Up @@ -189,5 +226,6 @@ def parse_args() -> argparse.Namespace:
config_path=args.config,
replay_sentiment_log=args.replay_sentiment_log,
replay_sentiment_max_age_hours=args.replay_sentiment_max_age_hours,
execution_profile=args.execution_profile,
)
)
Loading
Loading