Skip to content

feat(rl): 3-seed confirmation + go/no-go verdict report - #17

Merged
lgbarn merged 2 commits into
mainfrom
issue-5-verdict-report
Jul 12, 2026
Merged

feat(rl): 3-seed confirmation + go/no-go verdict report#17
lgbarn merged 2 commits into
mainfrom
issue-5-verdict-report

Conversation

@lgbarn

@lgbarn lgbarn commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • Negative result, reported honestly: the sweep winner fails every PRD acceptance gate on the holdout, and the verdict is ITERATE — do not pay for a combine on this evidence.
  • Headline numbers (holdout test months 2025-06 -> 2026-06, 13 walk-forward windows): pooled pass rate 0.35% (3/852 attempts) vs the 60% bar; 813 busted-MLL; per-seed 0.7% / 0.0% / 0.4%; and the policy underperforms both no-skill baselines (random-take 11.5%, take-every-signal 5.2%) — the report says so plainly and marks the verdict DISCARD-leaning.
  • New futures_foundation/rl/seed_confirm.py: reruns the logged Optuna winner (from study storage via --storage, or explicit --params) on N seeds with only the seed varying over the same 3m/1m walk-forward protocol the sweep used, on holdout test months the sweep never touched; runs both no-skill baselines through the identical evaluate/summarize seam; scores the three PRD gates (>= 60% pooled pass rate, both baselines strictly beaten, every seed >= 50% with zero blowups).
  • New self-contained verdict report docs/topstep_100k_verdict.html (inline CSS + inline SVG, zero external references): gate scoreboard, bust breakdown, per-symbol attribution, attempt counts, winning knobs, explicit ITERATE verdict with next levers.
  • Tests: tests/test_seed_confirm.py — pure gate/verdict seams with stub runners (no SB3), following the test_optuna_sweep.py pattern.

Closes

Closes #5

Acceptance criteria

  • 3-seed retrain runs from the logged winner configuration with only the seed varying
  • The report states pass/fail against every acceptance gate from the parent PRD (>= 60% pass, baselines beaten, all seeds >= 50% with zero blows)
  • Report is one self-contained HTML file in the repo docs (inline CSS/SVG, no external references)
  • The verdict line is explicit: Ship / Iterate / Discard, with the one-paragraph reason

The real run (evidence)

Winner: Optuna trial 19 of study topstep-100k-v1 (40-trial sweep, 2023-01-02 -> 2025-06-01, 26 walk-forward windows, 10k PPO timesteps per window): activate_r=0.554, trail_atr_k=2.518, dll_penalty=1.769, mll_penalty=0.650.

Confirmation command (run 2026-07-12; winner knobs at 3-decimal precision via --params--storage/--study-name re-derives the full-precision equivalent):

uv run python -m futures_foundation.rl.seed_confirm \
    --data-dir data \
    --params '{"activate_r": 0.554, "trail_atr_k": 2.518, "dll_penalty": 1.769, "mll_penalty": 0.65}' \
    --start 2025-03-01 --end 2026-06-04 \
    --timesteps 10000 --seeds 0 1 2 \
    --out seed_confirm_results.json

Real output (per-run summaries + gate table, from seed_confirm_run.log):

-- policy seed 0 --
windows=13  signals=89814  taken=28369  skipped-while-open=60869
attempts=304  passed=2 (1%)  busted=290 [busted-MLL=290]  timeout=12  median-days-to-pass=3.0
-- policy seed 1 --
windows=13  signals=89814  taken=27872  skipped-while-open=59727
attempts=278  passed=0 (0%)  busted=266 [busted-MLL=266]  timeout=12  median-days-to-pass=n/a
-- policy seed 2 --
windows=13  signals=89814  taken=27205  skipped-while-open=58986
attempts=270  passed=1 (0%)  busted=257 [busted-MLL=257]  timeout=12  median-days-to-pass=6.0
-- baseline random-take --
attempts=87  passed=10 (11%)  busted=64 [busted-MLL=64]  timeout=13  median-days-to-pass=6.0
-- baseline take-every-signal --
attempts=116  passed=6 (5%)  busted=97 [busted-MLL=97]  timeout=13  median-days-to-pass=6.0
== PRD acceptance gates ==
gate 1  pooled pass rate 0.4% (bar 60%, 852 attempts)  FAIL
gate 2  baselines beaten (random-take: no, take-every-signal: no)  FAIL
gate 3  seed stability (seed 0: FAIL, seed 1: FAIL, seed 2: FAIL; 813 busts total)  FAIL
VERDICT: NO-SHIP — Iterate vs Discard is the report's call

Test plan

  • uv run pytest tests/test_seed_confirm.py tests/test_optuna_sweep.py tests/test_multi_combine.py -q — 25 passed, 2 skipped locally
  • Open docs/topstep_100k_verdict.html directly from a clone — renders fully offline, light and dark
  • Smoke the CLI: uv run python -m futures_foundation.rl.seed_confirm --data-dir data --storage sqlite:///optuna_topstep_100k.db --study-name topstep-100k-v1 --symbols NQ ES --start 2025-03-01 --end 2025-07-15 --timesteps 1000 --seeds 0 (minutes)

Notes for reviewer

  • This PR reports a negative result on purpose. All 40 sweep trials scored ~0% combine pass rate (best objective -544; every completed trial busted accounts). The 3-seed confirmation exists to make that verdict solid, not to rescue it; the gates fail and the report says ITERATE with concrete next levers (more PPO timesteps per window, wider reward-shaping ranges, entry-quality filter; the study is resumable so round two extends it).
  • Existing modules untouched: the runner recombines optuna_sweep / multi_combine / pipeline seams. Only new files are added.
  • 3 pre-existing local failures in tests/test_evaluate_overfit.py (pooled-AUC tests) reproduce on the base commit and are environment-only — CI on main is green.
  • The Optuna study DB (optuna_topstep_100k.db) and raw run JSONs stay out of the repo (run artifacts, kept locally); the report embeds all numbers.
  • Gotcha for anyone rerunning this: the data/*.parquet training inputs are gitignored — a fresh clone or a git worktree has no data/ directory. Restore the parquet files (or point --data-dir at a checkout that has them) before running; the report's Reproduction section says the same.

Out of scope

  • Round-two iteration itself (extending the study, longer training, entry filters) — the report's "Next levers" section is the plan of record.
  • Fixing the pre-existing test_evaluate_overfit.py local-env failures.

lgbarn added 2 commits July 12, 2026 11:54
seed_confirm re-derives the logged Optuna winner (winning_config ->
build_strategy), retrains it on N seeds with only the seed varying over
the same walk-forward protocol the sweep used, on a holdout slice the
sweep never touched, runs both no-skill baselines through the identical
evaluate/summarize seam, and scores the three PRD gates (pooled pass
rate >= 60%, both baselines strictly beaten, every seed >= 50% with
zero blowups). CLI prints per-run summaries + gate table and writes the
full result dict as JSON for the verdict report.

Refs #5
Self-contained HTML (inline CSS + inline SVG, zero external references)
reporting the 3-seed confirmation of Optuna trial 19 on the untouched
2025-06 -> 2026-06 holdout: 3/852 attempts passed (0.35%) vs the 60%
bar, 813 busted-MLL, and the policy underperforms both no-skill
baselines (random-take 11.5%, take-every-signal 5.2%). Every PRD gate
fails; verdict ITERATE (explicitly DISCARD-leaning), with next levers:
more PPO timesteps per window, wider reward-shaping ranges, an
entry-quality filter, and extending the resumable study.

Refs #5
@lgbarn
lgbarn marked this pull request as ready for review July 12, 2026 17:59
@lgbarn
lgbarn merged commit 97a3992 into main Jul 12, 2026
2 checks passed
@lgbarn
lgbarn deleted the issue-5-verdict-report branch July 12, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3-seed confirmation + go/no-go verdict report

1 participant