feat(rl): 3-seed confirmation + go/no-go verdict report - #17
Merged
Conversation
seed_confirm re-derives the logged Optuna winner (winning_config -> build_strategy), retrains it on N seeds with only the seed varying over the same walk-forward protocol the sweep used, on a holdout slice the sweep never touched, runs both no-skill baselines through the identical evaluate/summarize seam, and scores the three PRD gates (pooled pass rate >= 60%, both baselines strictly beaten, every seed >= 50% with zero blowups). CLI prints per-run summaries + gate table and writes the full result dict as JSON for the verdict report. Refs #5
Self-contained HTML (inline CSS + inline SVG, zero external references) reporting the 3-seed confirmation of Optuna trial 19 on the untouched 2025-06 -> 2026-06 holdout: 3/852 attempts passed (0.35%) vs the 60% bar, 813 busted-MLL, and the policy underperforms both no-skill baselines (random-take 11.5%, take-every-signal 5.2%). Every PRD gate fails; verdict ITERATE (explicitly DISCARD-leaning), with next levers: more PPO timesteps per window, wider reward-shaping ranges, an entry-quality filter, and extending the resumable study. Refs #5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
futures_foundation/rl/seed_confirm.py: reruns the logged Optuna winner (from study storage via--storage, or explicit--params) on N seeds with only the seed varying over the same 3m/1m walk-forward protocol the sweep used, on holdout test months the sweep never touched; runs both no-skill baselines through the identical evaluate/summarize seam; scores the three PRD gates (>= 60% pooled pass rate, both baselines strictly beaten, every seed >= 50% with zero blowups).docs/topstep_100k_verdict.html(inline CSS + inline SVG, zero external references): gate scoreboard, bust breakdown, per-symbol attribution, attempt counts, winning knobs, explicit ITERATE verdict with next levers.tests/test_seed_confirm.py— pure gate/verdict seams with stub runners (no SB3), following thetest_optuna_sweep.pypattern.Closes
Closes #5
Acceptance criteria
The real run (evidence)
Winner: Optuna trial 19 of study
topstep-100k-v1(40-trial sweep, 2023-01-02 -> 2025-06-01, 26 walk-forward windows, 10k PPO timesteps per window):activate_r=0.554, trail_atr_k=2.518, dll_penalty=1.769, mll_penalty=0.650.Confirmation command (run 2026-07-12; winner knobs at 3-decimal precision via
--params—--storage/--study-namere-derives the full-precision equivalent):Real output (per-run summaries + gate table, from
seed_confirm_run.log):Test plan
uv run pytest tests/test_seed_confirm.py tests/test_optuna_sweep.py tests/test_multi_combine.py -q— 25 passed, 2 skipped locallydocs/topstep_100k_verdict.htmldirectly from a clone — renders fully offline, light and darkuv run python -m futures_foundation.rl.seed_confirm --data-dir data --storage sqlite:///optuna_topstep_100k.db --study-name topstep-100k-v1 --symbols NQ ES --start 2025-03-01 --end 2025-07-15 --timesteps 1000 --seeds 0(minutes)Notes for reviewer
optuna_sweep/multi_combine/pipelineseams. Only new files are added.tests/test_evaluate_overfit.py(pooled-AUC tests) reproduce on the base commit and are environment-only — CI on main is green.optuna_topstep_100k.db) and raw run JSONs stay out of the repo (run artifacts, kept locally); the report embeds all numbers.data/*.parquettraining inputs are gitignored — a fresh clone or a git worktree has nodata/directory. Restore the parquet files (or point--data-dirat a checkout that has them) before running; the report's Reproduction section says the same.Out of scope
test_evaluate_overfit.pylocal-env failures.