PR-map #20 child: Optuna sweep / seed-confirm gap audit vs John's spec

[lgbarn]

Part of: #20
Type: research
Status: open
Blocked by: #23

Question

Given the decided episode structure, what changes do rl/optuna_sweep.py and rl/seed_confirm.py need to match John's sweep/retrain spec — which levers to search for a signal-gated policy (e.g. probability-threshold floor, sizing budget, trail geometry, reward-shaping coefficients), is the existing "every busted trial ranks below every zero-blow trial" objective equivalent to John's hard zero-blow gate, and what does "hyperparameters as a locked trained contract" require in the bundle?

Context

• rl/optuna_sweep.py (PR feat(rl): Optuna combine sweep — composite objective + median pruning #16): searches activate_r, trail_atr_k, dll_penalty, mll_penalty; composite objective; MedianPruner; resumable SQLite.
• rl/seed_confirm.py (PR feat(rl): 3-seed confirmation + go/no-go verdict report #17): 3-seed retrain, pooled >=60% / beat both baselines / every seed >=50% zero-blow.
• John's doc sections 4-5: few load-bearing levers at a time; zero-blow crushes any blowing recipe; multi-seed full-budget retrain; locked hyperparameter sidecar.

Done when

A durable answer lists the concrete deltas (search space, objective, gates, contract sidecar) with file:line refs, ready for /slice-issues.

[lgbarn]

[lgbarn]

Sweep/retrain deltas for full-combine MC episodes (optuna_sweep.py / seed_confirm.py)

Conclusion

The sweep and confirm modules survive as protocol shells (composite score with the bust floor, resumable study, winner re-derivation, PRD gates, baselines) but their evaluator core is replaced: the walk-forward month-window generator (_window_months / walk_forward_windows) that currently defines both the training set and the eval set gives way to (a) a full-combine CombineEpisodeEnv for training and (b) a frozen manifest of MC-sampled episodes past a fixed leak cutoff for evaluation. The load-bearing search space becomes signal-probability floor, per-trade risk ceiling (cushion fraction), activate_r, trail_atr_k — reward coefficients (pass bonus / blow penalty / speed bonus) should be fixed constants of the episode design, not levers, and dll_penalty/mll_penalty lose their reason to exist once BLOW is a true episode-terminal reward. The current objective already crushes busted trials below all clean trials, but it evaluates exactly one seed per trial and has no evaluation-seed dimension at all, so it is not yet John's "worst evaluation seed" gate; and no policy artifact is ever persisted (PPO returns an in-memory closure), so the "locked trained contract" is built from zero, modeled on the finetune _signal.json sidecar. Blast radius: ~3 new files, ~6 changed files + their tests, roughly 6-8 implementation slices.

Evidence

1. Search-space levers: load-bearing vs confounded

Current space (futures_foundation/rl/optuna_sweep.py:66-73): activate_r [0.5, 1.0], trail_atr_k [0.5, 3.0], dll_penalty [0.0, 2.0], mll_penalty [0.0, 4.0].

Lever	Verdict	Why
Signal-probability floor	Load-bearing — add	It defines the candidate set the policy is even shown (doc section 1: "the signal gates where the policy is allowed to look"). It changes the training episode distribution monotonically and attributably — exactly the entry-selectivity knob the doc's section 4 wants. The untracked combine_bridge.py already sweeps score-threshold x contracts the same way, so there is prior evidence the threshold is the dominant knob. Note: it partially overlaps the learned veto (env.py:151-154), but the floor is a hard pre-filter, cheap and interpretable; the veto learns the residual.
Per-trade risk ceiling (fraction of cushion)	Load-bearing — add	Doc section 2 mandates "a hard ceiling on how much of the cushion a single trade may put at risk". Nothing implements it today: max_size = rules.max_contracts = 10 flat (topstep_zigzag.py:101), independent of remaining cushion. This is the single most zero-blow-relevant lever — it directly bounds per-trade blow probability. Implement as a per-step cap on the take-size action derived from dist_mll x cushion, sweep the fraction.
Trail geometry (activate_r, trail_atr_k)	Load-bearing — keep	Direct, mechanical exit behavior (env.py:124-138), monotone and attributable, already the validated levers of the incumbent sweep.
Reward coefficients (pass bonus, blow penalty, speed bonus)	Confounded — fix, don't sweep	Reward shaping interacts with every other lever and with PPO's own optimization non-monotonically; sweeping it tunes the proxy, not the outcome, and the zero-blow gate is enforced at eval anyway. Doc section 3 treats the PASS(+speed)/BLOW/TIMEOUT reward structure as part of the environment definition, not the recipe. Today pass_bonus (default 2.0, topstep_zigzag.py:99) is a CLI knob in multi_combine.py:260 but is NOT in the sweep space; the blow penalty is a hardcoded -1.0 at ppo.py:63; a speed bonus does not exist. All three become fixed constants of the new episode env.
dll_penalty / mll_penalty	Drop from the space	They exist to proxy bust proximity because the current per-trade episode never sees the terminal outcome (topstep_zigzag.py:145-149). With full-combine episodes the true BLOW terminal reward supersedes them. Caveat: a 30-session episode with sparse terminal reward is a hard credit-assignment problem for PPO, so per-trade proximity shaping may need to survive as fixed-coefficient dense shaping — but as a constant, not a sweep dimension.

Net: 4 levers (signal floor, risk-ceiling fraction, activate_r, trail_atr_k) — consistent with "a few load-bearing levers at a time".

2. Objective vs John's per-seed zero-blow gate — NOT equivalent yet

• The crushing mechanism itself matches: composite_score returns BLOWN_SCORE(-100) - busted on any bust (optuna_sweep.py:59-60) and the trial-level re-application at optuna_sweep.py:133-135 guarantees a bust can never be mean-diluted by clean windows. Zero-blow scores live in [-0.5, 1.0], so every busted trial ranks below every clean one. That part carries over unchanged.
• But each trial runs exactly ONE training seed: seed = base_seed + trial.number (optuna_sweep.py:119). There is no per-trial multi-seed evaluation anywhere in the sweep; multi-seed only appears later in seed_confirm.py (gate 3, seed_confirm.py:65-66). A recipe that happens to be clean on its one derived seed but blows on others wins the sweep and only dies in confirmation — John's "any recipe that blows on the worst evaluation seed is crushed" is enforced one stage too late.
• There is also no evaluation-seed dimension at all today: given the data slice, the held-out attempt sequence is deterministic (rolling windows over fixed months, optuna_sweep.py:211-224). MC episodes make evaluation itself stochastic, which is what makes "worst evaluation seed" meaningful.
• Delta: per trial, evaluate the trained policy on K (2-3) independently seeded draws of the held-out MC episode set (or K policy-training seeds at reduced budget — pick one, the doc's wording points at evaluation seeds) and score the trial by the worst draw, keeping the BLOWN_SCORE floor. The trial.report/MedianPruner streaming (optuna_sweep.py:126-128) survives with "window k" reinterpreted as "eval batch k".

3. "Locked trained contract" — what exists, what's missing

• Exists: the Optuna study SQLite persists {params, seed} per trial and winning_config() re-derives the winner (optuna_sweep.py:162-166); seed_confirm.py --out writes a results JSON whose config block records params, seeds, symbols, slice, window sizes, timesteps (seed_confirm.py:298-307); strategies carry provenance config_dict() (topstep_zigzag.py:103-109, multi_combine.py:66-67) — but nothing in the RL path ever writes it.
• Missing — the policy itself is never persisted. ppo.py:99-103 returns an in-memory prediction closure; grep confirms zero save( / .zip calls under futures_foundation/rl/. There is no RL analog of the finetune produce contract (finetune/produce.py:319-320 writes <base>_signal.json with contract_version: 1.0, input spec, channel names, standardize mu/sd next to the ONNX pair).
• Concretely required (doc sections 5 and 7):
  1. Policy weights artifact — SB3 model.save() zip (or ONNX export of the MlpPolicy) emitted by the confirm step for each seed of the winning ensemble.
  2. Sidecar contract JSON, contract_version'd like the finetune one: locked swept params, training seed(s), obs-vector layout (ctx_dim + 4 position + 3 account + symbol one-hot + which ctx column is the signal probability — multi_combine.py:293-294 prints this layout today but nothing records it), action semantics (Discrete 1+max_size, veto/take/hold/flatten), normalization stats (currently none — no VecNormalize; if one is ever added its stats MUST enter the bundle), env constants (CombineRules, max_days, friction_r, dollars_per_r per symbol, signal floor, risk-ceiling fraction), leak-cutoff date, and eval-manifest hash.
  3. A load-and-parity test: reload bundle, replay a frozen episode, assert action-for-action equality with the training-time policy (doc section 7 calls divergence "the highest-severity class of bug"). None of this exists.

4. Walk-forward machinery survival

• walkforward.py:16-37 (walk_forward_windows) and its multi-symbol lift _window_months (optuna_sweep.py:211-224): retired as the training/eval protocol. MC start-date sampling with a fixed leak cutoff (doc section 3) replaces window slicing. The only afterlife is the gauntlet's "held-out OOS months, month-by-month, zero blows every month" check (doc section 6), where month bucketing of eval episodes is still useful.
• pipeline.py run_walkforward (pipeline.py:162-197): its verdict is per-trade-R based (_agg, monthly-PF>1 gate pipeline.py:49-61) — these metrics don't map to pass/blow rates; retire from this path (it can stay for legacy per-trade strategies). pipeline._episodes (pipeline.py:64-82) survives: per-candidate SingleTradeEnv construction is the inner mechanic of a combine episode.
• robustness.py: multiseed_verdict (robustness.py:24) is superseded by seed_confirm gate 3 (per-seed pass-rate + zero-bust, seed_confirm.py:65-66), which is already the right shape. shuffle_robust (robustness.py:11) survives in spirit but must be rebuilt at episode level: shuffle/zero the signal-probability column and require pass rate to collapse toward the baselines — the existing random_take_policy / take_every_signal_policy (multi_combine.py:174-186) already provide the no-skill floor through the identical seam.
• The biggest structural move: evaluate_account_attempts (multi_combine.py:83-143) — chronological account-wide rollout, fills into simulate_combine, fresh account per attempt — is already 80% of a full-combine episode, but it only runs at evaluation time. Training today uses _EpisodeSamplingEnv, which samples per-trade episodes uniformly at random (ppo.py:47-50) with synthetic session days (trades_per_day, topstep_zigzag.py:131-133). The new CombineEpisodeEnv promotes the evaluate_account_attempts loop into a gymnasium training env: reset() MC-samples (start date, instrument set, starting cushion), step() walks candidates chronologically threading one simulate_combine account, terminal reward on PASS/BLOW/TIMEOUT. Real dates replace trades_per_day; run_state["topstep"]/account_features (topstep_zigzag.py:45-77) survive as the observation plumbing, extended with sampled-cushion normalization.
• Eval set replacing window slicing: a frozen, versioned manifest of MC-sampled episode specs (start_date, instrument, starting_cushion, rng_seed) drawn exclusively from after the leak cutoff — fixed across all trials of a study so trials are comparable (and the pruner is fair), freshly resampled only for the final gauntlet. seed_confirm keeps its current property that its slice postdates the sweep's (sweep default ends 2024-05, confirm default starts 2025-03: optuna_sweep.py:287, seed_confirm.py:235-239) — now expressed as disjoint episode manifests instead of disjoint month ranges.

5. Blast radius

New (~3 files):

1. rl/combine_env.py — CombineEpisodeEnv + MC sampler (start date / instrument / cushion) + terminal reward constants. The one genuinely new hard piece.
2. rl/eval_manifest.py (or folded into combine_env) — leak-cutoff enforcement, frozen held-out episode manifest, episode-level evaluation seam (pass/blow over manifest) reusing summarize_attempts.
3. rl/bundle.py — save/load policy + contract sidecar + parity check (produce.py analog).

Changed (~6 files + tests):

• rl/ppo.py — train over CombineEpisodeEnv; retire _EpisodeSamplingEnv for this path; expose the SB3 model for saving (today only a closure escapes, ppo.py:99-103).
• rl/optuna_sweep.py — new suggest_params space; make_data_evaluator swaps _window_months for the MC manifest; worst-of-K-seeds trial score; composite_score/BLOWN_SCORE/pruner streaming/winning_config/ranked table survive nearly verbatim.
• rl/seed_confirm.py — same three PRD gates re-pointed at MC eval manifests; emits the winner bundle per seed.
• rl/multi_combine.py — evaluate_account_attempts gains cushion parameterization (currently hardcoded fresh TOPSTEP_100K account); SixSymbolTopstepStrategy gains risk-ceiling sizing; baselines unchanged.
• rl/topstep_zigzag.py — synthetic-day shape_reward path retires for training; account_features extended for sampled cushion.
• rl/env.py — accept a per-step size cap (risk ceiling); the signal probability needs no env change (it is a precomputed ctx column per resolved child PR-map #20 child: ONNX signal -> env seam design #21).
• rl/pipeline.py, rl/walkforward.py, rl/robustness.py — demoted to legacy/per-trade path; episode-level shuffle control added on the new seam.
• Tests: test_optuna_sweep.py, test_seed_confirm.py, test_multi_combine.py reworked; new test_combine_env.py, test_bundle.py.

Slice estimate: 6-8 issues, in dependency order: (1) CombineEpisodeEnv + MC sampler + terminal rewards; (2) leak cutoff + frozen eval manifest + episode-level eval/shuffle/baseline seam; (3) risk-ceiling sizing; (4) signal-floor candidate gating (mostly lands with child #21); (5) optuna_sweep rewire (space + MC evaluator + worst-seed gate); (6) seed_confirm rewire; (7) bundle + locked contract + parity test; optionally (8) gauntlet runner (doc section 6).

Consequence

• The sweep/confirm protocol code is a keep: composite scoring, bust-floor ranking, resumable SQLite study, winner re-derivation, PRD gates, and the baseline seam all transfer with small edits. Budget the real work on the episode env, the eval manifest, and the bundle — none of which exist.
• Do NOT carry dll_penalty/mll_penalty into the new space by inertia; decide explicitly whether dense shaping is needed for credit assignment, and if so freeze its coefficients before sweeping anything else, or the four "load-bearing" levers become six confounded ones.
• The worst-seed gate belongs inside the sweep objective now (K eval draws per trial), otherwise the sweep keeps promoting single-seed-lucky recipes that seed_confirm then kills — wasted compute per iteration of the loop.
• The bundle/contract slice is a hard prerequisite for doc section 7 (parity-gated ship) and should not be deferred to "later": every confirm run that produces a SHIP verdict today produces no shippable artifact.

Remaining fog

• "Worst evaluation seed" semantics: K training seeds per trial (expensive: K full PPO trains per trial) vs one train + K seeded eval-episode draws (cheap). The doc's wording supports the cheap reading; John should confirm before slicing 3-seed confirmation + go/no-go verdict report #5.
• Sparse-reward trainability: whether PPO learns at all from PASS/BLOW/TIMEOUT-only rewards over ~30-session episodes at laptop timestep budgets, or dense per-trade shaping must stay. This is an empirical spike, not resolvable by reading code.
• Cushion sampling range: doc section 3 says starting cushion is MC-sampled but gives no distribution; CombineRules has a single fixed start_balance/max_loss. Needs a decision (uniform over [x% .. 100%] of full cushion?).
• Episode data plumbing: data/ currently holds CSV while every RL CLI loads <SYM>_3min.parquet (infra map section 5) — a pre-existing gap the MC sampler will trip over on day one.
• Signal npz join: child PR-map #20 child: ONNX signal -> env seam design #21 resolved the signal as a precomputed npz probability column in ctx features; the exact alignment contract (per-candidate vs per-bar, NaN policy off-candidate) determines whether the signal floor filters in detect_entries or in the sampler.

[lgbarn]

Resolved: sweep/seed-confirm survive as protocol shells; evaluator core swaps to CombineEpisodeEnv + frozen MC episode manifest. Full deltas in the answer above.

[lgbarn]