M52: Crazy Fruits combo curriculum — seeded-specials starts + combo-biased exploration

docs/prd/CRAZY_FRUITS_COMBO_CURRICULUM_PRD.md

@@ -0,0 +1,112 @@
+# Crazy Fruits — combo curriculum (seeded-specials training starts) — PRD
+
+**Status:** 🔜 planned 2026-07-25 (2-agent design verification: code-safety + RL sanity)
+**Owner:** Pieterjan
+**Milestone:** [PLAN.md](PLAN.md) M52 · extends [CRAZY_FRUITS_RANKING_PRD.md](CRAZY_FRUITS_RANKING_PRD.md)
+(M51, shipped; PR #39 merged 2026-07-25) · branch `m52-crazyfruits-combo-curriculum` → master
+
+## 1. Problem
+
+Owner observation after M51 shipped: the net still occasionally skips a 5-in-a-row, and — the sharper
+question — it has plausibly *never experienced* what a wrapped+wrapped or bomb+bomb combo pays. The M51
+per-kind probe quantified the residuals on `cf9train`: combo take-rate **63%** (greedy 92%, specials-greedy
+90% — combos pay 1.5–6.4 reward *immediately*, so unlike creations a high take-rate is genuinely correct
+here), bomb-bucket opportune take **80%**.
+
+What the net already has without any engineering (M51's dense loss + per-action planes): the *deterministic*
+combo payoff is in its INPUT (the planes run `executeCombo` in simulation) and in its dense TARGETS (every
+legal combo swap's Q is supervised toward that payoff even when never played). What it lacks:
+1. **Frequency** — combo-legal states are ~2.8% of natural random-walk states (845/30 000; adjacent
+   bomb+bomb far rarer), so those dense targets get few gradient hits and the dueling trunk barely shapes
+   itself around them.
+2. **The aftermath** — the realized post-combo continuation (a huge random refill after a board wipe) is
+   only learnable by actually PLAYING combos.
+3. **Creating the setup** — deliberately steering two specials together is a multi-move plan a γ=0 reactive
+   net cannot represent; that is the e1→e2 hold-for-combo gap, explicitly OUT OF SCOPE here (reserved for
+   the search-guided lever, RANKING PRD lever C).
+
+## 2. Design — seed the situations, keep the yardstick honest
+
+Two levers, per the M52.0 review (which also killed one idea — see Rejected):
+
+**Lever 1 (primary) — combo-biased ε-exploration, `CrazyFruitsEnv.ComboExploreBias` (q, default 0).** When an
+ε-exploration step fires and a legal special+special swap exists, pick uniformly among the combo swaps with
+probability q instead of uniformly over all legal swaps. This buys realized post-combo experience (the piece
+dense targets can't supply) with ZERO distribution shift — the boards stay natural. Trainer seam:
+`DqnOptions.ExploreBias` (`Func<rng,int>`, −1 = no suggestion), consulted only inside the ε-branch; null ⇒
+zero extra RNG draws, so every other game stays bitwise-identical.
+
+**Lever 2 — seeded-specials training starts, `CrazyFruitsEnv.SeedSpecialsProb` (p, default 0 — train env
+only; the eval env and every gate stay on the natural distribution).** On `Reset` (seeded and autoreset paths
+alike), with probability p the fresh board is dealt *combo-ready*:
+- one **adjacent special pair** (a special+special swap is legal immediately) — ADJACENT ONLY: a near pair
+  one swap apart was considered and rejected, because at γ=0 the bring-together swap pays ~0 immediately and
+  the oracle prices only immediate firing, so its training label is flat — such states teach nothing (the
+  cross-move setup skill belongs to the search-distillation lever, RANKING PRD lever C);
+- plus up to 2 extra singles on random plain cells;
+- kinds drawn uniformly over {stripedH, stripedV, wrapped, bomb}.
+This lever exists specifically to manufacture the states ε-bias can't reach naturally (adjacent bomb+bomb).
+
+**Injection is invariant-safe by construction:** a cell's packed value is overwritten keeping its fruit type
+(striped/wrapped) — typewise match structure unchanged, so the dealt board's no-instant-match and
+has-legal-swap guarantees survive; a bomb (type 0) never joins runs, and bomb swaps are always legal, so
+legality only grows. All randomness comes from the env's own RNG stream (`_rng`), never the engine's refill
+stream — seeded runs stay deterministic, and the unseeded autoreset path continues the same stream.
+
+**Why this is sound off-policy:** DQN learns from the replay distribution, not the on-policy one, and at γ=0
+every update is a pure per-(s,a) regression — there is no bootstrap through which rare-state value error can
+propagate. Seeding shifts *which states* get gradient, nothing else. The natural-board 500-ep score gate
+guards against forgetting the common case.
+
+**Checkpoint reuse (owner question — YES, upgraded to a full RESUME):** the observation is unchanged (1040),
+and the M52.0 review flagged warm-start's two weaknesses (cold Adam moments, empty replay buffer) — both of
+which a full training-state resume solves for free: copy BOTH `crazyfruits.dqn.ckpt` + `crazyfruits.dqn-state.ckpt`
+into the new data dir and raise `--steps` to the absolute 800k. The run continues with cf9train's optimizer
+state, its 100k natural-board replay buffer (new seeded transitions blend in gradually — the reviewer's
+pre-fill concern mooted), ε already at its 0.05 floor (the reviewer's ≤0.1 recommendation), and keep-best
+seeds its baseline from the resumed net's own eval — the deployable net is only overwritten by something
+that BEATS it. The curriculum options live on the env instance, not in the state file, so the resumed run
+picks them up on the next reset.
+
+**Rejected for v1:** ε-exploration biased toward combo swaps (adds an on-policy knob the seeding may make
+unnecessary; registered as the escalation if gates miss) · prioritized replay for seeded episodes (same
+reason) · mid-episode injection (Reset-only keeps the invariant argument trivial).
+
+## 3. Milestones
+
+- **M52.0 — Design verification.** ✅ (2026-07-25) 2-agent check. Code-safety verdicts: injection
+  invariant-safe (kind overwrite keeps the fruit type ⇒ typewise match structure byte-identical; a bomb
+  never joins runs and its swaps are always legal — note the precise invariant is "≥1 legal swap
+  guaranteed", not "legality only grows", since a bomb can remove a match-based swap elsewhere); reshuffle
+  with seeded specials is bounded (Fisher-Yates keeps the multiset; 1000-cap → plain re-deal); env-stream
+  RNG keeps per-seed determinism; save/restore round-trips packed specials untouched; hard requirement =
+  option defaults OFF (all existing env tests build default envs). RL verdicts: at γ=0 there is NO bootstrap
+  propagation path — curriculum risk reduces to benign covariate shift with identical labels; **p = 0.25**
+  (fallback 0.1); **adjacent pairs only** (rejected near-pairs: the bring-together swap's γ=0 label is flat);
+  **combo-biased ε is the PRIMARY lever** (q, realized combo experience at zero distribution shift); no PER;
+  prefer full resume over warm-start (Adam moments + natural replay carry over).
+- **M52.1 — Env + trainer seam + Lab.** ✅ (2026-07-25) `SeedSpecialsProb` (facade-level `GridSnapshot` →
+  modify → `LoadGrid` on Reset) + `ComboExploreBias`/`SuggestComboExploration` on the env; generic
+  `DqnOptions.ExploreBias` hook consulted only inside the ε-branch (null ⇒ zero extra RNG draws ⇒ every
+  other game bitwise-identical); campaign wires the hook from the injected train env; Lab
+  `--seed-specials`/`--combo-explore`; per-kind created counts in `--baselines` (gate 5's instrument).
+  **Gate:** 67/67 green — 4 new tests: seeded invariants + adjacent pair + legal combo swap, per-seed
+  determinism + default-env-stays-plain, suggestion hook legality, agent bias redirect.
+- **M52.2 — Spike.** ✅ (2026-07-25) Scratch-dir resume of cf9train +10k curriculum steps: resumed at 400k,
+  keep-best baseline 6014 armed, loss 0.082 (normal band), 20-ep eval 5012 (inside cf9train's own eval noise
+  band — no collapse).
+- **M52.3 — Train (`cf10train`).** 🔄 (launched 2026-07-25) Full resume of cf9train (both ckpt files) →
+  `--steps 800000` absolute = **400k new steps** (owner choice: reuse ckpt + full budget), `--seed-specials
+  0.25 --combo-explore 0.5 --gamma 0 --dense --seed 1`.
+  **Gates (pre-registered):** (1) 500-ep natural-board score NOT CI-separated below `cf9train`'s 5666.4;
+  (2) probe (1000 eps, natural boards): combo take-rate ≥ 75% (from 63%); (3) bomb-bucket opportune ≥ 85%
+  (from 80%); (4) all M50.3 bars stay green (≥+30%/random · gap-share ≥64% · created ≥7.3 / fired ≥5.6);
+  (5) per-kind created non-regression vs cf9train, ESPECIALLY bomb-created (free seeded specials are exactly
+  the pressure to under-create the rarest kind). Keep-best ships only a net that beats the resumed net's
+  eval. Escalation if (2)/(3) miss: raise q, one run, then stop-loss.
+- **M52.4 — Ship + PR.** Best net → `wwwroot/models/crazyfruits.dqn.ckpt` (drop-in, no web change needed);
+  docs/memory synced; PR → master.
+
+## 4. Results
+
+*(to be filled at each gate)*

docs/prd/PLAN.md

@@ -2035,6 +2035,37 @@ reward and the input plane; combo shaping would double-count (shape what pays la
 - **M51.4 — Ship.** ✅ `cf9train` → `wwwroot/models/crazyfruits.dqn.ckpt`; round-over bar ~4 000 → ~5 650;
   parity pin 481681208 unchanged; docs synced.
 
+## M52 — Crazy Fruits combo curriculum  *(2026-07-25; see `CRAZY_FRUITS_COMBO_CURRICULUM_PRD.md`; branch `m52-crazyfruits-combo-curriculum`)* 🔄
+
+**Why:** owner's post-M51 question — the net has plausibly never *experienced* wrapped+wrapped or bomb+bomb.
+Half-true: the deterministic combo payoff is in its input planes AND its dense targets (combo take already
+63% vs random's 10% with zero engineering); what's missing is frequency (combo-legal states ≈2.8% of natural
+play; adjacent bomb+bomb ≈ never) and the realized post-combo refill continuation, which only playing them
+teaches. Deliberately *setting up* combos stays out of scope (γ=0 can't represent it — that's the
+search-distillation lever). 2-agent design verification (code-safety + RL sanity) before build.
+
+- **M52.0 — Design verification.** ✅ Injection invariant-safe (kind overwrite keeps fruit type ⇒ typewise
+  match structure unchanged; bombs never join runs and self-guarantee legality; reshuffle bounded);
+  env-stream RNG keeps determinism; save/restore untouched. RL verdicts: γ=0 has NO bootstrap propagation
+  path, so curriculum risk reduces to benign covariate shift (labels identical on both distributions) —
+  p=0.25; **adjacent pairs only** (a near pair's bring-together swap has a flat γ=0 label — teaches nothing);
+  **combo-biased ε is the PRIMARY lever** (realized combo experience, zero distribution shift); prefer full
+  RESUME of cf9train (keeps Adam moments + the 100k natural replay buffer) over warm-start.
+- **M52.1 — Env + trainer seam + Lab.** ✅ `CrazyFruitsEnv.SeedSpecialsProb` (adjacent pair + ≤2 singles on
+  Reset, env RNG stream, default OFF — eval env stays natural) + `ComboExploreBias` /
+  `SuggestComboExploration`; generic `DqnOptions.ExploreBias` hook consulted only in the ε-branch (null ⇒
+  zero extra draws ⇒ every other game bitwise-identical); Lab `--seed-specials` / `--combo-explore`;
+  per-kind created counts in `--baselines`. **Gate:** 67/67 tests green (4 new: seeded invariants +
+  determinism + combo suggestion + agent bias redirect).
+- **M52.2 — Spike.** ✅ Resume at 400k confirmed, keep-best baseline 6014 armed, 10k curriculum steps, loss
+  normal, no eval collapse.
+- **M52.3 — Train `cf10train`.** 🔄 Resume cf9train → 800k absolute (400k new; owner choice: reuse ckpt +
+  full budget), p=0.25, q=0.5. **Gates (pre-registered):** (1) 500-ep natural score not CI-separated below
+  cf9train's 5666.4; (2) probe combo take ≥75% (from 63%); (3) bomb-bucket opportune ≥85% (from 80%);
+  (4) M50.3 bars stay green; (5) per-kind created non-regression, especially bomb-created. Escalation if
+  (2)/(3) miss: raise q, one run, stop-loss.
+- **M52.4 — Ship + PR.** Best net → wwwroot ckpt (drop-in); docs/memory synced; PR → master.
+
 **Rejected up front:** bigger shaping bonuses (reward-hack trap), combo shaping (double-count), γ>0 schedules
 (n=2 losses: M49 γ=0.99, M50.3 `cf8train`), regret-prioritized replay (subsumed by dense regression).
 

src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs

@@ -48,6 +48,8 @@ public sealed partial class CrazyFruitsDqnCampaign : DqnScoreCampaign
         EvalEpisodes = 20,
         DenseTargets = Typed.DenseRegression ? DenseTargetsFromObservation : null,
         DenseTargetWeight = Typed.DenseTargetWeight,
+        // Combo-biased ε (M52): the injected TRAIN env carries the knob; wire its suggestion hook through.
+        ExploreBias = TrainEnv is CrazyFruitsEnv { ComboExploreBias: > 0 } cf ? cf.SuggestComboExploration : null,
     };
 
     // The observation's shaped per-action plane (deterministicValueShaped(a)/300, 0 = illegal — a legal swap

src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs

@@ -63,6 +63,12 @@ public sealed record DqnOptions
     /// reward's refill-expectation signal 30:1.</summary>
     public float DenseTargetWeight { get; init; } = 1.0f;
 
+    /// <summary>Optional exploration bias, consulted only when an ε-exploration step fires: return a LEGAL
+    /// action to play it, −1 to fall back to the uniform-random legal pick. Null (default) draws nothing
+    /// extra from the policy RNG, so existing runs stay bitwise-identical. Crazy Fruits routes rare
+    /// combo swaps through this (COMBO_CURRICULUM PRD, M52).</summary>
+    public Func<Xoshiro256StarStar, int>? ExploreBias { get; init; }
+
     public int EvalEvery { get; init; } = 5_000;
     public int EvalEpisodes { get; init; } = 20;
 
@@ -76,8 +82,11 @@ public sealed record DqnOptions
 
 public sealed record DqnResult(GreedyQAgent Agent, IValueNet Network, int StepsTrained, double FinalEvalReturn, DqnTrainingState State);
 
-/// <summary>Acts greedily from a Q-network (evaluation/playback); epsilon-greedy when given an RNG.</summary>
-public sealed class GreedyQAgent(IValueNet network, int actionCount, Xoshiro256StarStar? rng = null) : IAgent<float[], int>
+/// <summary>Acts greedily from a Q-network (evaluation/playback); epsilon-greedy when given an RNG. An
+/// optional <paramref name="exploreBias"/> (see <see cref="DqnOptions.ExploreBias"/>) can redirect an
+/// exploration step to a targeted legal action.</summary>
+public sealed class GreedyQAgent(IValueNet network, int actionCount, Xoshiro256StarStar? rng = null,
+    Func<Xoshiro256StarStar, int>? exploreBias = null) : IAgent<float[], int>
 {
     public double Epsilon { get; set; }
 
@@ -95,6 +104,8 @@ public int Act(float[] observation, bool[]? mask, bool greedy = false)
     {
         if (!greedy && rng is not null && rng.NextDouble() < Epsilon)
         {
+            int biased = exploreBias?.Invoke(rng) ?? -1;
+            if (biased >= 0) return biased;
             if (mask is null) return rng.NextInt(actionCount);
             int legal = mask.Count(m => m);
             int pick = rng.NextInt(legal);
@@ -201,7 +212,7 @@ IValueNet WarmStart(IValueNet w) => w.InputSize == obsDim ? w
             }
         }
 
-        var agent = new GreedyQAgent(state.Online, actionCount, state.PolicyRng);
+        var agent = new GreedyQAgent(state.Online, actionCount, state.PolicyRng, options.ExploreBias);
         var accumulator = state.Accumulator!; // set above for both fresh and resumed runs
         var maskProvider = env as IActionMaskProvider;
         float lastLoss = state.LastLoss;

src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs

@@ -100,6 +100,20 @@ public CrazyFruitsEnv(int moveBudget = 30)
     public float WrappedShaping { get; set; } = 60f;
     public float BombShaping { get; set; } = 100f;
 
+    // ── Combo curriculum (COMBO_CURRICULUM PRD, M52 — train env only; both default OFF) ─────────────────────
+
+    /// <summary>Probability that a fresh episode's board is dealt combo-ready: one ADJACENT special pair plus
+    /// up to two singles, injected by overwriting plain cells' kinds (fruit types unchanged, so the deal's
+    /// no-instant-match and has-legal-swap guarantees survive; a bomb never joins runs and its swaps are
+    /// always legal). Draws come from the env RNG stream, never the board's refill stream — seeded runs stay
+    /// deterministic, and the default 0 leaves every draw count untouched.</summary>
+    public double SeedSpecialsProb { get; set; }
+
+    /// <summary>Probability that an ε-exploration step with a legal special+special swap available picks
+    /// uniformly among the combo swaps (the <c>DqnOptions.ExploreBias</c> hook,
+    /// <see cref="SuggestComboExploration"/>) — realized combo experience on natural boards.</summary>
+    public double ComboExploreBias { get; set; }
+
     /// <summary>The §3.6 escalation's shaping (use INSTEAD of <see cref="ShapeCreationRewards"/>, with γ&gt;0):
     /// potential-based, Φ(s) = Σ option value of on-board specials (same 40/60/100 weights), reward +=
     /// γ·Φ(s′) − Φ(s). Policy-invariant when <see cref="PotentialGamma"/> matches the learner's γ — it prices
@@ -128,11 +142,58 @@ private float Potential()
         // Each episode's board deals from the env RNG stream: reproducible under an explicit seed, fresh
         // boards across unseeded resets.
         _board.Reset(_rng.NextUInt64());
+        if (SeedSpecialsProb > 0 && _rng.NextDouble() < SeedSpecialsProb)
+            SeedSpecials();
         _moves = 0;
         _done = false;
         return (_board.BuildObservation(), EnvInfo.Empty);
     }
 
+    /// <summary>Deal the fresh board combo-ready (see <see cref="SeedSpecialsProb"/>): an adjacent special
+    /// pair plus up to two singles, all on plain cells with their fruit type kept.</summary>
+    private void SeedSpecials()
+    {
+        var grid = _board.GridSnapshot();
+        bool horizontal = _rng.NextDouble() < 0.5;
+        int r = _rng.NextInt(horizontal ? CrazyFruitsBoard.Size : CrazyFruitsBoard.Size - 1);
+        int c = _rng.NextInt(horizontal ? CrazyFruitsBoard.Size - 1 : CrazyFruitsBoard.Size);
+        int pairA = r * CrazyFruitsBoard.Size + c;
+        int pairB = horizontal ? pairA + 1 : pairA + CrazyFruitsBoard.Size;
+        grid[pairA] = Seeded(grid[pairA]);
+        grid[pairB] = Seeded(grid[pairB]);
+        int extras = _rng.NextInt(3);
+        for (int i = 0; i < extras; i++)
+        {
+            int cell = _rng.NextInt(CrazyFruitsBoard.Cells);
+            if (grid[cell] < 16) grid[cell] = Seeded(grid[cell]); // plain cells only
+        }
+        _board.LoadGrid(grid);
+    }
+
+    // Kinds 1..4 uniform (stripedH · stripedV · wrapped · bomb); striped/wrapped keep the cell's fruit type
+    // (typewise match structure unchanged), the bomb is colorless.
+    private int Seeded(int packedPlain)
+    {
+        int kind = 1 + _rng.NextInt(4);
+        return kind == 4 ? 4 * 16 : kind * 16 + packedPlain % 16;
+    }
+
+    /// <summary>The <c>DqnOptions.ExploreBias</c> hook (see <see cref="ComboExploreBias"/>): −1 when the
+    /// roll passes or no special+special swap is legal, else a uniform pick among the legal combo swaps.</summary>
+    public int SuggestComboExploration(Xoshiro256StarStar rng)
+    {
+        if (rng.NextDouble() >= ComboExploreBias) return -1;
+        var mask = _board.LegalMask();
+        var combos = new List<int>();
+        for (int a = 0; a < ActionCount; a++)
+        {
+            if (!mask[a]) continue;
+            var (cellA, cellB) = _board.SwapCells(a);
+            if (_board.Kind(cellA) > 0 && _board.Kind(cellB) > 0) combos.Add(a);
+        }
+        return combos.Count == 0 ? -1 : combos[rng.NextInt(combos.Count)];
+    }
+
     public StepResult<float[]> Step(int action)
     {
         if (_done)