diff --git a/docs/prd/CRAZY_FRUITS_COMBO_CURRICULUM_PRD.md b/docs/prd/CRAZY_FRUITS_COMBO_CURRICULUM_PRD.md new file mode 100644 index 0000000..2bfa8c0 --- /dev/null +++ b/docs/prd/CRAZY_FRUITS_COMBO_CURRICULUM_PRD.md @@ -0,0 +1,112 @@ +# Crazy Fruits โ€” combo curriculum (seeded-specials training starts) โ€” PRD + +**Status:** ๐Ÿ”œ planned 2026-07-25 (2-agent design verification: code-safety + RL sanity) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M52 ยท extends [CRAZY_FRUITS_RANKING_PRD.md](CRAZY_FRUITS_RANKING_PRD.md) +(M51, shipped; PR #39 merged 2026-07-25) ยท branch `m52-crazyfruits-combo-curriculum` โ†’ master + +## 1. Problem + +Owner observation after M51 shipped: the net still occasionally skips a 5-in-a-row, and โ€” the sharper +question โ€” it has plausibly *never experienced* what a wrapped+wrapped or bomb+bomb combo pays. The M51 +per-kind probe quantified the residuals on `cf9train`: combo take-rate **63%** (greedy 92%, specials-greedy +90% โ€” combos pay 1.5โ€“6.4 reward *immediately*, so unlike creations a high take-rate is genuinely correct +here), bomb-bucket opportune take **80%**. + +What the net already has without any engineering (M51's dense loss + per-action planes): the *deterministic* +combo payoff is in its INPUT (the planes run `executeCombo` in simulation) and in its dense TARGETS (every +legal combo swap's Q is supervised toward that payoff even when never played). What it lacks: +1. **Frequency** โ€” combo-legal states are ~2.8% of natural random-walk states (845/30 000; adjacent + bomb+bomb far rarer), so those dense targets get few gradient hits and the dueling trunk barely shapes + itself around them. +2. **The aftermath** โ€” the realized post-combo continuation (a huge random refill after a board wipe) is + only learnable by actually PLAYING combos. +3. **Creating the setup** โ€” deliberately steering two specials together is a multi-move plan a ฮณ=0 reactive + net cannot represent; that is the e1โ†’e2 hold-for-combo gap, explicitly OUT OF SCOPE here (reserved for + the search-guided lever, RANKING PRD lever C). + +## 2. Design โ€” seed the situations, keep the yardstick honest + +Two levers, per the M52.0 review (which also killed one idea โ€” see Rejected): + +**Lever 1 (primary) โ€” combo-biased ฮต-exploration, `CrazyFruitsEnv.ComboExploreBias` (q, default 0).** When an +ฮต-exploration step fires and a legal special+special swap exists, pick uniformly among the combo swaps with +probability q instead of uniformly over all legal swaps. This buys realized post-combo experience (the piece +dense targets can't supply) with ZERO distribution shift โ€” the boards stay natural. Trainer seam: +`DqnOptions.ExploreBias` (`Func`, โˆ’1 = no suggestion), consulted only inside the ฮต-branch; null โ‡’ +zero extra RNG draws, so every other game stays bitwise-identical. + +**Lever 2 โ€” seeded-specials training starts, `CrazyFruitsEnv.SeedSpecialsProb` (p, default 0 โ€” train env +only; the eval env and every gate stay on the natural distribution).** On `Reset` (seeded and autoreset paths +alike), with probability p the fresh board is dealt *combo-ready*: +- one **adjacent special pair** (a special+special swap is legal immediately) โ€” ADJACENT ONLY: a near pair + one swap apart was considered and rejected, because at ฮณ=0 the bring-together swap pays ~0 immediately and + the oracle prices only immediate firing, so its training label is flat โ€” such states teach nothing (the + cross-move setup skill belongs to the search-distillation lever, RANKING PRD lever C); +- plus up to 2 extra singles on random plain cells; +- kinds drawn uniformly over {stripedH, stripedV, wrapped, bomb}. +This lever exists specifically to manufacture the states ฮต-bias can't reach naturally (adjacent bomb+bomb). + +**Injection is invariant-safe by construction:** a cell's packed value is overwritten keeping its fruit type +(striped/wrapped) โ€” typewise match structure unchanged, so the dealt board's no-instant-match and +has-legal-swap guarantees survive; a bomb (type 0) never joins runs, and bomb swaps are always legal, so +legality only grows. All randomness comes from the env's own RNG stream (`_rng`), never the engine's refill +stream โ€” seeded runs stay deterministic, and the unseeded autoreset path continues the same stream. + +**Why this is sound off-policy:** DQN learns from the replay distribution, not the on-policy one, and at ฮณ=0 +every update is a pure per-(s,a) regression โ€” there is no bootstrap through which rare-state value error can +propagate. Seeding shifts *which states* get gradient, nothing else. The natural-board 500-ep score gate +guards against forgetting the common case. + +**Checkpoint reuse (owner question โ€” YES, upgraded to a full RESUME):** the observation is unchanged (1040), +and the M52.0 review flagged warm-start's two weaknesses (cold Adam moments, empty replay buffer) โ€” both of +which a full training-state resume solves for free: copy BOTH `crazyfruits.dqn.ckpt` + `crazyfruits.dqn-state.ckpt` +into the new data dir and raise `--steps` to the absolute 800k. The run continues with cf9train's optimizer +state, its 100k natural-board replay buffer (new seeded transitions blend in gradually โ€” the reviewer's +pre-fill concern mooted), ฮต already at its 0.05 floor (the reviewer's โ‰ค0.1 recommendation), and keep-best +seeds its baseline from the resumed net's own eval โ€” the deployable net is only overwritten by something +that BEATS it. The curriculum options live on the env instance, not in the state file, so the resumed run +picks them up on the next reset. + +**Rejected for v1:** ฮต-exploration biased toward combo swaps (adds an on-policy knob the seeding may make +unnecessary; registered as the escalation if gates miss) ยท prioritized replay for seeded episodes (same +reason) ยท mid-episode injection (Reset-only keeps the invariant argument trivial). + +## 3. Milestones + +- **M52.0 โ€” Design verification.** โœ… (2026-07-25) 2-agent check. Code-safety verdicts: injection + invariant-safe (kind overwrite keeps the fruit type โ‡’ typewise match structure byte-identical; a bomb + never joins runs and its swaps are always legal โ€” note the precise invariant is "โ‰ฅ1 legal swap + guaranteed", not "legality only grows", since a bomb can remove a match-based swap elsewhere); reshuffle + with seeded specials is bounded (Fisher-Yates keeps the multiset; 1000-cap โ†’ plain re-deal); env-stream + RNG keeps per-seed determinism; save/restore round-trips packed specials untouched; hard requirement = + option defaults OFF (all existing env tests build default envs). RL verdicts: at ฮณ=0 there is NO bootstrap + propagation path โ€” curriculum risk reduces to benign covariate shift with identical labels; **p = 0.25** + (fallback 0.1); **adjacent pairs only** (rejected near-pairs: the bring-together swap's ฮณ=0 label is flat); + **combo-biased ฮต is the PRIMARY lever** (q, realized combo experience at zero distribution shift); no PER; + prefer full resume over warm-start (Adam moments + natural replay carry over). +- **M52.1 โ€” Env + trainer seam + Lab.** โœ… (2026-07-25) `SeedSpecialsProb` (facade-level `GridSnapshot` โ†’ + modify โ†’ `LoadGrid` on Reset) + `ComboExploreBias`/`SuggestComboExploration` on the env; generic + `DqnOptions.ExploreBias` hook consulted only inside the ฮต-branch (null โ‡’ zero extra RNG draws โ‡’ every + other game bitwise-identical); campaign wires the hook from the injected train env; Lab + `--seed-specials`/`--combo-explore`; per-kind created counts in `--baselines` (gate 5's instrument). + **Gate:** 67/67 green โ€” 4 new tests: seeded invariants + adjacent pair + legal combo swap, per-seed + determinism + default-env-stays-plain, suggestion hook legality, agent bias redirect. +- **M52.2 โ€” Spike.** โœ… (2026-07-25) Scratch-dir resume of cf9train +10k curriculum steps: resumed at 400k, + keep-best baseline 6014 armed, loss 0.082 (normal band), 20-ep eval 5012 (inside cf9train's own eval noise + band โ€” no collapse). +- **M52.3 โ€” Train (`cf10train`).** ๐Ÿ”„ (launched 2026-07-25) Full resume of cf9train (both ckpt files) โ†’ + `--steps 800000` absolute = **400k new steps** (owner choice: reuse ckpt + full budget), `--seed-specials + 0.25 --combo-explore 0.5 --gamma 0 --dense --seed 1`. + **Gates (pre-registered):** (1) 500-ep natural-board score NOT CI-separated below `cf9train`'s 5666.4; + (2) probe (1000 eps, natural boards): combo take-rate โ‰ฅ 75% (from 63%); (3) bomb-bucket opportune โ‰ฅ 85% + (from 80%); (4) all M50.3 bars stay green (โ‰ฅ+30%/random ยท gap-share โ‰ฅ64% ยท created โ‰ฅ7.3 / fired โ‰ฅ5.6); + (5) per-kind created non-regression vs cf9train, ESPECIALLY bomb-created (free seeded specials are exactly + the pressure to under-create the rarest kind). Keep-best ships only a net that beats the resumed net's + eval. Escalation if (2)/(3) miss: raise q, one run, then stop-loss. +- **M52.4 โ€” Ship + PR.** Best net โ†’ `wwwroot/models/crazyfruits.dqn.ckpt` (drop-in, no web change needed); + docs/memory synced; PR โ†’ master. + +## 4. Results + +*(to be filled at each gate)* diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index bd37ee9..4843ca0 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -2035,6 +2035,37 @@ reward and the input plane; combo shaping would double-count (shape what pays la - **M51.4 โ€” Ship.** โœ… `cf9train` โ†’ `wwwroot/models/crazyfruits.dqn.ckpt`; round-over bar ~4 000 โ†’ ~5 650; parity pin 481681208 unchanged; docs synced. +## M52 โ€” Crazy Fruits combo curriculum *(2026-07-25; see `CRAZY_FRUITS_COMBO_CURRICULUM_PRD.md`; branch `m52-crazyfruits-combo-curriculum`)* ๐Ÿ”„ + +**Why:** owner's post-M51 question โ€” the net has plausibly never *experienced* wrapped+wrapped or bomb+bomb. +Half-true: the deterministic combo payoff is in its input planes AND its dense targets (combo take already +63% vs random's 10% with zero engineering); what's missing is frequency (combo-legal states โ‰ˆ2.8% of natural +play; adjacent bomb+bomb โ‰ˆ never) and the realized post-combo refill continuation, which only playing them +teaches. Deliberately *setting up* combos stays out of scope (ฮณ=0 can't represent it โ€” that's the +search-distillation lever). 2-agent design verification (code-safety + RL sanity) before build. + +- **M52.0 โ€” Design verification.** โœ… Injection invariant-safe (kind overwrite keeps fruit type โ‡’ typewise + match structure unchanged; bombs never join runs and self-guarantee legality; reshuffle bounded); + env-stream RNG keeps determinism; save/restore untouched. RL verdicts: ฮณ=0 has NO bootstrap propagation + path, so curriculum risk reduces to benign covariate shift (labels identical on both distributions) โ€” + p=0.25; **adjacent pairs only** (a near pair's bring-together swap has a flat ฮณ=0 label โ€” teaches nothing); + **combo-biased ฮต is the PRIMARY lever** (realized combo experience, zero distribution shift); prefer full + RESUME of cf9train (keeps Adam moments + the 100k natural replay buffer) over warm-start. +- **M52.1 โ€” Env + trainer seam + Lab.** โœ… `CrazyFruitsEnv.SeedSpecialsProb` (adjacent pair + โ‰ค2 singles on + Reset, env RNG stream, default OFF โ€” eval env stays natural) + `ComboExploreBias` / + `SuggestComboExploration`; generic `DqnOptions.ExploreBias` hook consulted only in the ฮต-branch (null โ‡’ + zero extra draws โ‡’ every other game bitwise-identical); Lab `--seed-specials` / `--combo-explore`; + per-kind created counts in `--baselines`. **Gate:** 67/67 tests green (4 new: seeded invariants + + determinism + combo suggestion + agent bias redirect). +- **M52.2 โ€” Spike.** โœ… Resume at 400k confirmed, keep-best baseline 6014 armed, 10k curriculum steps, loss + normal, no eval collapse. +- **M52.3 โ€” Train `cf10train`.** ๐Ÿ”„ Resume cf9train โ†’ 800k absolute (400k new; owner choice: reuse ckpt + + full budget), p=0.25, q=0.5. **Gates (pre-registered):** (1) 500-ep natural score not CI-separated below + cf9train's 5666.4; (2) probe combo take โ‰ฅ75% (from 63%); (3) bomb-bucket opportune โ‰ฅ85% (from 80%); + (4) M50.3 bars stay green; (5) per-kind created non-regression, especially bomb-created. Escalation if + (2)/(3) miss: raise q, one run, stop-loss. +- **M52.4 โ€” Ship + PR.** Best net โ†’ wwwroot ckpt (drop-in); docs/memory synced; PR โ†’ master. + **Rejected up front:** bigger shaping bonuses (reward-hack trap), combo shaping (double-count), ฮณ>0 schedules (n=2 losses: M49 ฮณ=0.99, M50.3 `cf8train`), regret-prioritized replay (subsumed by dense regression). diff --git a/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs b/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs index d91059d..0860ed5 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs @@ -48,6 +48,8 @@ public sealed partial class CrazyFruitsDqnCampaign : DqnScoreCampaign EvalEpisodes = 20, DenseTargets = Typed.DenseRegression ? DenseTargetsFromObservation : null, DenseTargetWeight = Typed.DenseTargetWeight, + // Combo-biased ฮต (M52): the injected TRAIN env carries the knob; wire its suggestion hook through. + ExploreBias = TrainEnv is CrazyFruitsEnv { ComboExploreBias: > 0 } cf ? cf.SuggestComboExploration : null, }; // The observation's shaped per-action plane (deterministicValueShaped(a)/300, 0 = illegal โ€” a legal swap diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs index 2b316b7..51f3cb3 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs @@ -63,6 +63,12 @@ public sealed record DqnOptions /// reward's refill-expectation signal 30:1. public float DenseTargetWeight { get; init; } = 1.0f; + /// Optional exploration bias, consulted only when an ฮต-exploration step fires: return a LEGAL + /// action to play it, โˆ’1 to fall back to the uniform-random legal pick. Null (default) draws nothing + /// extra from the policy RNG, so existing runs stay bitwise-identical. Crazy Fruits routes rare + /// combo swaps through this (COMBO_CURRICULUM PRD, M52). + public Func? ExploreBias { get; init; } + public int EvalEvery { get; init; } = 5_000; public int EvalEpisodes { get; init; } = 20; @@ -76,8 +82,11 @@ public sealed record DqnOptions public sealed record DqnResult(GreedyQAgent Agent, IValueNet Network, int StepsTrained, double FinalEvalReturn, DqnTrainingState State); -/// Acts greedily from a Q-network (evaluation/playback); epsilon-greedy when given an RNG. -public sealed class GreedyQAgent(IValueNet network, int actionCount, Xoshiro256StarStar? rng = null) : IAgent +/// Acts greedily from a Q-network (evaluation/playback); epsilon-greedy when given an RNG. An +/// optional (see ) can redirect an +/// exploration step to a targeted legal action. +public sealed class GreedyQAgent(IValueNet network, int actionCount, Xoshiro256StarStar? rng = null, + Func? exploreBias = null) : IAgent { public double Epsilon { get; set; } @@ -95,6 +104,8 @@ public int Act(float[] observation, bool[]? mask, bool greedy = false) { if (!greedy && rng is not null && rng.NextDouble() < Epsilon) { + int biased = exploreBias?.Invoke(rng) ?? -1; + if (biased >= 0) return biased; if (mask is null) return rng.NextInt(actionCount); int legal = mask.Count(m => m); int pick = rng.NextInt(legal); @@ -201,7 +212,7 @@ IValueNet WarmStart(IValueNet w) => w.InputSize == obsDim ? w } } - var agent = new GreedyQAgent(state.Online, actionCount, state.PolicyRng); + var agent = new GreedyQAgent(state.Online, actionCount, state.PolicyRng, options.ExploreBias); var accumulator = state.Accumulator!; // set above for both fresh and resumed runs var maskProvider = env as IActionMaskProvider; float lastLoss = state.LastLoss; diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs index 7da7acf..914a56f 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs @@ -100,6 +100,20 @@ public CrazyFruitsEnv(int moveBudget = 30) public float WrappedShaping { get; set; } = 60f; public float BombShaping { get; set; } = 100f; + // โ”€โ”€ Combo curriculum (COMBO_CURRICULUM PRD, M52 โ€” train env only; both default OFF) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + /// Probability that a fresh episode's board is dealt combo-ready: one ADJACENT special pair plus + /// up to two singles, injected by overwriting plain cells' kinds (fruit types unchanged, so the deal's + /// no-instant-match and has-legal-swap guarantees survive; a bomb never joins runs and its swaps are + /// always legal). Draws come from the env RNG stream, never the board's refill stream โ€” seeded runs stay + /// deterministic, and the default 0 leaves every draw count untouched. + public double SeedSpecialsProb { get; set; } + + /// Probability that an ฮต-exploration step with a legal special+special swap available picks + /// uniformly among the combo swaps (the DqnOptions.ExploreBias hook, + /// ) โ€” realized combo experience on natural boards. + public double ComboExploreBias { get; set; } + /// The ยง3.6 escalation's shaping (use INSTEAD of , with ฮณ>0): /// potential-based, ฮฆ(s) = ฮฃ option value of on-board specials (same 40/60/100 weights), reward += /// ฮณยทฮฆ(sโ€ฒ) โˆ’ ฮฆ(s). Policy-invariant when matches the learner's ฮณ โ€” it prices @@ -128,11 +142,58 @@ private float Potential() // Each episode's board deals from the env RNG stream: reproducible under an explicit seed, fresh // boards across unseeded resets. _board.Reset(_rng.NextUInt64()); + if (SeedSpecialsProb > 0 && _rng.NextDouble() < SeedSpecialsProb) + SeedSpecials(); _moves = 0; _done = false; return (_board.BuildObservation(), EnvInfo.Empty); } + /// Deal the fresh board combo-ready (see ): an adjacent special + /// pair plus up to two singles, all on plain cells with their fruit type kept. + private void SeedSpecials() + { + var grid = _board.GridSnapshot(); + bool horizontal = _rng.NextDouble() < 0.5; + int r = _rng.NextInt(horizontal ? CrazyFruitsBoard.Size : CrazyFruitsBoard.Size - 1); + int c = _rng.NextInt(horizontal ? CrazyFruitsBoard.Size - 1 : CrazyFruitsBoard.Size); + int pairA = r * CrazyFruitsBoard.Size + c; + int pairB = horizontal ? pairA + 1 : pairA + CrazyFruitsBoard.Size; + grid[pairA] = Seeded(grid[pairA]); + grid[pairB] = Seeded(grid[pairB]); + int extras = _rng.NextInt(3); + for (int i = 0; i < extras; i++) + { + int cell = _rng.NextInt(CrazyFruitsBoard.Cells); + if (grid[cell] < 16) grid[cell] = Seeded(grid[cell]); // plain cells only + } + _board.LoadGrid(grid); + } + + // Kinds 1..4 uniform (stripedH ยท stripedV ยท wrapped ยท bomb); striped/wrapped keep the cell's fruit type + // (typewise match structure unchanged), the bomb is colorless. + private int Seeded(int packedPlain) + { + int kind = 1 + _rng.NextInt(4); + return kind == 4 ? 4 * 16 : kind * 16 + packedPlain % 16; + } + + /// The DqnOptions.ExploreBias hook (see ): โˆ’1 when the + /// roll passes or no special+special swap is legal, else a uniform pick among the legal combo swaps. + public int SuggestComboExploration(Xoshiro256StarStar rng) + { + if (rng.NextDouble() >= ComboExploreBias) return -1; + var mask = _board.LegalMask(); + var combos = new List(); + for (int a = 0; a < ActionCount; a++) + { + if (!mask[a]) continue; + var (cellA, cellB) = _board.SwapCells(a); + if (_board.Kind(cellA) > 0 && _board.Kind(cellB) > 0) combos.Add(a); + } + return combos.Count == 0 ? -1 : combos[rng.NextInt(combos.Count)]; + } + public StepResult Step(int action) { if (_done) diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEnvTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEnvTests.cs index a942197..de9bc29 100644 --- a/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEnvTests.cs +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEnvTests.cs @@ -149,4 +149,84 @@ public void Campaign_Registers_TrainsCheckpointsAndResumes() dir.Delete(recursive: true); } } + + // โ”€โ”€ Combo curriculum (M52, COMBO_CURRICULUM_PRD.md) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + + [Fact] + public void SeededReset_KeepsInvariants_AndDealsAnAdjacentSpecialPair() + { + var env = new CrazyFruitsEnv { SeedSpecialsProb = 1.0 }; + for (ulong seed = 0; seed < 20; seed++) + { + env.Reset(seed); + var b = env.Board; + Assert.False(b.AnyMatchOnBoard()); // injection keeps the deal's no-instant-match + Assert.True(b.HasLegalSwap()); // ...and its has-legal-swap guarantee + + int specials = 0; + bool adjacentPair = false; + for (int i = 0; i < CrazyFruitsBoard.Cells; i++) + { + if (b.Kind(i) == 0) continue; + specials++; + int r = i / CrazyFruitsBoard.Size, c = i % CrazyFruitsBoard.Size; + if (c < CrazyFruitsBoard.Size - 1 && b.Kind(i + 1) > 0) adjacentPair = true; + if (r < CrazyFruitsBoard.Size - 1 && b.Kind(i + CrazyFruitsBoard.Size) > 0) adjacentPair = true; + } + Assert.True(specials >= 2, $"seed {seed}: expected the seeded pair, saw {specials} specials"); + Assert.True(adjacentPair, $"seed {seed}: seeded specials are not adjacent"); + + // The adjacent pair makes a special+special swap legal โ€” the combo the curriculum manufactures. + var mask = b.LegalMask(); + bool comboLegal = false; + for (int a = 0; a < CrazyFruitsBoard.ActionCount && !comboLegal; a++) + { + if (!mask[a]) continue; + var (cellA, cellB) = b.SwapCells(a); + comboLegal = b.Kind(cellA) > 0 && b.Kind(cellB) > 0; + } + Assert.True(comboLegal, $"seed {seed}: no legal special+special swap on a seeded board"); + } + } + + [Fact] + public void SeededReset_IsDeterministicPerSeed_AndDefaultEnvStaysPlain() + { + var a = new CrazyFruitsEnv { SeedSpecialsProb = 1.0 }; + var b = new CrazyFruitsEnv { SeedSpecialsProb = 1.0 }; + a.Reset(42); + b.Reset(42); + Assert.Equal(a.Board.GridSnapshot(), b.Board.GridSnapshot()); + + var plain = new CrazyFruitsEnv(); + plain.Reset(42); + for (int i = 0; i < CrazyFruitsBoard.Cells; i++) + Assert.Equal(0, plain.Board.Kind(i)); + } + + [Fact] + public void SuggestComboExploration_ReturnsALegalComboSwap_OrMinusOne() + { + var env = new CrazyFruitsEnv { SeedSpecialsProb = 1.0, ComboExploreBias = 1.0 }; + env.Reset(5); + var rng = new Core.Random.Xoshiro256StarStar(9); + int action = env.SuggestComboExploration(rng); + Assert.True(action >= 0, "a seeded board has a legal combo swap โ€” the hook must find it"); + Assert.True(env.CurrentActionMask()[action]); + var (cellA, cellB) = env.Board.SwapCells(action); + Assert.True(env.Board.Kind(cellA) > 0 && env.Board.Kind(cellB) > 0); + + var never = new CrazyFruitsEnv { ComboExploreBias = 0.0 }; + never.Reset(5); + Assert.Equal(-1, never.SuggestComboExploration(rng)); // roll can never pass at q=0 + } + + [Fact] + public void GreedyQAgent_ExploreBias_RedirectsExplorationSteps() + { + var net = new MintPlayer.AI.ReinforcementLearning.Core.Nn.Mlp([4, 8, 3], new Core.Random.Xoshiro256StarStar(1), Core.Nn.Activation.Relu); + var agent = new GreedyQAgent(net, 3, new Core.Random.Xoshiro256StarStar(2), exploreBias: _ => 2) { Epsilon = 1.0 }; + for (int i = 0; i < 5; i++) + Assert.Equal(2, agent.Act([1f, 0f, 0f, 0f], [true, true, true])); + } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs index e1e13af..72aa279 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs @@ -67,6 +67,9 @@ public static void Run(string[] args) ShapeCreationRewards = shape, ShapeSpecialsPotential = pbrs, PotentialGamma = gamma, + // Combo curriculum (M52, COMBO_CURRICULUM PRD): train env only โ€” eval stays natural. + SeedSpecialsProb = a.Dbl("--seed-specials", 0), + ComboExploreBias = a.Dbl("--combo-explore", 0), }, evalEnv: new CrazyFruitsEnv(moveBudget), options), @@ -107,14 +110,14 @@ private static void RunBaselines(int episodes, int moveBudget, ulong seed, strin var agent = new GreedyQAgent(net, CrazyFruitsEnv.ActionCount); var env = new CrazyFruitsEnv(moveBudget); double sum = 0, sumSq = 0; - long created = 0, fired = 0; + long cs = 0, cw = 0, cb = 0, fired = 0; for (int e = 0; e < episodes; e++) { var (obs, _) = env.Reset((ulong)(5_000 + e)); while (true) { var step = env.Step(agent.Act(obs, env.CurrentActionMask(), greedy: true)); - created += env.Board.MoveCreatedStriped + env.Board.MoveCreatedWrapped + env.Board.MoveCreatedBombs; + cs += env.Board.MoveCreatedStriped; cw += env.Board.MoveCreatedWrapped; cb += env.Board.MoveCreatedBombs; fired += env.Board.MoveSpecialsFired; obs = step.Observation; if (step.Done) break; @@ -122,7 +125,7 @@ private static void RunBaselines(int episodes, int moveBudget, ulong seed, strin sum += env.Score; sumSq += (double)env.Score * env.Score; } - Console.WriteLine($" net specials/episode: created {(double)created / episodes:F2}, fired {(double)fired / episodes:F2}"); + PrintSpecialsLine("net", episodes, cs, cw, cb, fired); results.Add(Summarize($"net ({Path.GetFileName(netPath)})", episodes, sum, sumSq)); } else if (!File.Exists(netPath)) @@ -198,6 +201,11 @@ private static void RunProbe(int episodes, int moveBudget, string netPath) var opportunityTaken = new long[policies.Count]; var comboTaken = new long[policies.Count]; var oracleMatch = new long[policies.Count]; + // Opportune states bucketed by the best action's creation bonus (shaped โˆ’ plain immediate): + // 40 = striped (4-run), 60 = wrapped (L/T), โ‰ฅ100 = bomb (5-run) or multi-creation. Owner report + // 2026-07-25 round 2: a skipped 5-in-a-row โ€” the bomb bucket answers whether that's systemic. + var kindStates = new int[3]; + var kindTaken = new long[3 * policies.Count]; var env = new CrazyFruitsEnv(moveBudget); var creating = new bool[CrazyFruitsEnv.ActionCount]; @@ -228,7 +236,14 @@ private static void RunProbe(int episodes, int moveBudget, string netPath) statesSeen++; bool opportune = bestAction >= 0 && creating[bestAction]; - if (opportune) opportuneStates++; + int kind = -1; + if (opportune) + { + opportuneStates++; + int bonus = b.ImmediateScoreShaped(bestAction) - b.ImmediateScore(bestAction); + kind = bonus >= 100 ? 2 : bonus >= 60 ? 1 : 0; + kindStates[kind]++; + } if (anyCreating) opportunityStates++; if (anyCombo) comboStates++; @@ -237,7 +252,11 @@ private static void RunProbe(int episodes, int moveBudget, string netPath) int action = policies[p].Act(b, stateIndex); if (action < 0) continue; if (action == bestAction) oracleMatch[p]++; - if (anyCreating && creating[action]) { opportunityTaken[p]++; if (opportune) opportuneTaken[p]++; } + if (anyCreating && creating[action]) + { + opportunityTaken[p]++; + if (opportune) { opportuneTaken[p]++; kindTaken[kind * policies.Count + p]++; } + } if (anyCombo) { var (cellA, cellB) = b.SwapCells(action); @@ -253,21 +272,25 @@ private static void RunProbe(int episodes, int moveBudget, string netPath) Console.WriteLine($"Crazy Fruits opportunity probe: {episodes} episodes ร— {moveBudget} random-walk moves (seeds 9000+e)"); Console.WriteLine($" states {statesSeen}: creating swap available {opportunityStates}, creating swap is the " + $"shaped-optimal move {opportuneStates} | special+special legal {comboStates}"); - Console.WriteLine($" {"policy",-18} {"opportune take",16} {"raw take",10} {"oracle match",14} {"combo take",12}"); + Console.WriteLine($" opportune states by best-action creation: striped {kindStates[0]} ยท wrapped {kindStates[1]} ยท bomb/multi {kindStates[2]}"); + Console.WriteLine($" {"policy",-18} {"opportune take",16} {"striped",9} {"wrapped",9} {"bomb",9} {"raw take",10} {"oracle match",14} {"combo take",12}"); for (int p = 0; p < policies.Count; p++) { string Rate(long taken, int total) => total == 0 ? "n/a" : $"{(double)taken / total:P1}"; Console.WriteLine($" {policies[p].Name,-18} {Rate(opportuneTaken[p], opportuneStates),16} " + + $"{Rate(kindTaken[0 * policies.Count + p], kindStates[0]),9} " + + $"{Rate(kindTaken[1 * policies.Count + p], kindStates[1]),9} " + + $"{Rate(kindTaken[2 * policies.Count + p], kindStates[2]),9} " + $"{Rate(opportunityTaken[p], opportunityStates),10} {Rate(oracleMatch[p], statesSeen),14} " + $"{Rate(comboTaken[p], comboStates),12}"); } - Console.WriteLine(" gate (RANKING PRD M51.2): net opportune take-rate โ‰ฅ 90% (oracle = 100% by construction)."); + Console.WriteLine(" gate (RANKING PRD M51.2, final form): net opportune take-rate โ‰ฅ expectimax-1 โˆ’ 5 pts."); } private static (string, double, double) RunPolicy(string name, int episodes, int moveBudget, Func policy) { double sum = 0, sumSq = 0; - long created = 0, fired = 0; + long cs = 0, cw = 0, cb = 0, fired = 0; for (int e = 0; e < episodes; e++) { // Reset through the env seed path so every policy sees the same boards as the net eval. @@ -277,16 +300,23 @@ private static (string, double, double) RunPolicy(string name, int episodes, int for (int move = 0; move < moveBudget; move++) { b.ApplySwap(policy(b, move)); - created += b.MoveCreatedStriped + b.MoveCreatedWrapped + b.MoveCreatedBombs; + cs += b.MoveCreatedStriped; cw += b.MoveCreatedWrapped; cb += b.MoveCreatedBombs; fired += b.MoveSpecialsFired; } sum += b.Score; sumSq += (double)b.Score * b.Score; } - Console.WriteLine($" {name,-28} specials/episode: created {(double)created / episodes:F2}, fired {(double)fired / episodes:F2}"); + PrintSpecialsLine(name, episodes, cs, cw, cb, fired); return Summarize(name, episodes, sum, sumSq); } + // Per-kind created counts (M52 gate: kind-mix must not regress, ESPECIALLY bomb-created โ€” seeded boards + // hand the net free specials, exactly the pressure to under-create the rarest kind). + private static void PrintSpecialsLine(string name, int episodes, long cs, long cw, long cb, long fired) + => Console.WriteLine($" {name,-28} specials/episode: created {(double)(cs + cw + cb) / episodes:F2} " + + $"(striped {(double)cs / episodes:F2} ยท wrapped {(double)cw / episodes:F2} ยท bomb {(double)cb / episodes:F2}), " + + $"fired {(double)fired / episodes:F2}"); + private static (string, double, double) Summarize(string name, int n, double sum, double sumSq) { double mean = sum / n;