diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fa39914..7268ea3 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -192,7 +192,9 @@ to all forwards per chunk. See [`GPU_RESIDENT_CONV_PRD.md`](prd/GPU_RESIDENT_CON `1e-3`, `BufferCapacity` `50k`, `BatchSize` `64`, `WarmupSteps` `1000`, `TrainEvery` `1`, `TargetSyncEvery` `500`, `Epsilon` `LinearSchedule(1→0.05 over 10k)`, `MaxSteps` `100k`, `MaxGradNorm` `10`, `DoubleDqn` `true`, `Dueling` `false`, `NoisyNets` `false` (implies Dueling), `NStep` `1`, `StartStates`/`StartStateProb` -(reverse-curriculum), `EvalEvery`/`EvalEpisodes`, `SolveThreshold`. +(reverse-curriculum), `EvalEvery`/`EvalEpisodes`, `SolveThreshold`, `DenseTargets`/`DenseTargetWeight` +(γ=0-only dense all-action regression — every legal action regressed toward a caller-supplied target so a +wrong action RANKING costs loss; Crazy Fruits reads targets off its shaped observation plane, PLAN M51). ```csharp // DqnTrainer.cs — the TD target (n-step aware, masked, terminated-only bootstrap) @@ -495,7 +497,7 @@ logic modules mirroring the C# envs by hand. | [`snake-logic.ts`](../src/RLDemo.Web/ClientApp/src/app/snake/snake-logic.ts), [`mountaincar-logic.ts`](../src/RLDemo.Web/ClientApp/src/app/mountaincar/mountaincar-logic.ts), [`game-2048-logic.ts`](../src/RLDemo.Web/ClientApp/src/app/game-2048/game-2048-logic.ts), [`rush-hour-logic.ts`](../src/RLDemo.Web/ClientApp/src/app/rush-hour/rush-hour-logic.ts) | Browser-side rules for human play, mirroring the C# envs. | | [`cube/cube.ts`](../src/RLDemo.Web/ClientApp/src/app/cube/cube.ts) | Three.js scene + manual turns + Kociemba/AI solver playback. | | [`draughts/`](../src/RLDemo.Web/ClientApp/src/app/draughts/) `draughts_solver.ts` (generated, gitignored) · [`draughts-net.ts`](../src/RLDemo.Web/ClientApp/src/app/draughts/draughts-net.ts) · [`draughts-director.ts`](../src/RLDemo.Web/ClientApp/src/app/draughts/draughts-director.ts) · [`draughts.ts`](../src/RLDemo.Web/ClientApp/src/app/draughts/draughts.ts) | Draughts/checkers, fully client-side (the chess M40 pattern, PLAN M47.5): generated engine + **conv tower** + MCTS twin (edit `draughts_solver.pg` and build — the first CONV net in the browser); `.ckpt` parser for kind `selfplay-pv-conv` (byte reference: `DraughtsNetParityTests`); director (play + AI-vs-AI watch, tier manifest `wwwroot/models/draughts-difficulties.json`); board component. | -| [`crazy-fruits/`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/) `crazyfruits_solver.ts` (generated, gitignored) · [`crazyfruits-net.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazyfruits-net.ts) · [`crazy-fruits-game.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-game.ts) · [`crazy-fruits-director.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-director.ts) · [`crazy-fruits.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits.ts) | Crazy Fruits match-3, fully client-side (PLAN M49): generated engine incl. the scripted tiers + `PgCfDuelingNet` (edit `crazyfruits_solver.pg` and build); `.ckpt` parser for kind `dueling-q` (byte reference: `CrazyFruitsNetParityTests`); animating host driving the engine's **stepwise move API** (swap → clearStep → collapse → finishMove); watch director (tiers Random/Greedy/Expectimax/net); component with **unified pointer input** (drag-swap + tap-tap — one path for mouse and touch). | +| [`crazy-fruits/`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/) `crazyfruits_solver.ts` (generated, gitignored) · [`crazyfruits-net.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazyfruits-net.ts) · [`crazy-fruits-game.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-game.ts) · [`crazy-fruits-director.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-director.ts) · [`crazy-fruits.ts`](../src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits.ts) | Crazy Fruits match-3, fully client-side (PLAN M49): generated engine incl. the scripted tiers + `PgCfDuelingNet` (edit `crazyfruits_solver.pg` and build); `.ckpt` parser for kind `dueling-q` (byte reference: `CrazyFruitsNetParityTests`); animating host driving the engine's **stepwise move API** (swap → clearStep → collapse → finishMove); watch director (tiers Random/Greedy/Specials-greedy/Expectimax/Expectimax-2/net, with a stale-ckpt guard: input width ≠ observation width → expectimax fallback, PLAN M51.1); component with **unified pointer input** (drag-swap + tap-tap — one path for mouse and touch). | | [`screen-wake-lock.ts`](../src/RLDemo.Web/ClientApp/src/app/screen-wake-lock.ts) | Shared service: holds a screen wake lock during watch-AI; re-acquires on foreground. | Each component has a `mode` signal: `'human'` (browser timer) vs `'watch'` (server WebSocket stream). diff --git a/docs/prd/CRAZY_FRUITS_RANKING_PRD.md b/docs/prd/CRAZY_FRUITS_RANKING_PRD.md new file mode 100644 index 0000000..5ef9e59 --- /dev/null +++ b/docs/prd/CRAZY_FRUITS_RANKING_PRD.md @@ -0,0 +1,205 @@ +# Crazy Fruits — missed-opportunity ranking (net prefers a 3-match over an available 4-match) — PRD + +**Status:** 🔜 planned 2026-07-25 (4-agent investigation: reward structure, inference path, training loss, technique survey) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M51 · extends [CRAZY_FRUITS_SPECIALS_PRD.md](CRAZY_FRUITS_SPECIALS_PRD.md) (M50, shipped) · branch: continue the arc branch `m49-crazy-fruits` if PR #38 is still open, else `m51-crazy-fruits-ranking` + +## 1. Problem + +Owner observation on the shipped `cf5train` net: on boards where the same fruit offers **both** a 3-in-a-row +and a 4-in-a-row swap, the AI takes the 3-match — forfeiting the striped special the 4-match would create. +The net was trained short ("just enough to publish") and will train longer regardless; the owner's question +is whether we can additionally **punish the model harder when it misses important opportunities**. + +Follow-up question (owner, same day): should combining two specials earn a higher reward, or does that +already happen through the score? **Resolved by investigation — already correct, no change (§2.7).** + +## 2. Investigation findings (4 agents, 2026-07-25) + +1. **The loss cannot punish a ranking error — root cause.** `DqnTrainer.TrainStep` regresses ONLY the chosen + action's Q (`Gather(batch.Actions)`, `DqnTrainer.cs:291`; Huber δ=1, `:292`). At γ=0 the target is the raw + scaled reward (`:283-286`). Ranking a 3-match above a co-available 4-match costs **zero loss** — the + 4-match's Q is simply never updated on that transition. "Punish harder for missed opportunities" is + therefore not a reward tweak; it's a **loss-structure** change. +2. **The reward already distinguishes 3 vs 4 — not a reward-definition gap.** Clean single match at step 0 + (`crazyfruits_solver.pg:658,68-70`; RewardScale=100, shaping +40/+60/+100 on the train env only, + `CrazyFruitsEnv.cs:97-99,148-152`): + + | Match | game pts | train reward | eval reward | + |---|---|---|---| + | 3-run | 30 | 0.30 | 0.30 | + | 4-run | 60 (+20 line bonus) | **1.00** (+40 shaping) | 0.60 | + | 5-run | 100 (+50 bonus) | **2.00** (+100 shaping) | 1.00 | + + No clipping anywhere; ÷100 keeps both targets in Huber's quadratic zone — normalization is ruled out. +3. **But the signal is small against refill-cascade variance.** Mean reward ≈0.86/move with high variance; + a 3-match cascading one extra step (~6 cells at k=1 → 1.50) legitimately out-rewards a flat 4-match + (1.00 train / 0.60 eval). γ=0 must average this noise per (s,a) from sparse visits — an SNR problem. +4. **The creation bonus is invisible in the observation.** The +40/+60/+100 exists only in the training + TARGET. Both per-action input planes are **fire-only** (`immediateScore/300`, `deterministicValue/300`, + `pg:775-777`): the input ranks the 4-match only ~1.3× above the 3-match while the target ranks it + 2.6–3.3×. The net must infer "this geometry creates a striped worth +0.4" from raw fruit planes — + fighting its own most informative features. A per-action *shaped* plane was deliberately deferred in the + M50 v1 ("don't feed the answer"); it is now the identified missing feature. +5. **The web `net` tier does NOT search.** `netAction` (`pg:985` region) is a pure masked argmax over one + forward pass; `expectimax`/`expectimax2` are separate scripted tiers that never touch the net. (Corrects + the memory note "expectimax-1 uses the net".) There is no inference-side mechanism that could rescue the + ranking today. +6. **Data rarity compounds it.** Special-creating swaps are ~12% of random moves; post-warmup the collector + is ~95% greedy, so a net settled into the 3-match habit rarely revisits 4-matches — the chosen-action-only + loss then never corrects them. +7. **Special+special combos — already priced correctly, RESOLVED, no change.** Combos fire ON the swap + (staged combo executes in `clearStep(0)`), so their full score lands in the immediate reward the γ=0 net + regresses: striped+striped 1.50 · wrapped+wrapped ~2.50+ · striped+wrapped ~3.90+ · bomb+bomb ~6.40 — + versus ~0.80–1.10 for firing one special alone; combining always wins, and the `immediateScore/300` input + plane (a real `clearStep(0)` incl. `executeCombo`) makes that value visible to the net. Combo shaping + would double-count and invite reward-hacking: the deliberate asymmetry is **shape what pays later + (creation), never what pays now (firing/combining)**. +8. **Owner-confirmed constraint (2026-07-25): planning only sees fruits already on the board.** Refill is + unknowable noise — all search is and stays refill-free (`resolveCascades(false)`), so each extra search + ply plans on an increasingly depleted fictional board. Deeper search (depth 3+) is compute-feasible with + beams but decays in value; the *expected* refill continuation is the net's job, not the search's. +9. **γ=0 ceiling (context for expectations).** At γ=0 the net regresses a quantity the deterministic + simulator computes exactly, plus refill noise; its only genuine add over the scripted `specialsGreedy` + tier is the *expected refill continuation*. Every immediate-term lever below therefore converges the net + toward (a denoised) `specialsGreedy` — which fixes the owner's complaint but cannot claim the + hold-for-combo gap (M50.3 gate 2). That gap needs a teacher with cross-move vision (§3, lever C). + Exhausted levers (do NOT retry): γ>0 reward schedules (γ=0.99 M49, γ=0.5+3-step+PBRS `cf8train` — n=2 + losses), bigger shaping magnitudes (net chases shaping, gates measure bare score). + +## 3. Design — three levers, cheapest first + +**Lever A — inference re-rank: DROPPED at design time (2026-07-25), superseded by Lever B.** +The λ re-rank was scoped for the SHIPPED 928-input ckpt, but Lever B changes the observation width in the +same single-source `.pg` — the moment the obs grows, the old ckpt can't forward at all, so "quick fix on the +old net" and "retrain" cannot coexist in one working tree. Since the retrain runs immediately (this +session), the re-rank buys nothing. What replaces it: a **stale-ckpt guard** in the web loader (net input +width ≠ engine observation width → treat the net as missing, fall back to the expectimax tier) — needed for +the 928→1040 transition anyway and permanent robustness for every future obs change. + +**Lever B — retrain with a ranking-aware loss + the missing feature (the owner's "punish harder", made +precise).** Three changes, one from-scratch run: +1. **Shaped per-action plane:** add a NEW `deterministicValueShaped(a)/300` third per-action block — obs + 928 → **1040**. *(Design correction over the first draft's `immediateScoreShaped`: the shaped + DETERMINISTIC value — full refill-free cascade + creation weights, incl. cascade-made creations — matches + the realized training reward up to refill noise. An immediate-only shaped target would re-create the exact + bias we're fixing: a cascading 3-match's target would include its cascade while a 4-match's would not.)* + Keep the two fire-only planes (the eval scoreboard is fire-only and the net should see both). Closes the + §2.4 perception gap; this feature class is the exact lever that gated M49. +2. **Dense all-action regression:** per sampled state, supervise EVERY legal action — the taken action + toward its realized reward (keeps refill-cascade expectation learnable), every other legal action toward + `deterministicValueShaped(a)/100`, masked Huber. Weight semantics (locked): the dense term is normalized + per supervised entry so the WHOLE dense term carries `DenseTargetWeight` (default 1.0) × the realized- + reward term's total gradient mass — the ranking signal and the refill-expectation signal get equal say; + per-entry weighting would let ~30 legal actions/state drown the realized term 15:1. The targets are + already computed each step for the obs planes — this lever is nearly free, and it is the only one that + makes a wrong ORDERING cost loss. Trainer seam: `DqnOptions.DenseTargets` (per-obs delegate, NaN = + unsupervised action), guarded to γ=0 (only there is a dense target computable from s alone). +3. **Regret emphasis (pre-registered, only if the probe gate still fails):** add a margin hinge + `max(0, Q(a_best) + m − Q(a_taken))` on transitions where the behaviour policy missed a + strictly-better deterministic action (`a_best` from the shaped oracle). Registered up front to avoid + schedule-tinkering drift; margin distorts Q as a reward estimate, so it stays OFF unless needed. + Training length: 400k steps for comparability with `cf5train`; one optional 800k continuation if gates + are close (the owner intends longer training anyway). + +**Lever C — pre-registered escalation: expectimax-2 distillation (DAgger-style).** Trigger: gate 2 +(gap-share ≥64%) still missed after Lever B. Teacher = `expectimax2Action` labels on states the *student* +visits (fixes distribution shift); auxiliary cross-entropy on the teacher's argmax alongside the dense +regression. This is the only lever that teaches value beyond the deterministic oracle (create→fire, +hold-for-combo) — the M50.3 close-out already pointed here ("search-guided play, not another reward +schedule"). Ceiling: the teacher is 2-ply and refill-free. + +**Rejected:** bigger creation-shaping bonuses (known trap, §2.8) · combo shaping (§2.7, double-count) · +γ>0 schedules (n=2) · regret-prioritized replay (subsumed by dense regression — every action gets a target +every update, so per-state prioritization by the missed action loses its point). + +## 4. Probe — make "misses important opportunities" measurable (M51.0, before any fix) + +A deterministic seeded probe over N≥500 states sampled by random walks, two metrics: +- **Strict 3-vs-4 probe:** states whose legal mask contains both a special-creating swap and a plain 3-match + swap; metric = P(policy picks a creating swap). +- **Opportunity take-rate:** states where any special-creating swap exists; fraction taken. Baselines: + random, greedy, specialsGreedy, expectimax-1/2, shipped net. +Plus a **combo take-rate** stat (states with a legal special+special swap) — expected already-high (§2.7), +recorded to confirm, not gated. + +**M51.0 results (2026-07-25, 300 eps × 30 random-walk moves, seeds 9000+e; Lab `--probe 300`):** 9000 states, +3888 with a creating swap (all strict), 256 with a legal special+special combo. + +| policy | strict take-rate | combo take-rate | +|---|---|---| +| random | 14.2% | 8.6% | +| greedy | 71.7% | 95.7% | +| specials-greedy | 91.4% | 94.9% | +| expectimax-1 | 33.1% | 71.5% | +| expectimax-2 | 38.4% | 24.2% | +| **shipped net (`cf5train`)** | **17.6%** | **48.0%** | + +Two lessons locked in: (a) the shipped net is barely above random on the owner's exact scenario — complaint +confirmed and quantified; (b) **the gate must be RELATIVE to specials-greedy, not an absolute 95%** — the +draft assumed the shaped oracle takes a creating swap 100% of the time, but it measures 91.4%: sometimes a +non-creating swap (typically firing an existing special) legitimately out-scores creation even under shaped +scoring, and a net pinned to 100% would be WRONG in those states. Gate re-locked: **net strict take-rate ≥ +specials-greedy's − 5 pts on the same probe** (≥ 86.4% against today's baseline). The fire-only expectimax +tiers' 33–38% independently confirm §2 finding 4 (creation invisible to fire-only value). + +**Probe-bar post-mortem (2026-07-25, after the cf9train results — recorded in full because the bar moved +twice):** the ≥86.4% re-lock was ALSO mis-calibrated, and the cf9train run proved it: every policy that +actually scores well takes creating swaps far less often than specials-greedy (raw take: e1 33.1% at score +5951, e2 38.4% at 8098, cf9train net 35.5% at 5666 — vs specials-greedy's 91.4% at 3903). Creation-chasing +is simply not what optimal play looks like; a bar derived from specials-greedy selects for a 3903-scoring +behavior pattern. The probe was upgraded with the honest metric — **opportune take-rate = P(picks a creating +swap | a creating swap is the argmax of `deterministicValueShaped`)**, i.e. an opportunity only counts when +creating was actually the best move — plus an oracle-match column. On that metric the strong policies +cluster at 50–56% (e1 55.7%, e2 49.8%) because even the shaped 1-ply argmax is not score-optimal (e2 matches +it only 48% of the time while out-scoring everything). **Final gate (locked): net opportune take-rate within +5 pts of expectimax-1's — the strongest same-horizon planner — i.e. ≥ 50.7% against today's 55.7%.** The +absolute-90% idea died for the same reason the specials-greedy bar did: it demands behavior no strong policy +exhibits. What the probe is FOR is the before/after: cf5train sat at random-class behavior (raw 17.6% vs +random's 14.2%); the dense-loss net sits at search-class behavior on every column. + +## 5. Milestones + +- **M51.0 — Probe + baselines.** ✅ (2026-07-25) The §4 probe in the Lab (eval-only, seeded), run BEFORE the + obs change so the shipped 928-ckpt still forwards. Creating-swap detection = + `immediateScoreShaped(a) > immediateScore(a)` (works on the pre-change engine; step-0 creations are exactly + the owner's scenario). **Gate:** probe deterministic across runs; shipped-net strict-probe and take-rate + numbers recorded in this PRD. *Green: results table in §4 — net 17.6% vs specials-greedy 91.4%.* +- **M51.1 — Web stale-ckpt guard (replaces the dropped λ re-rank, §3).** ✅ (2026-07-25) Net input width ≠ + engine observation width ⇒ net treated as missing (expectimax fallback + console note) in the director's + load callback. **Gate:** with the 1040 engine and the old 928 ckpt the net tier plays expectimax with zero + console errors. +- **M51.2 — Retrain (Lever B).** ✅ (2026-07-25) Obs 1040 + dense loss (+ registered margin if probe-gated). + *Built: `deterministicValueShaped` + third obs plane in the `.pg` (parity pin UNCHANGED 481681208 — no rule + change, C#=TS re-verified); `DqnOptions.DenseTargets`/`DenseTargetWeight` in the shared trainer (γ=0 + guard; unsupervised entries via target:=prediction ⇒ zero grad; dense mass normalized per supervised + entry); campaign extractor reads the shaped plane ×3; Lab `--dense`/`--dense-weight`; tests 63/63 green + incl. 3 new dense-trainer tests (never-sampled-arm ranking, NaN unsupervised, γ-guard) + 3 engine tests. + Smoke (6k steps): eval mean score 4607 — already above cf5train's 400k-step 4040 (20-ep eval, noisy, but + the dense signal clearly bites). Full 400k run `cf9train` launched (--gamma 0 --dense --seed 1).* **Gates + (M50.3 bars + the §4 final probe gate):** ≥ +30% over random CI-separated · created ≥7.3 / fired ≥5.6 per + ep · gap-share ≥64% · net opportune take-rate ≥ e1 − 5 pts. + ***RESULTS (cf9train, 500-ep gate protocol, final-rules baselines): ALL GATES PASS — the first Crazy + Fruits net to do so.** Net **5666.4 ± 155.2** = **+117.2% over random** (bar +30%, CI-separated; cf5train + was +54.9%) · **gap-share 91%** (bar 64%; cf5train missed at 43%) · created **9.57** / fired **10.39** + (bars 7.3/5.6; cf5train missed at 5.81) · +61.4% over greedy, +45% over specials-greedy, 95% of e1's mean. + Probe: opportune take **54.9%** vs e1's 55.7% (bar ≥50.7% PASS — search-class; the margin hinge stays + unused) · raw take 17.6% → 35.5% (e1/e2-class) · combo take 48% → 66% · oracle match 54.2%. The margin + hinge and any escalation are NOT needed.* +- **M51.3 — Escalation (Lever C, trigger-gated).** ❌ Not triggered — gate 2 passed at 91% (bar 64%). The + remaining ceiling is the e1→e2 gap (hold-for-combo), out of scope per the M50.3 close-out. +- **M51.4 — Ship.** ✅ (2026-07-25) `cf9train` → `wwwroot/models/crazyfruits.dqn.ckpt` (the 1040-input net; + the director's new dims guard retires the stale-ckpt window); round-over bar "~4 000" → "~5 650"; + PLAN/this PRD synced (ARCHITECTURE.md needed no change — no hard-coded obs width). Live watch-tier check + on the running host + parity pin 481681208 unchanged. + +## 6. Key code references + +`DqnTrainer.cs` — `DqnOptions.DenseTargets`/`DenseTargetWeight` + the dense term in `TrainStep` · +`CrazyFruitsEnv.cs:89-99,134-156` (reward + shaping) · `CrazyFruitsBoard.cs` (per-action oracles: +`ImmediateScore`/`DeterministicValue`/`ImmediateScoreShaped`/**`DeterministicValueShaped`**) · +`crazyfruits_solver.pg` — `immediateScoreShaped`, `deterministicValue`, **`deterministicValueShaped`**, +`buildObservation` (3 per-action planes), `netAction` · `CrazyFruitsDqnCampaign.cs` — +`DenseTargetsFromObservation` (shaped plane ×3 → reward units) · `CrazyFruitsLab.cs` — +`--dense`/`--dense-weight`/`--probe N` (`cf9train` = γ=0 + creation shaping + dense, 400k steps) · +`crazy-fruits-director.ts` — stale-ckpt guard · `DqnDenseTargetsTests.cs` (three-arm ranking proof). diff --git a/docs/prd/CRAZY_FRUITS_SPECIALS_PRD.md b/docs/prd/CRAZY_FRUITS_SPECIALS_PRD.md index 93c6025..92fa484 100644 --- a/docs/prd/CRAZY_FRUITS_SPECIALS_PRD.md +++ b/docs/prd/CRAZY_FRUITS_SPECIALS_PRD.md @@ -1,6 +1,9 @@ # Crazy Fruits specials (striped / wrapped / sugar bomb) — PRD -**Status:** 🔜 planned 2026-07-24 (3-agent investigation: canonical Candy Crush mechanics, engine impact, AI/training impact) +**Status:** ✅ shipped 2026-07-24 (M50.3 closed via stop-loss — gates 2/3 honestly missed). **Superseded on +the net front by M51 ([CRAZY_FRUITS_RANKING_PRD.md](CRAZY_FRUITS_RANKING_PRD.md), 2026-07-25):** `cf9train` +(dense all-action regression + shaped obs plane, 928→1040) passes ALL the §5 gates — +117.2% over random, +gap-share 91%, created 9.57/fired 10.39 — and replaced `cf5train` as the shipped checkpoint. **Owner:** Pieterjan **Milestone:** [PLAN.md](PLAN.md) M50 · extends [CRAZY_FRUITS_PRD.md](CRAZY_FRUITS_PRD.md) (M49, shipped — specials were its §7 out-of-scope item) · branch `m49-crazy-fruits` (owner decision: ONE branch/PR — #38 — for the whole Crazy Fruits arc) diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 476be3e..bd37ee9 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1997,6 +1997,47 @@ zero deadlocks ever; game-over-on-deadlock would never fire). void `cf7train` run (no shield) discarded; training restarts from scratch on the final rules (`cf8train`). Plus the owner-requested on-page KidCity.be credit paragraph. +## M51 — Crazy Fruits missed-opportunity ranking *(2026-07-25; see `CRAZY_FRUITS_RANKING_PRD.md`)* ✅ (shipped same day: `cf9train` = the FIRST Crazy Fruits net to pass ALL gates — +117.2% over random, gap-share 91%, created 9.57/fired 10.39; probe: search-class behavior) + +**Why:** owner observes the shipped net taking a 3-match when the same fruit offers a 4-match (forfeiting the +striped), and asks whether the model can be **punished harder for missing important opportunities**. 4-agent +investigation 2026-07-25 found the root cause is the **loss, not the reward**: the DQN loss regresses only the +chosen action's Q, so a wrong ranking costs zero loss; the reward already prices the 4-match 2.6–3.3× above +the 3-match, but the +40/+60/+100 creation bonus lives only in the training target (no observation feature — +the per-action planes are fire-only), refill-cascade variance drowns the gap, and the web `net` tier is a pure +masked argmax (no search — corrects the "expectimax uses the net" memory). Owner follow-up on special+special +combos **resolved, no change**: combos fire on the swap, so their 1.5–6.4 reward is fully in the immediate +reward and the input plane; combo shaping would double-count (shape what pays later, never what pays now). + +- **M51.0 — Probe + baselines.** ✅ (2026-07-25) Seeded strict 3-vs-4 probe + opportunity/combo take-rates + (creating swap = `immediateScoreShaped > immediateScore`), run BEFORE the obs change. *Results (300 eps): + net takes a creating swap in only **17.6%** of the 3888 offering states (random 14.2%!) vs specials-greedy + **91.4%** — NOT 100%: sometimes firing an existing special honestly beats creating one, so the M51.2 probe + gate is RELATIVE (net ≥ specials-greedy − 5 pts), not an absolute 95%. Combo take-rate: net 48%. Fire-only + e1/e2 sit at 33/38% — confirms creation is invisible to fire-only value.* +- **M51.1 — Web stale-ckpt guard** (replaces the λ re-rank, dropped at design time: the obs change breaks + the old ckpt in the same working tree, and the immediate retrain makes it redundant). Net input width ≠ + observation width ⇒ treated as missing, expectimax fallback. **Gate:** old ckpt + new engine = clean fallback. +- **M51.2 — Retrain with a ranking-aware loss.** NEW shaped-deterministic per-action plane + (`deterministicValueShaped` = refill-free cascade + creation weights; obs 928→1040 — immediate-only shaped + targets would re-create the cascading-3-beats-flat-4 bias) + **dense all-action regression** (taken action + → realized reward; every other legal action → its shaped deterministic value; dense term normalized to + carry weight 1.0 × the realized term's total gradient mass; `DqnOptions.DenseTargets` seam, γ=0-guarded) + + pre-registered margin hinge only if the probe gate fails. 400k steps for `cf5train` comparability. + **Gates:** M50.3 bars (≥+30%/random · created ≥7.3 / fired ≥5.6 · gap-share ≥64%) + probe (final form: + opportune take-rate ≥ e1 − 5 pts — the specials-greedy-relative and absolute-90% bars both died on data: + creation-chasing is not optimal play, strong policies cluster at 50–56%; post-mortem in PRD §4). + ✅ **ALL GATES PASS** (500 eps): net **5666.4 ± 155.2 = +117.2% over random** (cf5train +54.9%) · + **gap-share 91%** (cf5train 43%) · created 9.57 / fired 10.39 · probe opportune take 54.9% vs e1's 55.7% + (raw take 17.6%→35.5%, combo 48%→66%) — random-class behavior became search-class. Margin hinge unused. +- **M51.3 — Escalation.** ❌ Not triggered (gap-share 91% ≥ 64%). Remaining ceiling = the e1→e2 + hold-for-combo gap, out of scope per the M50.3 close-out. +- **M51.4 — Ship.** ✅ `cf9train` → `wwwroot/models/crazyfruits.dqn.ckpt`; round-over bar ~4 000 → ~5 650; + parity pin 481681208 unchanged; docs synced. + +**Rejected up front:** bigger shaping bonuses (reward-hack trap), combo shaping (double-count), γ>0 schedules +(n=2 losses: M49 γ=0.99, M50.3 `cf8train`), regret-prioritized replay (subsumed by dense regression). + ## Testing strategy (cross-cutting, from research) 1. **Known-solved thresholds** as integration tests (median over ≥3 seeds) — slow bucket. diff --git a/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs b/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs index 495ea25..d91059d 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Campaigns/CrazyFruits/CrazyFruitsDqnCampaign.cs @@ -46,8 +46,27 @@ public sealed partial class CrazyFruitsDqnCampaign : DqnScoreCampaign TargetSyncEvery = 1_000, Epsilon = new LinearSchedule(Options.EpsilonStart, 0.05, 30_000), EvalEpisodes = 20, + DenseTargets = Typed.DenseRegression ? DenseTargetsFromObservation : null, + DenseTargetWeight = Typed.DenseTargetWeight, }; + // The observation's shaped per-action plane (deterministicValueShaped(a)/300, 0 = illegal — a legal swap + // always clears something) IS the dense target, converted from plane units to reward units: ×300 undoes + // the plane normalizer, ÷RewardScale matches the env's reward scaling. + private static readonly int ShapedPlaneOffset = + (CrazyFruitsBoard.FruitTypes + CrazyFruitsBoard.SpecialKinds + 1) * CrazyFruitsBoard.Cells + 2 * CrazyFruitsBoard.ActionCount; + + private static float[] DenseTargetsFromObservation(float[] obs) + { + var targets = new float[CrazyFruitsBoard.ActionCount]; + for (int a = 0; a < CrazyFruitsBoard.ActionCount; a++) + { + float plane = obs[ShapedPlaneOffset + a]; + targets[a] = plane > 0f ? plane * 300f / CrazyFruitsEnv.RewardScale : float.NaN; + } + return targets; + } + protected override (double Gate, IReadOnlyList Metrics, string Summary) EvaluateNet(IValueNet net) { var (score, meanReturn) = EvalNet(net); diff --git a/src/MintPlayer.AI.ReinforcementLearning.Campaigns/Shared/DqnScoreOptions.cs b/src/MintPlayer.AI.ReinforcementLearning.Campaigns/Shared/DqnScoreOptions.cs index 4d44674..ad74e48 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Campaigns/Shared/DqnScoreOptions.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Campaigns/Shared/DqnScoreOptions.cs @@ -35,6 +35,14 @@ public sealed record CrazyFruitsDqnOptions : DqnScoreOptions { /// n-step return horizon (1 = single-step DQN; the escalation uses 3). public int NStep { get; init; } = 1; + + /// Dense all-action regression (RANKING PRD M51.2, γ=0 only): every legal action is also + /// regressed toward its creation-shaped deterministic value read from the observation's shaped plane — + /// the loss that makes ranking a 3-match above an available 4-match costly. + public bool DenseRegression { get; init; } + + /// Total gradient mass of the dense term relative to the realized-reward term. + public float DenseTargetWeight { get; init; } = 1.0f; } /// diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs index c6477b0..2b316b7 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DqnTrainer.cs @@ -48,6 +48,21 @@ public sealed record DqnOptions /// public bool NoisyNets { get; init; } + /// + /// γ=0 dense-regression seam (RANKING PRD lever B): given ONE stored observation, per-action supervised + /// targets in reward units — leaves that action unsupervised. When set, every + /// non-NaN action except the sampled one is regressed toward its target in addition to the sampled + /// action's realized-reward term. This is the loss that makes a wrong RANKING costly: chosen-action-only + /// regression can rank a worse action above a better one at zero loss. Requires == 0 + /// (only there is a full-return target computable from the observation alone). + /// + public Func? DenseTargets { get; init; } + + /// Total gradient mass of the dense term relative to the sampled-action term (default 1.0 — + /// equal say). Normalized per supervised entry, so ~30 legal actions/state don't drown the realized + /// reward's refill-expectation signal 30:1. + public float DenseTargetWeight { get; init; } = 1.0f; + public int EvalEvery { get; init; } = 5_000; public int EvalEpisodes { get; init; } = 20; @@ -128,6 +143,9 @@ public static DqnResult Train(IEnvironment env, DqnOptions options int obsDim = ((BoxSpace)env.ObservationSpace).Dimensions; int actionCount = ((DiscreteSpace)env.ActionSpace).N; + if (options.DenseTargets is not null && options.Gamma != 0) + throw new ArgumentException("DenseTargets requires Gamma == 0: a dense per-action target is the full return only when nothing is bootstrapped."); + DqnTrainingState state; float[] obs; if (resume is null) @@ -288,8 +306,37 @@ private static float TrainStep(IValueNet online, IValueNet target, Adam adam, Re } adam.ZeroGrad(); - var q = online.Forward(new Tensor(batch.Obs, batch.Size, batch.ObsDim)).Gather(batch.Actions); + var qAll = online.Forward(new Tensor(batch.Obs, batch.Size, batch.ObsDim)); + var q = qAll.Gather(batch.Actions); var loss = q.HuberLoss(new Tensor(targets, batch.Size)); + if (options.DenseTargets is not null) + { + int actions = qAll.Cols; + // Unsupervised entries get target := prediction — zero Huber value AND zero gradient — so one + // full-matrix loss supervises exactly the non-NaN, non-sampled actions. + var dense = (float[])qAll.Data.Clone(); + var row = new float[batch.ObsDim]; + int supervised = 0; + for (int i = 0; i < batch.Size; i++) + { + Array.Copy(batch.Obs, i * batch.ObsDim, row, 0, batch.ObsDim); + var t = options.DenseTargets(row); + for (int a = 0; a < actions; a++) + { + if (a == batch.Actions[i] || float.IsNaN(t[a])) continue; // sampled action keeps its realized target + dense[i * actions + a] = t[a]; + supervised++; + } + } + if (supervised > 0) + { + // HuberLoss means over B·A entries while the gathered loss means over B; rescale so the dense + // term's TOTAL gradient mass is DenseTargetWeight × the sampled term's, however many actions + // happen to be supervised. + float scale = options.DenseTargetWeight * batch.Size * actions / supervised; + loss = loss.Add(qAll.HuberLoss(new Tensor(dense, batch.Size, actions)).MulScalar(scale)); + } + } loss.Backward(); adam.ClipGradNorm(options.MaxGradNorm); adam.Step(); diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsBoard.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsBoard.cs index d33143b..00e0f5a 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsBoard.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsBoard.cs @@ -18,9 +18,10 @@ public sealed class CrazyFruitsBoard public const int Cells = Size * Size; public const int ActionCount = 2 * Size * (Size - 1); // 112 public const int SpecialKinds = 4; // stripedH, stripedV, wrapped, bomb (armed is internal-only) - // 928: 6 fruit planes + 4 kind planes + would-act plane + 2 per-action feature planes (immediate score, - // deterministic cascade value ÷300 — SPECIALS PRD §3.5). - public const int ObservationSize = (FruitTypes + SpecialKinds + 1) * Cells + 2 * ActionCount; + // 1040: 6 fruit planes + 4 kind planes + would-act plane + 3 per-action feature planes (immediate score, + // deterministic cascade value, creation-shaped deterministic value — ÷300; SPECIALS PRD §3.5 + RANKING + // PRD lever B). + public const int ObservationSize = (FruitTypes + SpecialKinds + 1) * Cells + 3 * ActionCount; private readonly PgCrazyFruits _core = new(); @@ -82,8 +83,8 @@ public bool[] LegalMask() /// Re-deal the current fruit multiset until no instant match and ≥1 legal swap (the deadlock rule). public void Reshuffle() => _core.reshuffleBoard(); - /// The 672-dim observation: 6 one-hot fruit planes + the would-match plane + per-action - /// immediate-score and deterministic-cascade-value planes (f64 core → float32 net). + /// The -dim observation: fruit/kind/would-match planes + the three + /// per-action value planes (f64 core → float32 net). public float[] BuildObservation() { var core = _core.buildObservation(); @@ -124,6 +125,10 @@ public int RandomAction(ulong policySeed, int step) /// Deterministic cascade value of a swap (consumes no RNG; restores the board). public int DeterministicValue(int action) => _core.deterministicValue(action); + /// Deterministic cascade value plus the creation-shaping weights over the whole refill-free + /// cascade — the shaped observation plane and the dense-regression target (RANKING PRD lever B). + public int DeterministicValueShaped(int action) => _core.deterministicValueShaped(action); + // ── Host-only helpers ──────────────────────────────────────────────────────────────────────────────────── /// Overwrite the grid for directed tests (row-major, values 1..6). Score/moves are untouched. diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs index 20a206d..7da7acf 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/CrazyFruitsEnv.cs @@ -9,14 +9,15 @@ namespace MintPlayer.AI.ReinforcementLearning.Environments.CrazyFruits; /// one swap: apply the chosen adjacent swap, resolve cascades + refill in pure compute, return the new board. /// /// Action = one of the 112 adjacent swaps; only match-producing swaps are legal -/// ( — masking is mandatory here, PRD §3.3). Observation = 448 floats (6 -/// one-hot fruit planes + the would-match plane). Reward = points/30 (a plain 3-match ⇒ 1.0), no step -/// penalty. Episodes never terminate (a dead board reshuffles inside the engine); they truncate at the move -/// budget, so the learner bootstraps from the final state — the score-maximizing framing. +/// ( — masking is mandatory here, PRD §3.3). Observation = +/// floats (fruit/kind/would-match planes + three per-action value planes). +/// Reward = points/. Episodes never terminate (a dead board reshuffles inside the +/// engine); they truncate at the move budget, so the learner bootstraps from the final state — the +/// score-maximizing framing. /// public sealed class CrazyFruitsEnv : IEnvironment, IActionMaskProvider, IStatefulEnvironment { - public const int ObservationSize = CrazyFruitsBoard.ObservationSize; // 448 + public const int ObservationSize = CrazyFruitsBoard.ObservationSize; // 1040 public const int ActionCount = CrazyFruitsBoard.ActionCount; // 112 /// Reward normalization. Re-picked for specials (SPECIALS PRD §3.5): random-play means ~86 /// points/move with auto-firing specials, so ÷100 keeps a typical move ≈ O(1) and a bomb+bomb board @@ -53,6 +54,7 @@ private static string[] BuildObservationLabels() var actions = BuildActionLabels(); foreach (var label in actions) labels.Add($"Immediate points of \"{label}\" (÷300)"); foreach (var label in actions) labels.Add($"Guaranteed cascade points of \"{label}\" (÷300)"); + foreach (var label in actions) labels.Add($"Guaranteed cascade points + creation bonus of \"{label}\" (÷300)"); return [.. labels]; } diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/polyglot/crazyfruits_solver.pg b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/polyglot/crazyfruits_solver.pg index 614a937..34a5c94 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/polyglot/crazyfruits_solver.pg +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/CrazyFruits/polyglot/crazyfruits_solver.pg @@ -751,10 +751,11 @@ class PgCrazyFruits { this.generateBoard() } - // ── Observation (SPECIALS PRD §3.5) — 928 floats: 6 one-hot fruit planes + 4 kind planes + a would-act - // plane + two per-action feature planes (immediate score, deterministic cascade value — both ÷300; they - // price creations, activations and combos, the exact lever that gated M49). A colored special sets BOTH - // its fruit plane and its kind plane; a bomb only its kind plane. No plane for Armed (never stable). + // ── Observation (SPECIALS PRD §3.5, RANKING PRD lever B) — 1040 floats: 6 one-hot fruit planes + 4 kind + // planes + a would-act plane + three per-action feature planes (immediate score, deterministic cascade + // value, creation-SHAPED deterministic value — all ÷300; they price creations, activations and combos, + // the exact lever that gated M49; the shaped plane is also the dense-regression target source). A colored + // special sets BOTH its fruit plane and its kind plane; a bomb only its kind plane. No Armed plane. fn buildObservation(): List { var obs: List = List() for f in 1..(FruitTypes + 1) { @@ -776,6 +777,9 @@ class PgCrazyFruits { for a in 0..ActionCount { obs.add(if this.swapIsLegal(a) { this.deterministicValue(a) / 300.0 } else { 0.0 }) } + for a in 0..ActionCount { + obs.add(if this.swapIsLegal(a) { this.deterministicValueShaped(a) / 300.0 } else { 0.0 }) + } return obs } @@ -904,6 +908,40 @@ class PgCrazyFruits { return v } + // deterministicValue plus the creation-shaping weights (+40/+60/+100) over the WHOLE refill-free cascade + // (RANKING PRD lever B) — the shaped observation plane and the dense-regression target. Unlike + // immediateScoreShaped it also prices cascade-made creations, so it matches the training reward's shaping + // term up to refill noise; an immediate-only shaped target would credit a cascading 3-match's cascade but + // not a flat 4-match's striped — the exact bias this milestone removes. + fn deterministicValueShaped(action: i32): i32 { + if !this.swapIsLegal(action) { return 0 } + var saved: List = List() + for i in 0..Cells { saved.add(this.grid[i]) } + let savedA = this.moveCellA + let savedB = this.moveCellB + let savedCenter = this.comboCenter + let savedCombo = this.stagedCombo + let savedSteps = this.lastCascadeSteps + let savedCS = this.moveCreatedStriped + let savedCW = this.moveCreatedWrapped + let savedCB = this.moveCreatedBombs + let savedSF = this.moveSpecialsFired + this.stageSwap(action, this.cellB(action)) + var v = this.resolveCascades(false) + v += 40 * this.moveCreatedStriped + 60 * this.moveCreatedWrapped + 100 * this.moveCreatedBombs + for i in 0..Cells { this.grid[i] = saved[i] } + this.moveCellA = savedA + this.moveCellB = savedB + this.comboCenter = savedCenter + this.stagedCombo = savedCombo + this.lastCascadeSteps = savedSteps + this.moveCreatedStriped = savedCS + this.moveCreatedWrapped = savedCW + this.moveCreatedBombs = savedCB + this.moveSpecialsFired = savedSF + return v + } + // 1-ply expectimax tier: maximize the deterministic cascade value over legal swaps. fn expectimaxAction(): i32 { var best = -1 diff --git a/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-director.ts b/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-director.ts index fd697bd..3d805b1 100644 --- a/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-director.ts +++ b/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits-director.ts @@ -25,6 +25,13 @@ export class CrazyFruitsDirector { constructor(private readonly game: CrazyFruitsGame) { void loadCrazyFruitsNet().then(net => { + // Stale-ckpt guard (RANKING PRD M51.1): a net trained on an older observation layout cannot forward + // against the current engine — treat it as missing (expectimax fallback) instead of erroring per move. + const obsSize = this.game.board.buildObservation().length; + if (net && net.inputSize !== obsSize) { + console.warn(`crazy-fruits: checkpoint input ${net.inputSize} ≠ observation ${obsSize} — stale checkpoint, falling back to expectimax`); + net = null; + } this.net = net; this.netStatus = net ? 'ready' : 'missing'; }); diff --git a/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits.ts b/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits.ts index 29e5cc3..c9fe046 100644 --- a/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits.ts +++ b/src/RLDemo.Web/ClientApp/src/app/crazy-fruits/crazy-fruits.ts @@ -83,10 +83,10 @@ export class CrazyFruits implements AfterViewInit { }); } - /** Measured 500-episode tier means (SPECIALS PRD M50.2/.3) — the round-over screen's challenge lines. */ + /** Measured 500-episode tier means (RANKING PRD M51.2 `cf9train`) — the round-over screen's challenge lines. */ private static readonly ROUND_BARS = [ 'random plays ~2 600 a round', - 'the trained net ~4 000', + 'the trained net ~5 650', 'expectimax-2 plays ~8 000', ]; diff --git a/src/RLDemo.Web/wwwroot/models/crazyfruits.dqn.ckpt b/src/RLDemo.Web/wwwroot/models/crazyfruits.dqn.ckpt index 8dcc41d..57806cc 100644 --- a/src/RLDemo.Web/wwwroot/models/crazyfruits.dqn.ckpt +++ b/src/RLDemo.Web/wwwroot/models/crazyfruits.dqn.ckpt @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0a6d53d56f8eca870b769b1d62ab07e74ce817e642bab1696768d476de5ceedd -size 1330699 +oid sha256:88ff5fd73c62d1e07242df7460dc7896dc4bece336fe24ff259819a47c24d54a +size 1445387 diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEngineTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEngineTests.cs index f42c291..d743fb7 100644 --- a/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEngineTests.cs +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsEngineTests.cs @@ -266,6 +266,8 @@ public void Observation_OneHotPlanesMatchGrid_WouldMatchPlaneMatchesMask() Assert.Equal(board.ImmediateScore(a) / 300f, obs[baseOffset + a], 5); float expectedDet = mask[a] ? board.DeterministicValue(a) / 300f : 0f; Assert.Equal(expectedDet, obs[baseOffset + CrazyFruitsBoard.ActionCount + a], 5); + float expectedShaped = mask[a] ? board.DeterministicValueShaped(a) / 300f : 0f; + Assert.Equal(expectedShaped, obs[baseOffset + 2 * CrazyFruitsBoard.ActionCount + a], 5); } } diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsSpecialsTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsSpecialsTests.cs index fc2ea3a..bcf491d 100644 --- a/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsSpecialsTests.cs +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/CrazyFruitsSpecialsTests.cs @@ -608,4 +608,35 @@ public void Observation_CarriesKindPlanes_AndBothPlanesForColoredSpecials() Assert.Equal(1f, obs[(6 + 3) * cells + 5 * Size + 5]); // bomb plane (kind 4 → plane 9) for (int f = 0; f < 6; f++) Assert.Equal(0f, obs[f * cells + 5 * Size + 5]); } + + // ── deterministicValueShaped (RANKING PRD lever B) ───────────────────────────────────────────────────── + + [Fact] + public void DeterministicValueShaped_AddsCreationWeight_ForAStripedCreatingSwap() + { + var board = BoardWith((7, 0, 4), (7, 1, 4), (7, 3, 4), (6, 2, 4)); // VSwap(6,2) → horizontal 4 → striped + int action = VSwap(6, 2); + Assert.True(board.DeterministicValueShaped(action) >= board.DeterministicValue(action) + 40, + "a striped-creating swap's shaped deterministic value must carry at least the +40 creation weight"); + } + + [Fact] + public void DeterministicValueShaped_EqualsDeterministicValue_WhenNothingIsCreated() + { + var board = BoardWith((7, 0, 4), (7, 1, 4), (6, 2, 4)); // VSwap(6,2) → plain 3-match + int action = VSwap(6, 2); + Assert.Equal(board.DeterministicValue(action), board.DeterministicValueShaped(action)); + } + + [Fact] + public void DeterministicValueShaped_IsPure_AndZeroForIllegal() + { + var board = BoardWith((7, 0, 4), (7, 1, 4), (7, 3, 4), (6, 2, 4)); + var before = board.GridSnapshot(); + int scoreBefore = board.Score; + _ = board.DeterministicValueShaped(VSwap(6, 2)); + Assert.Equal(before, board.GridSnapshot()); + Assert.Equal(scoreBefore, board.Score); + Assert.Equal(0, board.DeterministicValueShaped(HSwap(0, 0))); // base pattern: no match, illegal + } } diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/DqnDenseTargetsTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/DqnDenseTargetsTests.cs new file mode 100644 index 0000000..04e2104 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/DqnDenseTargetsTests.cs @@ -0,0 +1,100 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Environments; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Schedules; +using MintPlayer.AI.ReinforcementLearning.Core.Training; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +/// +/// RANKING PRD M51.2: the γ=0 dense all-action regression () must fix +/// exactly the failure chosen-action-only regression cannot see — a policy that never samples the best +/// action still learns its Q, so the argmax ranking comes out right. +/// +public class DqnDenseTargetsTests +{ + /// Single constant state, 3 actions with fixed rewards, episodes truncate on a move budget — + /// the smallest environment where per-action ranking is the whole problem. + private sealed class ThreeArmEnv : IEnvironment + { + public static readonly float[] ArmRewards = [0.1f, 1.0f, 0.4f]; + private int _moves; + + public Space ObservationSpace { get; } = new BoxSpace(0f, 1f, 4); + public Space ActionSpace { get; } = new DiscreteSpace(3); + + public (float[] Observation, EnvInfo Info) Reset(ulong? seed = null) + { + _moves = 0; + return (Obs(), EnvInfo.Empty); + } + + public StepResult Step(int action) + { + _moves++; + return new StepResult(Obs(), ArmRewards[action], false, _moves >= 8, EnvInfo.Empty); + } + + public string RenderString() => $"moves={_moves}"; + + private static float[] Obs() => [1f, 0.5f, 0.25f, 0.125f]; + } + + [Fact] + public void DenseTargets_TeachRankingOfNeverSampledActions() + { + // Pure-greedy behavior (ε ≡ 0) from a random init: without dense targets the policy locks onto + // whatever arm the init favors and the other arms' Q-values are never updated. The dense targets + // must rank all three arms correctly anyway. + var options = new DqnOptions + { + Hidden = [16], + Gamma = 0, + LearningRate = 1e-2f, + BufferCapacity = 512, + BatchSize = 16, + WarmupSteps = 32, + Epsilon = new LinearSchedule(0, 0, 1), + MaxSteps = 600, + EvalEvery = 10_000, // never — keep the run pure data collection + training + DenseTargets = _ => ThreeArmEnv.ArmRewards, + }; + + var result = DqnTrainer.Train(new ThreeArmEnv(), options, new SeedSequence(7)); + + var q = result.Agent.QValues([1f, 0.5f, 0.25f, 0.125f]); + Assert.True(q[1] > q[2] && q[2] > q[0], + $"dense targets must rank Q(1) > Q(2) > Q(0); got [{q[0]:F3}, {q[1]:F3}, {q[2]:F3}]"); + } + + [Fact] + public void DenseTargets_NaN_LeavesActionsUnsupervised() + { + // Only arm 1 gets a dense target; the never-sampled, never-supervised arms must not converge to it. + var options = new DqnOptions + { + Hidden = [16], + Gamma = 0, + LearningRate = 1e-2f, + BufferCapacity = 512, + BatchSize = 16, + WarmupSteps = 32, + Epsilon = new LinearSchedule(0, 0, 1), + MaxSteps = 600, + EvalEvery = 10_000, + DenseTargets = _ => [float.NaN, 1.0f, float.NaN], + }; + + var result = DqnTrainer.Train(new ThreeArmEnv(), options, new SeedSequence(7)); + + var q = result.Agent.QValues([1f, 0.5f, 0.25f, 0.125f]); + Assert.True(Math.Abs(q[1] - 1.0f) < 0.15f, $"supervised arm must approach 1.0; got {q[1]:F3}"); + } + + [Fact] + public void DenseTargets_RequireGammaZero() + { + var options = new DqnOptions { Gamma = 0.99, DenseTargets = _ => ThreeArmEnv.ArmRewards }; + Assert.Throws(() => DqnTrainer.Train(new ThreeArmEnv(), options, new SeedSequence(1))); + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs index 5ae8710..e1e13af 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CrazyFruits/CrazyFruitsLab.cs @@ -38,12 +38,22 @@ public static void Run(string[] args) return; } + int probe = a.Int("--probe", 0); + if (probe > 0) + { + RunProbe(probe, moveBudget, netPath); + return; + } + var options = new CrazyFruitsDqnOptions { Seed = seed, ChunkSteps = chunkSteps, TargetSteps = targetSteps, EvalEpisodes = evalEpisodes, LearningRate = learningRate, EpsilonStart = explore, Hidden = hidden, Gamma = gamma, Grow = grow, GrowEvery = growEvery, NStep = a.Int("--nstep", 1), + // RANKING PRD M51.2: dense all-action regression toward the shaped observation plane (γ=0 only). + DenseRegression = a.Has("--dense"), + DenseTargetWeight = a.Flt("--dense-weight", 1.0f), }; // Shaping on the TRAIN env only; the eval env scores the bare game, so gates stay honest. // Default (γ=0 recipe): creation bonuses. --pbrs (the §3.6 escalation, with --gamma 0.5 --nstep 3): @@ -145,6 +155,115 @@ private static void RunBaselines(int episodes, int moveBudget, ulong seed, strin } } + /// + /// The M51.0 missed-opportunity probe (RANKING PRD §4): every policy is asked what it WOULD play on the + /// SAME random-walk states (seeded, never the eval line 5000+e), without applying. A "creating" swap is + /// one whose shaped immediate score exceeds its plain immediate score (it makes a special at step 0 — + /// the owner's reported scenario). Metrics per policy: + /// - opportune take-rate = P(picks a creating swap | a creating swap is the SHAPED-DETERMINISTIC-OPTIMAL + /// move) — the gate metric: an opportunity only counts as "missed" when creating was the best move + /// (the raw any-creating take-rate punished good play: expectimax-2 scores 8098 at a 38% raw rate, + /// while specials-greedy's immediate-only 91% scores 3903); + /// - oracle match = P(action == argmax deterministicValueShaped) — plan fidelity, reported not gated; + /// - raw take-rate + combo take-rate, kept for continuity with the M51.0 baseline table. + /// + private static void RunProbe(int episodes, int moveBudget, string netPath) + { + DuelingQNet? net = null; + if (File.Exists(netPath)) + { + using var stream = File.OpenRead(netPath); + net = DuelingQNetCheckpoint.Load(stream); + if (net.InputSize != CrazyFruitsEnv.ObservationSize) + { + Console.WriteLine($" (net at {netPath} has input width {net.InputSize} ≠ {CrazyFruitsEnv.ObservationSize} — skipped)"); + net = null; + } + } + var agent = net is null ? null : new GreedyQAgent(net, CrazyFruitsEnv.ActionCount); + + var policies = new List<(string Name, Func Act)> + { + ("random", (b, s) => b.RandomAction(0xABBAUL, s)), + ("greedy", (b, _) => b.GreedyAction()), + ("specials-greedy", (b, _) => b.SpecialsGreedyAction()), + ("expectimax-1", (b, _) => b.ExpectimaxAction()), + ("expectimax-2", (b, _) => b.Expectimax2Action()), + }; + if (agent is not null) + policies.Add(("net", (b, _) => agent.Act(b.BuildObservation(), b.LegalMask(), greedy: true))); + + int opportuneStates = 0, opportunityStates = 0, comboStates = 0, statesSeen = 0; + var opportuneTaken = new long[policies.Count]; + var opportunityTaken = new long[policies.Count]; + var comboTaken = new long[policies.Count]; + var oracleMatch = new long[policies.Count]; + + var env = new CrazyFruitsEnv(moveBudget); + var creating = new bool[CrazyFruitsEnv.ActionCount]; + for (int e = 0; e < episodes; e++) + { + env.Reset((ulong)(9_000 + e)); + var b = env.Board; + for (int move = 0; move < moveBudget; move++) + { + int stateIndex = e * moveBudget + move; + var mask = b.LegalMask(); + bool anyCreating = false, anyCombo = false; + int bestAction = -1, bestShaped = 0; + for (int a = 0; a < CrazyFruitsEnv.ActionCount; a++) + { + creating[a] = mask[a] && b.ImmediateScoreShaped(a) > b.ImmediateScore(a); + if (creating[a]) anyCreating = true; + if (!mask[a]) continue; + int shaped = b.DeterministicValueShaped(a); + if (shaped > bestShaped) { bestShaped = shaped; bestAction = a; } + } + for (int a = 0; a < CrazyFruitsEnv.ActionCount && !anyCombo; a++) + { + if (!mask[a]) continue; + var (cellA, cellB) = b.SwapCells(a); + anyCombo = b.Kind(cellA) > 0 && b.Kind(cellB) > 0; + } + + statesSeen++; + bool opportune = bestAction >= 0 && creating[bestAction]; + if (opportune) opportuneStates++; + if (anyCreating) opportunityStates++; + if (anyCombo) comboStates++; + + for (int p = 0; p < policies.Count; p++) + { + int action = policies[p].Act(b, stateIndex); + if (action < 0) continue; + if (action == bestAction) oracleMatch[p]++; + if (anyCreating && creating[action]) { opportunityTaken[p]++; if (opportune) opportuneTaken[p]++; } + if (anyCombo) + { + var (cellA, cellB) = b.SwapCells(action); + if (b.Kind(cellA) > 0 && b.Kind(cellB) > 0) comboTaken[p]++; + } + } + + // The walk itself is uniform-random over legal swaps — policy-neutral state coverage. + b.ApplySwap(b.RandomAction(0xCF51UL, stateIndex)); + } + } + + Console.WriteLine($"Crazy Fruits opportunity probe: {episodes} episodes × {moveBudget} random-walk moves (seeds 9000+e)"); + Console.WriteLine($" states {statesSeen}: creating swap available {opportunityStates}, creating swap is the " + + $"shaped-optimal move {opportuneStates} | special+special legal {comboStates}"); + Console.WriteLine($" {"policy",-18} {"opportune take",16} {"raw take",10} {"oracle match",14} {"combo take",12}"); + for (int p = 0; p < policies.Count; p++) + { + string Rate(long taken, int total) => total == 0 ? "n/a" : $"{(double)taken / total:P1}"; + Console.WriteLine($" {policies[p].Name,-18} {Rate(opportuneTaken[p], opportuneStates),16} " + + $"{Rate(opportunityTaken[p], opportunityStates),10} {Rate(oracleMatch[p], statesSeen),14} " + + $"{Rate(comboTaken[p], comboStates),12}"); + } + Console.WriteLine(" gate (RANKING PRD M51.2): net opportune take-rate ≥ 90% (oracle = 100% by construction)."); + } + private static (string, double, double) RunPolicy(string name, int episodes, int moveBudget, Func policy) { double sum = 0, sumSq = 0;