[Blocker 1/6] Make stall fix deterministic

[mgrz18]

Symptom

Same env vars, same code: bossMainloop receives 1 DONE per run sometimes, 152 DONEs other times.

• 2-3 of 5 runs stall at DONEs=1 (animation never advances)
• 1-2 of 5 runs reach DONEs=152 (game advances ~150 frames)

What we know

• __scTaskComplete (/tmp/ge_decomp/src/sched.c:406-433) forwards t->msg verbatim to clientQ.
• Real msg is &localGfxDoneMsg — a stack OSScMsg in bossMainloop, NOT the static g_bossGfxDoneMsg.
• Bogus OSTask submissions (data_ptr=0x7FC00000) have garbage in their enclosing OSScTask msg field.
• Our heal at lib/N64ModernRuntime/ultramodern/src/events.cpp:614+ overwrites bogus msg with hardcoded 0x803B38B8.
• That hardcoded address is NOT &g_bossGfxDoneMsg — was a guess. The OSScMsg there is uninitialised → bossMainloop reads gen.type=0 → switch falls through default → no decrement → stall.

What was tried

• ❌ Hardcoded fallback 0x803B38B8 — works sometimes, fails others
• ❌ Stamping gen.type=2 at hardcoded addr — caused regression (memory corruption?)
• ❌ Unconditional heal on every submit — overwrote healthy submissions, broke flow
• ❌ Recv-side gen.type ∈ [1..5] filter — too aggressive, dropped valid msgs

Try next

1. Lazy-learn &localGfxDoneMsg from FIRST healthy submission (no hardcoded fallback). Defer healing until learned.
2. Once learned, stamp gen.type=2 at the learned address as defensive measure.
3. Investigate timing race between our heal write and game-side __scTaskComplete.

Files

• lib/N64ModernRuntime/ultramodern/src/events.cpp:568-700
• lib/N64ModernRuntime/ultramodern/src/mesgqueue.cpp:206-224
• /tmp/ge_decomp/src/sched.c:406-433 (reference)

Acceptance criteria

Every run has DONEs ≥ 100 (currently 30-40% rate).

[mgrz18]

Investigation update — orchestrator iteration #1 (2026-05-05)

What we measured

Ran harness 5 times with current code (commit d4a1189 in N64ModernRuntime):
```
Run 1: dones=5 top_nz=174402 fbs_visible=3 heal=0 learn=0
Run 2: dones=3 top_nz=1 fbs_visible=0 heal=0 learn=0
Run 3: dones=2 top_nz=1 fbs_visible=0 heal=1 learn=0
Run 4: dones=2 top_nz=1 fbs_visible=0 heal=1 learn=0
Run 5: dones=1 top_nz=1 fbs_visible=0 heal=1 learn=0
```

Hypothesis #1 (FALSIFIED)

Removed hardcoded g_known_done_msg_ptr = 0x803B38B8 (set to 0). Result: WORSE — dones=[1,3,1,0,2], 0/5 visible.
Reverted.

Diagnostic finding

Added cur_msg logging — observed in 100% of GFX submissions: `cur_msg = 0x803B38B8`. The hardcoded fallback IS the right address. Agent A's earlier analysis was wrong.

`learn_count = 0` only because the value never CHANGES (initial = observed), so the "learned" log line condition is false. The learn IS happening (no-op assign).

Stack-deeper finding

Added sp_complete logging:

• sp_complete fires 26 times in 60s (includes audio + gfx tasks)
• dp_complete fires 3 times (gfx only)
• bossMainloop receives 20× type=1 (RETRACE), 0× type=2 (DONE)

So sp/dp_complete fire correctly, but the game's recompiled __scHandleSP → __scTaskComplete → osSendMesg(clientQ, doneMsg) chain doesn't produce DONEs in clientQ.

Implication

The stall is NOT a heal-message corruption bug. It's a recompiled scheduler bug: the game's __scTaskComplete doesn't successfully forward type=2 DONEs even when sp/dp completion signals arrive.

Where to look next (NOT recompilation expert here)

• Game-side __scHandleSP and __scTaskComplete C decomp at /tmp/ge_decomp/src/sched.c
• The recompiled versions of those functions in our binary
• Maybe `OS_EVENT_SP` → game's sp_msg pickup is broken at the recomp level (the msg is 0x29B which is the SP-DONE ID, not a pointer; if the game expects pointer dereference, this would fail)

This blocker likely needs N64Recomp / scheduler internals expertise. Marking as needs-specialist.

[mgrz18]

Iteration #2 — direct DONE injection (2026-05-05)

Approach

Bypass the recompiled __scHandleSP → __scTaskComplete → osSendMesg(clientQ, doneMsg) chain entirely. Inject a DONE msg directly into clientQ (gfxFrameMsgQ at 0x80141C90) with DONE_MSG = 0x803B38B8 (the static g_bossGfxDoneMsg) right after each gfx task processes.

Code: lib/N64ModernRuntime/ultramodern/src/events.cpp:368+ (commit e59480c on mac-port).

Verification of approach

Dumped queue state before inject:

[clientQ_state] @0x80141C90: validCount=0 first=6 msgCount=32 msgArr=0x80141CA8

Confirms 0x80141C90 IS a real OSMesgQueue (capacity 32, valid msg array). osSendMesg returns 0 (success) on every call.

Results (8 runs × 90s)

Run 1: dones=155          ✓ WIN
Run 2: dones=3            stalled
Run 3: dones=2            stalled
Run 4: dones=2            stalled
Run 5: dones=3 top=132k   ✓ WIN (visible content)
Run 6: dones=2            stalled
Run 7: dones=2            stalled
Run 8: dones=1            stalled

2/8 = 25% win rate, marginal improvement vs baseline 1/5 = 20%. Acceptance criterion (5/5 ≥ 100 DONEs) NOT met.

Why it doesn't fully work

bossMainloop's outer-loop NOBLOCK drain at workbench_theboy.c:616:

while (osRecvMesg(&gfxFrameMsgQ, ..., OS_MESG_NOBLOCK) == 0) {} // drain stale msgs

…eats every queue msg at iteration boundary, including our injection if it landed there. Inject only takes effect when it lands during the inner loop's OS_MESG_BLOCK recv where pendingGfx > 0.

In stalled runs, the game never reaches pendingGfx > 0 (something earlier in the boot sequence fails). The injection helps once that point is reached, but doesn't break through the prior bottleneck.

What's left for a specialist

The real fix is in the recompiled binary's __scHandleSP/__scTaskComplete (game-side scheduler, not our runtime). N64Recomp/scheduler internals expertise required. Files to inspect:

• RecompiledFuncs/sched.c (recompiled __scTaskComplete)
• lib/N64ModernRuntime/N64Recomp/src/main.cpp (recompiler — verify osSendMesg-style instructions are correctly translated)

Tried + falsified hypotheses (this iteration)

• ❌ Remove hardcoded 0x803B38B8 fallback → worse (0/5 visible)
• ❌ Single inject per task → 0/5 in last test
• ⚠️ 4× inject per task → 2/8 (current)
• ⚠️ Inject more aggressively (8x, etc.) — not tested, low expected gain
• 🚧 Continuous inject thread — would help with timing window but might over-fill queue

[mgrz18]

Refuted by runtime instrumentation — this was a red herring

Re-investigated with runtime-side logging in lib/N64ModernRuntime/ultramodern/src/mesgqueue.cpp (do_send/do_recv on the boss clientQ 0x80141C90). DONE delivery is NOT broken:

• __scTaskComplete generates type=2 DONEs, do_send enqueues them to clientQ, and do_recv returns them to bossMainloop with gen.type=2 intact (not overwritten, not starved).
• With injection on, the boss received 89 type=2 DONEs in 60s — gfx-task completion was never the limiter.

Two measurement traps invalidated the prior attempts here

1. The harness dones metric is capped at ≤5. It only counts funcs_0.c:6959 (stamp_log<=5). The historic "152 DONEs" came from since-reverted instrumentation that is not in HEAD. So a working state and a broken state both read ~2–5 — the "1 vs 152" variance was a metric artifact.
2. patches/*.c are no-ops in this build (SKIP_PATCHES=ON + missing llvm-objcopy). The build compiles the pre-generated RecompiledPatches/patches.c, so any boss-side "heal"/instrumentation edit to patches/workbench_theboy.c never actually ran.

The real gate (fixed elsewhere)

g_MainStageNum=-1 is the normal in-stage idle value, not the bug. The game never advanced because g_StageNum stayed LEVELID_TITLE(90): the -level_NN arg was parsed only to set autostart and then discarded, and osPiReadIo_recomp served the 0x00FFB000 token region from a hardcoded string with no -level_ token, so tokenFind(1,"-level_") always returned NULL. The game sat on the title forever; injecting DONEs couldn't help because the stage was never selected.

Fixed in #7 (the game now leaves the title and attempts a real level load). Full evidence + dead-ends in docs/INVESTIGATION.md (2026-06-15 sections).

Closing as superseded. The scheduler/DONE path is healthy; the next real frontier is the level-load crash + the F3D_Gold rendering blockers (#2–#6).