feat: integrate MTPLX 2.4.0 M5 runtime stack - #5
Draft
PhilipJohnBasile wants to merge 34 commits into
Draft
Conversation
…-MoE fusion, GDN post-conv fusion, row-owned router (PR youssofal#174) Adopted from davidtai's PR youssofal#174: compiled K1 target-prefix verify route via mx.compile fixed graphs, 3-stage whole-MoE fusion, GDN post-conv fusion, moepack layout, row-owned router, and combine-tail — 196-207 tok/s on 35B-A3B 4-bit (his M5 receipts, our suite + A/B gate rerun). All routes are env-gated and flags-off byte-identical; +221 new tests. Credit: David Tai (github.com/davidtai), PR youssofal#174.
…ragged KV, fold-in repair, AR row-packing (PR youssofal#200) Adopted from davidtai's PR youssofal#200 (delta over his PR youssofal#174 base): fixed-shape cohort scheduler with ragged KV, speculative fold-in repair, continuous batching with refill, and AR row-packing — 813 tok/s aggregate at 256 streams on 35B-A3B with per-stream byte-identity gates. Credit: David Tai (github.com/davidtai), PR youssofal#200.
…oussofal#208) Adopted from davidtai's PR youssofal#208: MTPLX_PROJ_REQUANT=q4 full-residency Hy3 lane (43-48 tok/s on his M5 receipts), corrected Hy3 MTP draft contract to post-final-norm hidden (0.773 vs 0.387 next-token agreement — matches the family contract our 27B/35B head training uses), env-flag parser unification, MTP payload guards, snapshot-free rejection repair with loud failure preserved for recurrent caches (Qwen lanes verified unaffected), decode-trace tolerance, and metadata scrub. Conflict with the PR youssofal#174 compiled route resolved by keeping the route guard outer and his _skip_verify_snapshot() helper inner. Credit: David Tai (github.com/davidtai), PR youssofal#208.
…tensor-leaf decode (PR youssofal#195) Adopted from davidtai's PR youssofal#195: Laguna S-2.1 (Poolside) mixed-precision 4-bit support as a target-only AR runtime — exact-pin config detection, LagunaARRuntime, compiled tensor-leaf decode step, 9 env-gated fused kernels (67.6 tok/s on his M5 receipts), system-memory preflight, and the batched-decode AR lane for target-only runtimes (superset of the PR youssofal#200 driver — hidden-state consumption now conditional on the spec lane). Integration deltas by us: batched_decode unified on his AR-lane superset; runtime.py kernel-config block guarded so Laguna skips the qwen3-next stack while keeping the PR youssofal#174/youssofal#208 lanes for Qwen models; _apply_runtime_compatibility_mode normalized for the string-tier compatibility contract (fixes bare 'mtplx start' onboarding crash his branch had — caught by main's regression suite); dashboard bundles rebuilt from source with bun (no contributor-built binaries shipped). Credit: David Tai (github.com/davidtai), PR youssofal#195.
…ion (ruff F401)
…nk-splitter bare-marker leak (youssofal#196/youssofal#197 root causes) Live repro tonight (chess-build baseline + 24-run grid): a multi-file parallel tool-call turn that hits max_tokens loses the in-flight call silently AND reports finish_reason=tool_calls — the agent client treats the batch as complete (files silently missing, projects broken, 'weird tool calls'). bigctx grid: 6/6 runs surfaced ONE of five requested calls with finish=tool_calls. Now: engine finish=length is preserved on tool-call turns (stats.tool_calls_truncated_by_length marks the swallow) so agent clients continue the turn per protocol. Second root cause (leak variant): the think/content splitter treated bare 'function=' / 'parameter=' substrings as tool-control markers — ordinary reasoning prose mentioning them flipped the split and leaked the rest of the think block into content (reproduced as arg-fragments-as-text in the chess events). Markers now angle-bracketed forms only; chunk-split openers were already covered by the partial-prefix hold. Integration repairs from the union suite: batched_decode _admit_rows made AR-lane/hidden-None safe (PR youssofal#195's AR lane x PR youssofal#200's refill was an unexercised intersection — 2 tests), two inspect.getsource guard tests re-anchored to the resolved source forms (invariants unchanged).
….py actually reads The harness computes LAB/specs/<project>-spec.md but the four spec files shipped at the lab root — every fresh checkout failed with 'no spec' until someone hand-copied them (caught twice tonight).
…e Swift/Python sync pair Laguna S-2.1 enters both catalogs (mlx-community/Laguna-S-2.1-oQ4e, 64.1 GB, peak ~74 GiB, modern-Apple tier) with a new arOnly capability: target-only AR models have no MTP head and the engine hard-rejects MTP loads, so the command builder now carries --no-mtp and suppresses any persisted/tuned --depth for them from one seam that covers every app launch path. Auto-tune stays out via the existing family gate (laguna is not a tunable family). Swift suite green.
…irror test already green)
…t run away (youssofal#213) The think prelude added in youssofal#186 lets a model out of the <think> block that Qwen-style templates open inside the generation prompt. Its terminal was unbounded free text, which makes one failure mode *legal*: a model that never emits </think> stays inside the prelude and fills max_tokens with prose, returning no document at all. The grammar cannot stop it because the grammar permits it. Measured on Qwen3.6 VL MoE (35B-A3B lineage, MLX, M5 Max, mtplx 2.3.0, --generation-mode mtp --depth 1, temp 0.6 and 0.2, response_format json_schema): ~25% of requests with thinking on returned finish=length, reasoning_content empty and 13k-41k characters of prose in content. Raising max_tokens from 4000 to 16000 did not fix it, it only made each failure cost 167s instead of 40s. Give the prelude its own terminal and bound it (default 4000 characters, MTPLX_THINK_PRELUDE_MAX_CHARS=0 restores the previous behaviour). The bound is part of the grammar cache key. tail/free text keep the unbounded TAG_TEXT, so the assistant's visible answer is never capped. After the change, same hardware and settings: 62/62 valid documents, and the REASON arithmetic suite goes from 5-6/8 with 2-3 runaways to 8/8 with none -- i.e. two-pass quality in a single call. The cap engages only when needed (3/10 on a short prompt, 0/20 on a longer one) and every capped request then emitted a valid document with finish=stop. Enforcement rides the sampling-time token mask rather than scheduler state, so unlike a scheduler-side thinking budget it cannot go stale under speculative decoding -- the failure mode that silently disabled vLLM's thinking budget whenever MTP was on, fixed there only in 0.21.0. The tool-call grammar shares the same prelude and is bounded too; that is the symptom reported in youssofal#196 (the content channel filling with reasoning narration until finish: length, with no tool call ever emitted).
…#210) Raise the onboarding probe timeout 12s -> 30s (generated MLX configs with per-tensor quant maps exceed 500 KB and tripped it), and on transport errors, transient HTTP failures, or malformed raw responses fall back to the HF model API metadata: retry the complete config at the immutable revision SHA, and accept the compact indexed config only when it carries a positive MTP marker so sparse metadata cannot create false 'no MTP' verdicts. Raw-file 404/auth behavior unchanged; original diagnostics preserved when the fallback also fails. Regression tests cover the slow-Hy3-config fallback and the diagnostic-preservation path.
…ool pass-through (youssofal#195 delta) Aligns tool-call serving with the Laguna reference configuration (Blackwellboy/laguna-s21-lab) — but the fixes harden the shared path for every model: - Balanced string/escape-aware scanner for the bracket dialect ([Calling tool: name({...})]): the old non-greedy regex ended the block at the first '})]', which any code-file argument contains inside a string, so large bracket calls always failed JSON decode and fell to prose. - Streaming 3-state classifier (complete/incomplete/invalid): buffer still-completing bracket blocks instead of streaming them as content and re-emitting the same call at finish (double delivery that also taught the model its drift dialect was accepted). - Poolside arg_key/arg_value dialect in the omlx bridge, strict contract: residue or duplicate keys mark the call malformed, never silent drops. - Unknown-named tool calls pass through under their raw name per the OpenAI contract; the client owns rejection and answers the model so it self-corrects. Previously the whole turn degraded to prose. - Hidden-tool stream-guard ceilings env-tunable (MTPLX_STREAM_HIDDEN_TOOL_GUARD_TOKENS/_S), defaults unchanged (2048 tokens / 30s). - Laguna fused-stack install report printed as a server startup line. Tests: bracket rescue, stream-guard env, omlx bridge dialect, tool-aware translator buffering, unknown-tool pass-through contract.
…egenerated on current base)
…#204, applied on current base)
…oussofal#205, applied on current base)
… applied on current base)
…restore giveup + unverified daemon-socket restore)
Root cause, two layers, both in the smart fan mode every app user runs by
default:
1. SmartFanController._do_restore treated a FAILED restore as restored:
on verification failure it logged one warning, set commanded_max=False,
and never tried again. Physical fans stayed pinned at max forever while
/health claimed a clean idle state. Now a failed restore re-arms with
backoff (5/15/30/60s cadence) until the fan rows verify back on the
Apple auto curve or a new lease legitimately re-ramps. Recovery and
every failure are logged; restore_verified / restore_failures are
surfaced in status() and /health for field diagnosis.
2. 2.3.0's new ThermalForge daemon-socket restore path trusted the
daemon's "ok" reply without checking the fans. A wedged daemon that
acknowledges without acting made set_thermal_profile('silent') report
success while the hardware stayed ramped. The socket path now verifies
the fan rows are back on auto (3s bounded) and otherwise falls through
to the CLI candidates in the same call.
Plus a guard for the "request doesn't cut off" bookkeeping class: a
stale-lease reconciler in the controller worker. A smart-fan lease held
while the engine activity probe (foreground counter + both scheduler
lanes + 30s recency window) reports the engine continuously idle for
MTPLX_SMART_FAN_STALE_LEASE_S (default 120s, 0 disables) is leaked
bookkeeping from a wedged request path, not a running request: it is
dropped with a loud log line asking for a report on youssofal#201, and the normal
restore flow brings the fans back. The probe fails open (busy) so a
broken probe can never restore fans under a live workload, and the
reconciler can never fire during legitimate generation, queued work, or
postcommit because all of those report busy.
Verified: 5 new/updated unit tests (restore retry until verified,
reconciler drops leaked lease, reconciler never fires while busy, socket
ack-without-effect falls back to CLI, socket verify happy path); live
serve on 4B QA model with real ThermalForge hardware: ramp on request,
restore verified 2s after completion, fans mode manual->auto, zero
spurious reconciles across happy-path/stream/disconnect shapes.
Version bump + changelog + user-facing notes for 2.4.0, built from a commit-by-commit diff audit of the 17 commits since v2.3.0 plus a closed-issue crosscheck (house notes rule). Contents: 35B-A3B compiled decode stack + continuous batched serving (David Tai youssofal#174/youssofal#200), Hy3 295B hardening (youssofal#208), honest finish_reason on length-cut tool calls + think-splitter leak fix (youssofal#196/youssofal#197 layers 1-2), bracket/Poolside tool-call dialects + unknown-tool pass-through (youssofal#195), bounded think prelude for structured output (Jozef Kristek youssofal#213), Forge HF probe recovery (Philip John Basile youssofal#210), the youssofal#201 fan restore fix with verification + retry + stale-lease watchdog, Laguna S-2.1 AR-only support, dependabot bumps. Authoritative pillar A/B vs released PyPI 2.3.0 (27B Optimized Speed, turbo, verified max fans, ABBA): decode pooled ratio 1.012, prefill-8k 1.043, prefill-32k 0.971 — flat within noise, no pillar regressed. Clean-window absolute: decode 58-61 tok/s both builds, prefill8k 564 vs 518, prefill32k 622 vs 635. Second half of the chain ran under external contention (mediaanalysisd + Chrome, load 4.7) which depressed both arms equally; verdict read pairwise.
The long_output_decay leg measures decode throughput over a 6k-token generation. At temperature the model occasionally answers the prompt with a short 'I am ready to proceed' preamble and a clean stop (seen 2026-07-31 at a healthy 52 tok/s, passing on immediate retry with 6000 tokens at 59.7 tok/s and a 1.05 last/first quintile ratio). A short answer is insufficient data for the decay measurement, not evidence of decay; retry once with a fresh request before failing the gate, and record the short attempt in the report.
…tegration-20260730
…ark drafter it is Forge correctly refuses DeepSeek-V4-Flash-0731 with backend_pending_mtp:deepseek-v4-mtp, but the entry misdescribes the architecture. V4 ships no MTP head: config's num_nextn_predict_layers=1 maps to zero tensors, and the checkpoint's mtp.0/1/2.* weights are a 3-stage DSpark block drafter (~20B, embed/lm_head shared, taps at layers 40-42, dspark_block_size=5). Any backend needs the DSpark block-draft contract, not the V3 MTP one. arch_id is left as-is (it is a public identifier in diagnostics, artifacts and tests); display_name, backend id, aliases and notes now state the truth, and the notes record the prerequisite: mlx-lm has no deepseek_v4 module at all, and V4 is not a V3 delta (MQA with joint 512-dim K=V, per-head q-norm, attention sinks, inverse-RoPE on the attention output, grouped low-rank O, per-layer compression ratios, frozen hash-routing layers, 4-copy Sinkhorn residual). tests/test_artifacts.py + tests/test_forge_cli.py pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…le) scaffold The compiled target-prefix control is only 13/24 byte-identical to generate_ar, which blocks the DFlash release; the divergence predates DFlash. This records a static analysis of why, plus a scaffold for the eventual measurement. Both are marked for what they are rather than what was hoped. The analysis proposed accumulation-order numerics as the leading hypothesis. External review superseded that framing: the supporting evidence came from the wide lane, whose eight-row verify falls through to the stock MoE and so differs on an axis staged-K1 does not have. A correction banner is added in place rather than rewriting, so the reasoning error stays visible. The scaffold does NOT run: it calls prepare_a3b_compiled_target_prefix with the wrong signature (TypeError after loading the model) and compares two stock-MoE prefill forwards with no attention_phase, no mx.compile and no shadow cache -- none of the axes in dispute. Guarded to fail immediately with an explanation, kept so the rewrite has something to diff against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y audited before the deciding measurement Its value is the timestamp: it lands before the measurement it governs, so no outcome can be rationalized afterward. It is a proposal, not a finished gate -- its own audit corrections are folded in and the contradicted original rules are replaced in place, not merely appended to. Separates two questions that byte-equality conflates: (A) is the drafter's declared q the distribution it sampled from -- the Leviathan-Chen guarantee and MTPLX's identity claim -- versus (B) is the compiled target path numerically identical to another implementation. Byte-equality vs generate_ar tests B. Receipts print separate A and B verdicts plus a final A-AND-B release verdict. Audit corrections applied, several against the document's own first draft: the L0-L6 ladder is NOT one-axis and a first-nonzero rung names the first EXPOSED difference (a provisional hypothesis, not a mechanism verdict); the proposed thresholds were looser than batch_equivalence's existing 1e-3 while claiming to be anchored to the strictest in-tree gate; rule-of-three needs zero flips; positions within a continuation are correlated; the A-side test as specified was confounded and must use the same compiled verifier as its control. Hand-verified rather than taken on trust: the suite genuinely cannot emit PASS (release_gate_pass hardcoded False at all four sites, deliberately fail-closed), both named A-blockers are absent, and -- load-bearing -- the whole-MoE dispatch docstring claims byte-comparability with generate_ar as its design intent. That demotes the numerics hypothesis: 13/24 reads as a possible defect in already-shipping compiled-target-prefix behaviour, not as evidence the gate measures the wrong thing. Also adds BACKLOG.md and, per the pre-registration's own same-push requirement, points the superseded gate statements in mtp-dflash-coordination.md and dflash_engine_suite.py at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gine evidence Enumerates the exact committed-token law for vocab-4 / block-3 by brute force over every draft draw and accept/reject outcome, across adversarial (p,q) families, with a planted defect proving the check can fail. It exercises the Leviathan-Chen ARITHMETIC and nothing else. External audit verdict: NOT EVIDENCE. Two reasons, both recorded in the file: 1. It never calls the production path. Oracle and planted defect are both local functions, so recomputing acceptance and residuals independently buys nothing -- independence of implementation is worthless without contact with the artifact. A1-A4 in the pre-registration all remain unmet. 2. Its conditioning 'fix' was a weakening. The first version asserted an unconditional two-token joint, it failed, and the assertion was conditioned until it passed. That was backwards: in a real decoder a rejection does not end generation, so over the emitted stream the unconditional law holds. The assertion was right and the ORACLE was wrong, terminating at the first rejection instead of continuing into the next cycle. Filed under tests/prototypes/ with a non-test filename so default collection does not count 47 passing cases as coverage for a claim they do not support. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… claimed were folded in Post-push verification caught that commit bdec012 overclaimed: it said the audit corrections were applied in place, but section 4.2 still asserted the false strictest-gate anchor, section 5 still issued definitive mechanism verdicts, section 7 still said 'two verdicts never one', section 9 still said H1 removes the B blocker, and the backlog and divergence doc repeated the one-axis/names-the-mechanism claims plus a link to the oracle's old path. All corrected at the point of claim now: 4.2's threshold table is explicitly marked looser than batch_equivalence's 1e-3 and unsettled; the sample plan is marked superseded (zero flips, clustered positions, B6 non-certifying); 5's table header says every mechanism verdict is provisional; 7 prints separate A and B verdicts plus a final A-AND-B; 9 states H1 does not remove the B blocker. The irony is noted: the commit that landed the 'correct the claim where it is made' discipline itself corrected claims only in an appendix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…te-checkpoints-20260811 feat: add exact sparse checkpoint placement planner
…ce-gate-20260811 feat: add SessionBank checkpoint replay telemetry
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the root integration PR for the MTPLX 2.4.0 development baseline. It consolidates the adopted runtime work and the later default-off research/gating infrastructure that had previously existed only on
codex/moe-dflash-integration-20260730.Major production-facing areas:
Experimental/default-off areas:
Stack and scope
1d1a02662205a29f78a6f3d119300334632db40amain: 0 commits behind, 34 commits aheadThis is intentionally a draft. Packaging success is not whole-stack approval. The compiled kernels, model-specific serving routes, and performance claims require their own exact-head evidence before this PR can become ready.
Evidence already attached
Promotion checklist
Known limitations