OpenAI/Pi tool round crashes Qwen3.8 MTP at ~19K sequence: broadcast_shapes (16384 vs 19295)

[leoli-dev]

Summary

A normal Pi coding-agent tool round crashes the MTPLX OpenAI-compatible server with a tensor broadcast error. The server stays alive, but Pi receives the error and stops the agent. This is not an out-of-memory event and should not require callers to shorten/split an otherwise ordinary coding task.

The server advertises a 66,560-token context window, but the failing tensors show a 16,384-token buffer being combined with a 19,295-token sequence.

Error: [broadcast_shapes] Shapes (16384,4,256) and (19295,4,256) cannot be broadcast.

Environment

• Hardware: MacBook Pro, Apple M5 Max, 128 GB unified memory
• OS: macOS 26.6.2 (25G83), arm64
• MTPLX app/runtime: 2.9.0 (build 2009001)
• MLX: 0.32.1; mlx-lm: 0.31.3
• Model: Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality
• Served model id: mtplx-qwen38-27b-optimized-quality
• Client: Pi 0.84.2, OpenAI-compatible chat-completions provider at http://127.0.0.1:8001/v1
• Pi model setting at reproduction: contextWindow=66560, maxTokens=16384, reasoning=true

Relevant server configuration resolved by the MTPLX app:

backend-id qwen3_next
generation-mode mtp
profile turbo
scheduler-mode ar_batch
batching-preset agent
context-window 66560
paged-kv-quantization q8
reasoning-mode off

The server process also has MTPLX_DYNAMIC_PAGED_KV_MAX_INITIAL_NEW_TOKENS=4096.

Reproduction

1. Launch the MTPLX app with the model above and its Pi/agent target.
2. Start Pi with mtplx/mtplx-qwen38-27b-optimized-quality.
3. Give it a normal multi-file UI task: create an HTML/CSS/JS future-city energy-control dashboard with an interactive SVG map, animations, chart, controls and modal.
4. Pi creates the project directory, then emits a write tool call for index.html.
5. The response contains a normal 233-line HTML document. The write payload itself is only 9,905 characters / 3,379 Qwen tokenizer tokens.
6. Immediately after this tool round, Pi receives the broadcast-shape error and stops. MTPLX remains listening on port 8001.

Pi side:

Now creating the three files.
write .../arcology-control-room/index.html
... (223 more lines, 233 total)

Error: [broadcast_shapes] Shapes (16384,4,256) and (19295,4,256) cannot be broadcast.
Released pi-caffeinate (agent finished).
Warning: pi-lens deferred format: 0 changed, 1 failed

The pi-lens warning is downstream: the inference exception ended the agent round.

Why this looks like a server bug

Pi sends a standard OpenAI-compatible tool-call flow. A coding agent emitting one substantial file is normal usage, and the visible write payload is far below the advertised context window. The service must either grow/repage its MTP/KV state correctly, cap the request using a truthful capability limit, or reject it cleanly before inference. A low-level broadcast_shapes exception is not recoverable by the client.

The 2.9.0 server source includes a dynamic paged-KV reservation path that derives a reservation from prompt_tokens + reserved_new_tokens + mtp_depth, and the active process uses a 4,096 initial-new-token cap. The observed (16384, ...) versus (19295, ...) failure suggests that a fixed or compiled 16K state is still being reused for a longer request/continuation. Please confirm the exact path and whether it is related to ar_batch, MTP verification, or a warmup/compiled-cache bucket.

Expected

• A standard Pi tool-call turn should complete without a tensor-shape crash.
• The effective safe request/output budget should honor the advertised 66,560 context window, or the model endpoint should publish/enforce the real limit.
• If a request cannot fit, return a structured, actionable API error before entering MLX kernels.
• Please add a regression test for a Pi/OpenAI tool-call round containing a 3K+ token file write followed by the next agent continuation.

Related but not duplicate: #282 reports unexpected MTPLX manipulation/truncation of Pi tool behavior. This report is specifically a server-side tensor shape crash in that normal agent workflow.

[leoli-dev]

Update: reproduced on Optimized Speed too (not Quality-specific)

I reproduced the same continuation crash on the second published Qwen3.8 optimized pack, so this is not a Quality-quantization-specific failure.

New reproduction environment

• Same hardware / OS / MTPLX app/runtime / Pi version as the original report
  • MacBook Pro, Apple M5 Max, 128 GB unified memory
  • macOS 26.6.2 (25G83)
  • MTPLX 2.9.0 (build 2009001); MLX 0.32.1; mlx-lm 0.31.3
  • Pi 0.84.2
• Model: Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed
• Served model id: mtplx-qwen38-27b-optimized-speed
• Pi session explicitly records thinkingLevel: off.
• Active server configuration includes:

backend-id qwen3_next
generation-mode mtp
profile turbo
depth 3
scheduler-mode ar_batch
batching-preset agent
context-window 66560
paged-kv-quantization q8
reasoning-mode off
preserve-thinking off

The same normal Arcology HTML/CSS/JS coding task was used.

Exact sequence

1. Pi sent the task.
2. Speed successfully emitted a standard bash tool call to create the directory.
3. Speed successfully emitted a standard write tool call containing a complete index.html payload (7,433 bytes written).
4. That successful file-write response used 2,497 output tokens and 11,251 cache-read tokens (13,982 total in Pi's recorded usage).
5. Immediately after the successful tool result, the next continuation failed before producing any output:

Error: [broadcast_shapes] Shapes (16384,4,256) and (18269,4,256) cannot be broadcast.

Pi records the failing response as output: 0, stopReason: error.

Why this narrows the failure

The original Quality report failed at (16384,4,256) versus (19295,4,256) with Quality depth 2. This Speed run fails at the identical fixed 16,384 dimension versus a different continuation length (18,269), while using Speed depth 3 and thinking off.

Therefore both published Optimized Qwen3.8 27B packs hit the same class of failure after a successful OpenAI/Pi tool round. This strongly points to the shared qwen3_next MTP/KV/session-continuation path (potentially the ar_batch/compiled-cache path), rather than a prompt issue, Pi's tool encoding, thinking, or the Quality model's quantization.

As a temporary client-side workaround, forcing Pi to send max_tokens=8192 may avoid crossing the apparent 16K internal boundary for some tasks, but it is not a fix: routine coding-agent file writes should not crash a server that advertises a 66,560-token context window.

Could you please confirm whether the continuation allocation/compiled bucket is still derived from a fixed 16K bound in this path, and add coverage for both Optimized-Speed (D3) and Optimized-Quality (D2) OpenAI tool-call continuations?

[youssofal]

Fixed in 2.9.1. Paged KV capacity now comes from the pages actually allocated, so long agent sessions run to the model's full advertised context instead of dying near 19k. Closing, reopen if you still hit it on 2.9.1.