Qwen3.8 DFlash2 benchmark against Optimized Speed and Quality MTP

[davidtai]

What changed

This adds a closed, benchmark-only Qwen3.8 DFlash2 channel against the existing MTPLX Optimized-Speed target:

• replaces the obsolete Qwen3.6 DFlash baseline with z-lab/Qwen3.8-27B-DFlash2;
• loads the target exactly once through MTPLX and binds the unchanged stock DFlash2 Qwen-GDN target ops and eager draft backend to that same model object;
• validates the target artifact, checkpoint revision, block geometry, layer taps, quantization, and turbo profile before generation;
• adds guarded fixed-workload CLI entrypoints, C0/B/C1 depth sweeps, atomic receipts, determinism checks, prefill TPS, decode TPS, and speculative hit rate;
• pins the DFlash dependency to the immutable cache-ownership-fixed source used for the measurements;
• adds focused dependency, runtime, CLI, orchestration, guard, and receipt tests.

Why

We wanted to establish the best stock DFlash2 depth on the real Qwen3.8 27B model before considering any custom width-specific Metal kernel. The comparison must reuse the promoted MTPLX Optimized-Speed profile and target model, replacing only the speculative proposer. This PR provides that controlled measurement lane; it does not promote DFlash2 into serving defaults and does not add a custom kernel.

Notable decisions

• Target: Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed only.
• Draft: z-lab/Qwen3.8-27B-DFlash2 at revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4.
• DFlash dependency: davidtai/dflash-mlx@95a16104c5462793f9f14d2cd0b97b856fb643a1, which contains the cache-ownership correction needed when alternating MTPLX MTP and DFlash2 against one shared target.
• Greedy means temperature=1.0, top_p=1.0, top_k=1, seed 0. Thinking is disabled.
• M is DFlash2 physical block width; accepted draft depth is K=M-1. The checkpoint supports physical widths through M=8, therefore K=1 through K=7 are the meaningful speculative candidates.
• Invariant validation happens once at construction. The enabled benchmark path has no silent eligible-or-stock or exception fallback.
• Token divergence is recorded, hashed, and required to be deterministic, but is not used as an exact cross-engine parity gate. Semantic impact is measured separately with full HumanEval+.

How to verify

uv sync --extra dev --extra competitors

python -m pytest -q \
  tests/test_dflash2_dependency.py \
  tests/test_dflash2_contract.py \
  tests/test_dflash2_runtime.py \
  tests/test_dflash2_depth_sweep.py \
  tests/test_dflash2_cli.py \
  tests/test_qwen38_dflash2_depth_guarded.py

ruff check \
  mtplx/benchmarks/dflash2_contract.py \
  mtplx/benchmarks/dflash2_runtime.py \
  mtplx/benchmarks/runners/competitor_baselines.py \
  mtplx/benchmarks/runners/dflash2_depth_sweep.py \
  mtplx/cli.py scripts/qwen38_dflash2_depth_guarded.py \
  tests/test_dflash2_*.py tests/test_qwen38_dflash2_depth_guarded.py

scripts/hygiene_scan.sh
uv lock --check
python -m build
scripts/fresh_venv_smoke.sh

The full suite is green with one pre-existing upstream race deselected:

uv run --extra dev --extra competitors python -m pytest -q \
  --deselect tests/test_cold_tier_stats_cache.py::test_repeated_stats_polls_reuse_cached_aggregate

The deselected test fails identically on unchanged upstream/main (2b0360ca) because its connection spy also counts an asynchronous cold-tier background scan. It fails 3/3 on that exact base and is untouched by this branch.

The complete guarded benchmark campaign and HumanEval+ results are in the PR benchmark comment.

[davidtai]

Complete Qwen3.8 DFlash2 benchmark campaign

This comment records every benchmark run used to narrow the stock DFlash2 depth, including the invalid 64K attempt and its corrected replacement. No custom DFlash2 kernel was built; the purpose of this campaign was to choose the depth first.

Fixed setup

• Hardware: Apple M5 Max, 128 GiB unified memory.
• GPU serialization: every real-model run held /tmp/mtplx-gpu-exclusive.lock; the guard stopped and restored the exact Qwen service around each window.
• Target: Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed.
• Control: unchanged promoted MTPLX turbo profile, MTP depth 3, capture_commit, linear-gdn-from-conv-tape, persistent cache, cycle history.
• Candidate: z-lab/Qwen3.8-27B-DFlash2 at revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4, q4/group-64 draft, layer taps (5, 19, 33, 47, 61).
• Dependency: davidtai/dflash-mlx@95a16104c5462793f9f14d2cd0b97b856fb643a1 (version metadata 0.1.10), including the shared-target cache-ownership fix.
• One MTPLX target load per campaign. DFlash2 reused runtime.model; only the proposer and verification cycle changed.
• Sampling: greedy temperature=1.0, top_p=1.0, top_k=1, seed 0, thinking disabled.
• Throughput runs: exact forced 1,024 output tokens with an empty stop set. Each width had a 32-token warmup, then three rotated C0/B/C1 brackets: MTP-before, DFlash2 candidate, MTP-after.
• Reported numbers below are medians. speedup is the runner's paired candidate/control normalization. Prefill and decode timing are separate.
• Width notation: physical DFlash block width is M; accepted draft depth is K=M-1. M=8 is therefore K=7, not K=8.
• Every throughput row produced all 1,024 tokens, reported the requested width, had no AR fallback, and was deterministic across repetitions.
• AR-oracle token hashes diverged between engines. We retained deterministic drift rather than pretending bitwise cross-engine equivalence, then used HumanEval+ to measure semantic impact.

1. Initial exhaustive 1K/1K sweep

This was the reference MTP bracket and first complete physical-width sweep. It established K=3 / M=4 as the short-context winner.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	MTP prefill tok/s	DFlash hit rate
0	1	24.831	39.816	-37.44%	741.3	796.5	0.00%
1	2	37.614	39.789	-5.70%	746.9	759.8	44.53%
2	3	46.046	39.829	+15.65%	745.8	768.9	57.52%
3	4	51.059	39.904	+27.92%	739.9	766.3	60.74%
4	5	49.200	39.891	+23.44%	746.8	765.8	63.96%
5	6	49.969	39.824	+25.46%	747.3	777.1	65.23%
6	7	41.625	39.802	+4.58%	747.4	791.8	65.33%
7	8	41.477	39.719	+4.23%	740.3	785.3	66.21%

Receipt SHA-256: b29fe67161c2ccf20b58ce5a22845a4229c2289b0ae221077700385552e2129d.

2. Narrowed 1K/1K K=4..6 repeat

This repeat checked the cluster around the apparent K=5 secondary peak.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	Hit rate
4	5	48.972	39.694	+23.52%	731.0	63.96%
5	6	49.813	39.636	+25.74%	730.0	65.23%
6	7	41.532	39.694	+4.64%	742.0	65.33%

Receipt SHA-256: d1646b0a9e012287d7ce8b0a9cc6e9d327edcec074533d04940e08427ee33c76.

3. 16K/1K K=4..6

This used an actual 16,384-token Python prompt, not a warmed cache with a shorter measured input. Longer context changed the ranking: deeper blocks became progressively better.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	MTP prefill tok/s	Hit rate
4	5	46.556	32.274	+44.25%	763.4	782.9	67.68%
5	6	52.841	32.272	+63.45%	762.4	783.0	71.29%
6	7	55.693	32.313	+72.41%	763.5	783.3	76.95%

Receipt SHA-256: ccc884b7e3775c5ce3b484061f6834fcbd6f71c3d234bcf81d8e2a9e610326c7.

4. Standalone 16K/1K K=7

We then tested the checkpoint maximum: physical M=8 / accepted K=7.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	MTP prefill tok/s	Hit rate
7	8	59.939	32.482	+84.85%	773.4	794.3	79.00%

Receipt SHA-256: b1b9bb0c04feba5cbad5bb1f843dcabea4873c8f2d2a75df88b1212df880aa82.

5. Joined 16K/1K K=3..7 sweep

The standalone K=7 result was then joined into one same-load, same-rotation sweep so the K=3..7 ranking did not depend on separate process windows.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	MTP prefill tok/s	Hit rate
3	4	48.861	32.725	+49.45%	788.0	808.9	64.94%
4	5	47.756	32.649	+46.27%	788.6	807.5	67.68%
5	6	53.988	32.697	+65.24%	788.2	809.1	71.29%
6	7	56.544	32.710	+72.92%	788.9	808.4	76.95%
7	8	60.248	32.686	+84.12%	788.6	805.9	79.00%

Receipt SHA-256: 064ad2341e58ebfdfa9121797e8051e659f61375b30b48d8a787ea50101e9a6f.

6. Naturalistic 109-token Python prompt, 1,024 output tokens

This used a normal request to implement and test merge_intervals, rather than the repeated exact-length synthetic prompt. K=5 / M=6 won here.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	Hit rate
1	2	39.485	45.685	-13.50%	483.0	46.68%
2	3	53.843	45.669	+17.90%	485.8	62.89%
3	4	67.175	45.730	+46.93%	489.6	69.63%
4	5	67.767	45.700	+48.29%	488.5	73.73%
5	6	72.790	45.719	+59.12%	490.5	75.98%
6	7	67.341	45.674	+47.43%	487.0	78.52%
7	8	66.356	45.651	+45.32%	488.4	78.81%

Receipt SHA-256: 4315aa59614a4286c7aeb91e05de50e2aeb01052cc8053686c0f97d75e3fda83.

7. Invalid “1K vs 64K” attempt — retained and explicitly rejected

This was supposed to measure 64K prefill with 1,024 generated tokens, but the receipt proves prompt_tokens=1024. It simply repeated the 1K/1K test. A “64K prefill setting” cannot substitute for feeding 64K actual prompt tokens, so these numbers are not 64K evidence.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	Hit rate
1	2	37.543	39.859	-5.85%	743.9	44.53%
2	3	46.003	39.750	+16.16%	736.7	57.52%
3	4	51.121	39.961	+27.91%	747.1	60.74%
4	5	49.264	39.915	+23.37%	745.7	63.96%
5	6	49.998	39.799	+25.24%	748.0	65.23%
6	7	41.627	39.844	+4.44%	748.1	65.33%
7	8	41.489	39.792	+4.13%	747.0	66.21%

Rejected receipt SHA-256: 33459a496f561b24433df64206357f0c515e2a99d7b228f9858405c85a19beb6.

8. Corrected cold 64K/1K K=3..7 sweep

The replacement run fed an actual cold 65,536-token Python prompt and generated exactly 1,024 tokens for every arm. At 64K, K=3 / M=4 becomes the clear winner even though deeper widths have higher acceptance rates. This is the depth to beat before considering a custom kernel for the 64K workload.

K	M	DFlash decode tok/s	MTP decode tok/s	Speedup	DFlash prefill tok/s	MTP prefill tok/s	Hit rate	Total s
3	4	48.351	30.035	+60.93%	603.1	624.1	68.36%	129.78
4	5	39.626	30.048	+31.88%	603.5	623.6	71.48%	134.43
5	6	44.768	30.066	+48.81%	603.1	624.1	73.73%	131.54
6	7	39.958	30.054	+32.91%	603.0	624.2	74.41%	134.31
7	8	39.951	30.063	+32.80%	603.4	624.1	75.20%	134.22

Receipt SHA-256: 0e2579ac5ee44cfe1f5fe4bcd952ff18416a78fc90e84e6f8c4eb3a3a9cfe8a0.

9. Full paired HumanEval+

We generated one completion for all 164 HumanEval+ tasks with both engines from the same loaded target. We used the EvalPlus 0.3.1 OpenAI chat prompt, thinking disabled, greedy top-k-1 sampling, seed 0, EOS stopping, and a 768-token cap. The programs were sanitized with EvalPlus itself, then the MTP and DFlash2 sample sets were scored concurrently with four evaluator workers each.

For quality we used the 64K throughput winner, K=3 / M=4.

Evaluation	MTPLX MTP D3	DFlash2 K3/M4	Delta
HumanEval base tests	152/164 (92.68%)	151/164 (92.07%)	-0.61 points
HumanEval+ base + extra	150/164 (91.46%)	149/164 (90.85%)	-0.61 points

• 140/164 sanitized programs were byte-identical.
• There was one status swap and no DFlash-only win: HumanEval/93 passed both suites under MTP and failed both under DFlash2.
• The DFlash program interpreted “two places ahead in the alphabet” as two positions ahead in the vowel list; MTP emitted the correct alphabet shift.
• Exact paired McNemar p-value is 1.0 with only one discordant task. That is not evidence of a systematic regression, but the point estimate is still a real one-task loss and must not be described as exact quality parity.
• DFlash2 reported zero AR fallbacks.

Comparison receipt SHA-256: 56d0c0ec97499570342246856ab310d63e84c8d34776e72e70fafc84460acf68.

Conclusion

Stock DFlash2 is materially faster than the optimized MTP control for K>=2 on most measured prompts, but the best width is workload-dependent:

• 109-token natural prompt: K=5 / M=6, +59.12% decode.
• 1K synthetic prompt: K=3 / M=4, +27.92% decode.
• 16K prompt: K=7 / M=8, +84.12% decode.
• Cold 64K prompt: K=3 / M=4, +60.93% decode.

The 64K result answers the immediate selection question: optimize K=3/M=4 first if we proceed to a custom kernel for that workload. The HumanEval+ result says performance promotion should carry a quality caveat: one fewer pass out of 164, with no compensating DFlash-only win in this deterministic run.

[davidtai]

Optimized-Quality cold-prefix rerun

I reran the DFlash2 depth matrix against the Quality target artifact, using the new pinned/reproducible runner in scripts/qwen38_dflash2_quality_cold_matrix.py.

Exact artifacts and runtime contract

• Target: Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality
  • pinned HF revision: 09f71b39a75c416be3c974840b53f9fbe9aa1841
  • target weights: affine q8 / group 64
  • MTP sidecar: prequantized-mlx-affine
  • recommended turbo profile, native MTP depth 3
• Draft: z-lab/Qwen3.8-27B-DFlash2
  • pinned revision: 50307d4c4cde6860d4eee73e2547cd786fe8e8a4
  • block size 8, layers (5, 19, 33, 47, 61)
  • draft-only LM head installed as affine q4 / group 64
• DFlash dependency: dflash_mlx_0_1_10
• Sampling: greedy, temperature 1.0, top-p 1.0, top-k 1, seed 0, thinking disabled
• Output limit: exactly 1,024 generated tokens for every measured arm

The benchmark was run inside the verified GPU-lock guard. The normal Qwen service was stopped for the measurement window, the lock was held continuously, and mtplx-qwen38-27b-optimized-speed was restored successfully afterward.

Workloads and method

There are four independently measured workloads:

1. A natural Python request that encodes to exactly 105 test-input tokens, with no synthetic prefix.
2. A fresh 1,024-token coding-agent prefix followed by a fixed 1,024-token Python test input (2,048 total input tokens).
3. A fresh 16,384-token prefix followed by the same fixed 1,024-token test input (17,408 total).
4. A fresh 65,536-token prefix followed by the same fixed 1,024-token test input (66,560 total).

The cold-prefix generator repeats/truncates the repository's coding-agent prefill text to the exact requested prefix length. It constructs the 1,024-token test suffix separately, then concatenates the two token arrays. All three cold rows use the exact same suffix hash:

f7f31dbea69b5034cea1bb56aff73165fc51e35e3ac8e3b71f6445d6c5193450

No prefix/session cache was reused. DFlash ran with prefix caches disabled and cache boundaries cleared. The MTP controls used no session bank.

For each workload and K, there were three rotated repetitions. Each candidate was bracketed by the unchanged Quality MTP control:

MTP D3 C0 -> DFlash2 candidate -> MTP D3 C1

The tables report:

• DFlash decode TPS: median candidate decode TPS across three repetitions
• MTP D3 control: median of the local (C0 + C1) / 2 control values
• Gain: DFlash decode TPS relative to that local control
• Prefill TPS: median DFlash prefill TPS
• Spec hit: median DFlash speculative decode hit rate

DFlash's public K is one less than its physical width, so K=1..7 maps to physical M=2..8.

Natural Python, 105 input / 1,024 output

K	DFlash decode	MTP D3 control	Gain	Prefill TPS	Spec hit
1	32.561	38.000	-14.3%	428.6	48.63%
2	45.280	38.039	+19.0%	429.4	64.26%
3	61.037	38.075	+60.3%	433.4	72.75%
4	68.258	38.037	+79.4%	432.1	77.54%
5	71.795	38.035	+88.8%	431.1	78.71%
6	57.329	38.038	+50.7%	431.2	81.54%
7	57.354	37.934	+51.2%	432.4	82.13%

Selection: K=5.

Cold 1K prefix + fixed 1K test, 2,048 input / 1,024 output

K	DFlash decode	MTP D3 control	Gain	Prefill TPS	Spec hit
1	29.840	34.997	-14.7%	690.1	45.51%
2	40.797	35.019	+16.5%	743.0	61.43%
3	47.953	34.922	+37.3%	733.2	66.70%
4	51.905	34.845	+49.0%	695.9	71.09%
5	56.235	34.844	+61.4%	690.2	73.44%
6	40.138	34.912	+15.0%	691.8	74.51%
7	40.279	34.867	+15.5%	691.3	75.29%

Selection: K=5.

Cold 16K prefix + fixed 1K test, 17,408 input / 1,024 output

K	DFlash decode	MTP D3 control	Gain	Prefill TPS	Spec hit
1	26.535	30.475	-12.9%	742.6	45.80%
2	34.831	30.284	+15.0%	742.9	60.06%
3	40.582	30.333	+33.8%	743.7	65.43%
4	43.934	30.294	+45.0%	743.1	69.63%
5	43.064	30.455	+41.4%	741.4	69.14%
6	35.758	30.455	+17.4%	742.2	73.24%
7	34.900	30.485	+14.5%	740.7	73.44%

Selection: K=4 has the highest median decode TPS; the prespecified normalized-ratio tolerance classifies K=4 and K=5 as a tie.

Cold 64K prefix + fixed 1K test, 66,560 input / 1,024 output

K	DFlash decode	MTP D3 control	Gain	Prefill TPS	Spec hit
1	20.130	24.590	-18.1%	567.5	46.09%
2	26.952	24.678	+9.2%	572.0	60.74%
3	38.240	24.669	+55.0%	571.2	67.38%
4	35.245	24.645	+43.0%	569.1	71.48%
5	37.925	24.637	+53.9%	568.1	73.14%
6	30.101	24.595	+22.4%	568.0	74.02%
7	30.550	24.625	+24.1%	567.9	75.00%

Selection: K=3. K=5 is close by raw median decode TPS, but K=3 wins the prespecified bracket-normalized selection rule.

Correctness and interpretation

• Every candidate and both adjacent controls generated all 1,024 requested tokens.
• Candidate repeats were deterministic at every K; the MTP control was stable within every workload.
• No DFlash arm entered autoregressive fallback, and the requested physical width was used in every arm.
• Cross-engine token hashes differed for every DFlash K. Per the benchmark policy agreed for this investigation, that drift is recorded but is not a throughput-selection blocker. This run therefore proves deterministic/full-length execution and absence of fallback; it does not claim exact token parity or replace a task-quality evaluation.

The central result is that there is no single best DFlash depth across context lengths. K=5 wins the short workloads, K=4/K=5 are the 16K frontier, and K=3 wins at 64K. Higher K keeps raising speculative hit rate, but after the context-dependent optimum the extra verification work costs more than the additional accepted draft tokens save. This argues for a construction-time/context-phase route rather than a single globally hard-coded width.

Receipt hashes

• natural 105: 8c311d70eeff239b7f938272c1165ec8d3f81c38de85cb53d166e82282ea81c7
• cold prefix 1,024: e3b191a7e1713780f6577413ec91862d7a10cbe5e8cdb169dd262d64b2cc4825
• cold prefix 16,384: bd843f26cf38d27a5f0d0becde99de1cc57e996d0c8be1d63d4fa59967b71739
• cold prefix 65,536: f22460ddf8e6fb90a7486db72cd63a6a4300c194ee36c8a576b71ce4cf1e32fc
• matrix manifest: a698cbc25dcb15ebadef77e8889ce1b0e1ede7d1cd810f86dfe49c8646414c18

Verification

• Focused DFlash2/Quality suite: 85 passed
• Focused Ruff on all changed files: passed
• Receipt-wide schema/hash/runtime validation: passed
• Full repository pytest: one unrelated existing branch failure in test_repeated_stats_polls_reuse_cached_aggregate; all DFlash2 tests passed
• Existing PR checks before this update: wheel, no-MLX smoke, and repository hygiene all green

Implementation commit: 3a7c9f16