TPS Drop significantly - No Change

[Deesha08]

Benchmark JSON envelope

MacBook Pro 48GB RAM

Speed of TPS went from 41 to 4, nothing changed on my side

Fan mode and whether --max was used

smart

Profile

default

[youssofal]

Cannot act on this without numbers. Post mtplx doctor --json, the model and profile you run, and a flight recorder line from a slow request, ~/.mtplx/metrics has per request tok/s now. Also worth retesting on 2.9.1 first, it fixed a turbo configuration bug that cost speed.

[Deesha08]

{
"compiled_verify": {
"above_fence_behavior": "eager verify per call",
"default_model": "/Users/XXX/.mtplx/models/Youssofal--Qwen3.8-27B-MTPLX-Optimized-Speed",
"fenced": true,
"max_context_source": "turbo profile",
"max_context_tokens": 32768,
"mode": "on",
"mode_source": "turbo profile",
"resolved_default_profile": "turbo"
},
"diagnostics": {
"checks": [
{
"command": null,
"docs_url": "https://ml-explore.github.io/mlx/build/html/install.html",
"expected": "macOS >= 14.0 on Apple Silicon",
"fix": "Upgrade to macOS 14+; MLX does not support older macOS.",
"id": "os.macos_version",
"observed": "26.6.1",
"severity": "error",
"status": "pass"
},
{
"command": "python3 -c "import platform; print(platform.machine(), platform.processor())"",
"docs_url": "https://ml-explore.github.io/mlx/build/html/install.html",
"expected": "native arm64 Python, not Rosetta",
"fix": "Install/use a native arm64 Python. If needed, reinstall via Homebrew arm64 or uv.",
"id": "python.native_arm64",
"observed": {
"machine": "arm64",
"processor": "arm"
},
"severity": "error",
"status": "pass"
},
{
"command": null,
"docs_url": "https://ml-explore.github.io/mlx/build/html/install.html",
"expected": "Python >= 3.11",
"fix": "Install Python 3.11 or newer.",
"id": "python.version",
"observed": "3.13.15",
"severity": "error",
"status": "pass"
},
{
"command": "python3 -m pip install mlx",
"docs_url": "https://ml-explore.github.io/mlx/build/html/install.html",
"expected": "mlx importable",
"fix": "Install MLX into this same native Python environment.",
"id": "mlx.import",
"observed": {
"default_device": "Device(gpu, 0)",
"get_active_memory": 0,
"get_peak_memory": 0,
"mlx": "0.32.1",
"mlx_lm": "0.31.3"
},
"severity": "error",
"status": "pass"
},
{
"command": null,
"docs_url": null,
"expected": "the model this Mac's default routes to fits: measured peak <= unified memory (comfortable at 1.5x)",
"fix": null,
"id": "resource.memory",
"observed": {
"default_model": "Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed",
"estimated_peak_gib": 25.0,
"unified_memory_gib": 48.0
},
"severity": "warning",
"status": "pass"
},
{
"command": null,
"docs_url": null,
"expected": "free space for model + temp download + safety headroom",
"fix": "Free disk space or set MTPLX_MODEL_DIR to a larger volume.",
"id": "resource.model_cache_disk",
"observed": {
"cache_dir": "/Users/XXX/.mtplx/models",
"free_gib": 360.63,
"required_gib": 49.63
},
"severity": "warning",
"status": "pass"
},
{
"command": "mtplx pull Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed",
"docs_url": "https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed",
"expected": "default model available in the HF cache or as the verified local startup model",
"fix": "No action needed.",
"id": "model.cache",
"observed": {
"hf_cache_exists": true,
"hf_cache_path": "/Users/XXX/.mtplx/models/Youssofal--Qwen3.8-27B-MTPLX-Optimized-Speed",
"hf_cache_validation": {
"contract_arch_id": "qwen3-next-mtp",
"contract_error": null,
"contract_present": true,
"missing_files": [],
"mtp_sidecar_candidates": [
"mtp.safetensors",
"mtp/weights.safetensors",
"model-mtp.safetensors"
],
"ok": true,
"required_files": [
"config.json",
"tokenizer.json",
"model.safetensors.index.json",
"mtplx_runtime.json",
"mtp.safetensors"
]
},
"startup_default_model": null
},
"severity": "warning",
"status": "pass"
},
{
"command": "mtplx pull Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed",
"docs_url": "https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed",
"expected": "a published Youssofal/... repo (not a local mtplx/ or models/ path)",
"fix": "Pull the default model, or pass --model to serve a different one.",
"id": "model.default_repo",
"observed": "Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed",
"severity": "error",
"status": "pass"
},
{
"command": null,
"docs_url": "https://docs.docker.com/desktop/setup/install/mac-install/",
"expected": "Docker Desktop installed for Open WebUI Docker path",
"fix": "Install Docker Desktop if you want the Open WebUI Docker integration.",
"id": "docker.binary",
"observed": "/Applications/Docker.app/Contents/Resources/bin/docker",
"severity": "warning",
"status": "pass"
},
{
"command": null,
"docs_url": null,
"expected": "port free before starting mtplx serve, or already a healthy MTPLX server",
"fix": "A healthy MTPLX server already on this port is fine to keep using; if something else holds it, stop that process or use --port 8001.",
"id": "port.mtplx_server",
"observed": {
"host": "127.0.0.1",
"open": false,
"port": 8000
},
"severity": "warning",
"status": "pass"
},
{
"command": null,
"docs_url": null,
"expected": "port free before starting Open WebUI, or already an Open WebUI container",
"fix": "Use a different Open WebUI host port or stop the process on 3000.",
"id": "port.openwebui",
"observed": {
"host": "127.0.0.1",
"open": false,
"port": 3000
},
"severity": "warning",
"status": "pass"
},
{
"command": null,
"docs_url": null,
"expected": "ThermalForge or TG Pro available for explicit --max only",
"fix": "Install ThermalForge only if you want opt-in fan boost.",
"id": "thermal.control",
"observed": {
"kind": "thermalforge",
"path": "/Users/XXX/.mtplx/bin/thermalforge",
"version": {
"command": [
"/Users/XXX/.mtplx/bin/thermalforge",
"--version"
],
"ok": true,
"returncode": 0,
"stderr": "",
"stdout": "0.1.0"
}
},
"severity": "warning",
"status": "pass"
},
{
"command": null,
"docs_url": null,
"expected": "Low Power Mode off for best sustained decode",
"fix": "Turn off Low Power Mode before benchmarking or serving long responses.",
"id": "power.low_power_mode",
"observed": {
"available": true,
"lowpowermode": null,
"powermode": "0",
"thermal": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded",
"thermal_ok": true
},
"severity": "warning",
"status": "pass"
},
{
"command": null,
"docs_url": null,
"expected": "no recorded thermal or performance warning",
"fix": "Let the Mac cool down or improve airflow before sustained benchmarks.",
"id": "power.thermal_pressure",
"observed": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded",
"severity": "warning",
"status": "pass"
}
],
"created_at": "2026-08-23T07:56:52-0700",
"host": {
"cache_dir": "/Users/XXX/.mtplx/models",
"chip": "Apple M5 Pro",
"disk_free_bytes": 387226394624,
"disk_free_gib": 360.63,
"mac_model": "Mac17,9",
"machine": "arm64",
"macos_version": "26.6.1",
"memory_bytes": 51539607552,
"memory_gib": 48.0,
"platform": "macOS-26.6.1-arm64-arm-64bit-Mach-O",
"processor": "arm",
"python_executable": "/opt/homebrew/var/mtplx/venv-2.8.3/bin/python",
"python_version": "3.13.15",
"system": "Darwin"
},
"overall": "pass",
"resources": {
"default_model_size_bytes": 21313949792,
"estimated_runtime_memory_bytes": 42788786272,
"required_download_free_bytes": 53284874480
},
"schema_version": 1,
"support_matrix": {
"preview_test_targets": [
"M3 Max",
"M4 Max",
"M3 Ultra / Mac Studio",
"M5 Max"
],
"supported": {
"default_model": "Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed",
"default_profile": "turbo",
"docker": "Docker Desktop current plus previous two macOS major releases",
"macos": ">= 14.0",
"platform": "Apple Silicon arm64 Mac",
"python": "native arm64 Python >= 3.11"
}
}
},
"environment": {
"git_branch": "not a git worktree",
"git_status": "not a git worktree",
"hf_path": null,
"mlx": {
"default_device": "Device(gpu, 0)",
"get_active_memory": 0,
"get_peak_memory": 0,
"mlx": "0.32.1",
"mlx_lm": "0.31.3"
},
"platform": "macOS-26.6.1-arm64-arm-64bit-Mach-O",
"project_root": "/Users/XXX",
"python_executable": "/opt/homebrew/var/mtplx/venv-2.8.3/bin/python",
"python_version": "3.13.15 (main, Aug 5 2026, 12:25:43) [Clang 21.0.0 (clang-2100.1.1.101)]",
"uv_path": "/Users/XXX/.local/bin/uv"
},
"huggingface": {
"cache_dir": "/Users/XXX/.mtplx/models",
"cache_exists": true,
"cache_writable": true,
"cached_models": 1,
"disk_free_bytes": 387226394624,
"disk_free_gb": 387.226,
"token_present": false,
"token_source": null
},
"policy": {
"benchmark_exactness_smoke_context": 2048,
"fanmax_counts_for_product_gate": false
},
"thermal_control": {
"available": true,
"clock_anchor_enabled": false,
"clock_anchor_policy": "explicit experimental only; never used for product claims",
"instructions": "Install ThermalForge and ensure the thermalforge CLI is on PATH.",
"selected": {
"kind": "thermalforge",
"path": "/Users/XXX/.mtplx/bin/thermalforge",
"version": {
"command": [
"/Users/XXX/.mtplx/bin/thermalforge",
"--version"
],
"ok": true,
"returncode": 0,
"stderr": "",
"stdout": "0.1.0"
}
},
"tools": [
{
"kind": "thermalforge",
"path": "/Users/XXX/.mtplx/bin/thermalforge",
"version": {
"command": [
"/Users/XXX/.mtplx/bin/thermalforge",
"--version"
],
"ok": true,
"returncode": 0,
"stderr": "",
"stdout": "0.1.0"
}
}
]
},
"tools": {
"powermetrics": "/usr/bin/powermetrics",
"python": "/opt/homebrew/var/mtplx/venv-2.8.3/bin/python",
"smc_atlas": null,
"smc_atlas_exists": false,
"sovereign": null,
"sovereign_exists": false,
"sudo": "/usr/bin/sudo"
}
}

[Deesha08]

@youssofal - Claude extracted Metrics from ~/.mtplx/metrics

Follow-up data — M5 Pro / 48GB / mtplx 2.9.1 / macOS 26.6.1

Model: Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed
Flags: --depth 3 --generation-mode mtp --profile turbo --scheduler-mode serial --batching-preset latency --ssd-session-cache on --prefill-chunk-tokens 2048

Source: ~/.mtplx/metrics/flight-8000.jsonl — 147 requests, 6,412 generation rounds, ~5.5 hours.

---

1. Decode tok/s degrades monotonically with context length

context	median tok/s	rounds
8–16k	23.3	215
16–24k	18.3	654
24–32k	16.4	768
32–40k	16.4	1017
40–48k	14.8	969
48–56k	13.5	1320
56–64k	12.0	550
64–72k	11.1	468
72–80k	10.5	327

A 2.2x decay from ~12k to ~76k context.

2. The "41 → 4" is two different measurements

• Decode speed is 10–23 tok/s depending on context.
• End-to-end (completion_tokens / elapsed_s) is ~3.5 tok/s — median 321 completion tokens in median 92.8s wall time.

The gap is TTFT, not decode.

3. Main anomaly: TTFT is disproportionate to the work actually done

prompt tokens        median 34,410
cached_tokens        median 31,978   (93% of prompt; 107/142 requests hit cache)
new_prefill_tokens   median  1,528
TTFT                 median   11.3s    p90 90.8s    max 247.9s

Only ~1,528 tokens require fresh prefill, yet TTFT is 11.3s median and 90.8s at p90.
Prefilling 1.5k tokens should be well under a second, so something in the cache
restore path appears to dominate TTFT rather than prefill compute itself.

Related: pc (prefix-cache store) events show a median store time of 2.6s
(60 of 85 stores succeeded).

4. MTP acceptance is healthy — drafting is not the bottleneck

draft position	accepted
1	81.4%
2	61.6%
3	45.8%
overall	62.9% (5,346,563 / 8,499,549)

5. Per-request wall time

median 92.8s · p90 300.0s · max 1,219.0s · total 353 min across 147 requests

[Deesha08]

@youssofal let me know if you need any other data from me