feat: GGUF load guard for MLX engines - #423
Merged
Merged
Conversation
mlx-lm/mlx-vlm/mlx-embeddings have no GGUF load path (mx.save_gguf is one-way export). Loading a .gguf file or GGUF-only dir crashes inside mlx_lm.load with an opaque error. Add a shared guard (fusion_mlx/engine/gguf_guard.py) called before every load entry: - engines/batched.py (LLM, mlx_lm.load) - engines/vlm.py (VLM, mlx_vlm.load) - engines/embedding.py (Embedding, mlx_embeddings.load) - engines/reranker.py (Reranker, 3 sites: _load_causal_lm / _load_jina_reranker / embedding fallback) Guard inspects the local filesystem target only (HF repo ids pass through to mlx_lm). GGUF-only targets raise GGUFLoadError with an actionable message pointing at mlx-community repos or POST /v1/convert. Non-GGUF targets are a no-op. 10 unit tests cover: direct .gguf file, GGUF-only dir, MLX dir, GGUF dir with config.json, empty name, nonexistent path, HF repo id, convert-endpoint message, no-op normal path. Closes audit action item from fusion-mlx_vs_*.md reports. Duplicate-engine debt tracked separately in #422. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dahai80
added a commit
that referenced
this pull request
Aug 8, 2026
* chore: bump version 0.8.11 → 0.8.12 Patch: GGUF load guard (#423), Wan2 staged VAE Stream fix (#419), DPO logprobs TypeError fix (#421). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: black-format training/reward.py (CI lint fix) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
mlx-lm / mlx-vlm / mlx-embeddings have no GGUF load path (
mx.save_ggufis one-way export). Loading a.gguffile or a GGUF-only directory crashes insidemlx_lm.loadwith an opaque error. This adds a shared guard that detects GGUF targets up front and raises a clear, actionableGGUFLoadError.This closes the "GGUF 加载桥" action item from the competitor audit reports (
fusion-mlx_vs_*.md): rapid-mlx has GGUF identification in its download gate; fusion-mlx had zero — now it has a load-time guard with a helpful message.How
New
fusion_mlx/engine/gguf_guard.py—assert_not_gguf(model_name, engine_kind)called before every load entry:engines/batched.pyengines/vlm.pyengines/embedding.pyengines/reranker.pyGuard inspects local filesystem targets only (HF repo ids pass through to mlx_lm, which handles remote resolution). GGUF-only targets (
.gguffile, or dir with.ggufweights but noconfig.json/safetensors) raiseGGUFLoadErrorpointing the user atmlx-community/<model>-mlxrepos orPOST /v1/convert. Non-GGUF targets are a no-op.Verification
pytest tests/unit/test_gguf_guard.py→ 10 passedpytest tests/unit/→ 8120 passed, 0 failed (was 8110; +10 new tests; 282 skipped = 3.4%)test_no_mllm_flag,test_batched_engine_output_router) fail identically with/without this change — pre-existing debt, unrelatedScope
Purely additive: +159 lines, 0 deletions. No public API signatures changed. The duplicate-engine tech debt (
engine/batched/vsengines/batched.py) is tracked separately in #422 — not mixed into this PR (risk asymmetry).🤖 Generated with Claude Code