Skip to content

fix(weight_utils): name weight_type companions after the last weight only - #131

Open
KaigeGao1110 wants to merge 1 commit into
vllm-project:mainfrom
KaigeGao1110:fix/gguf-weight-type-companion-name
Open

KaigeGao1110 wants to merge 1 commit into
vllm-project:mainfrom
KaigeGao1110:fix/gguf-weight-type-companion-name

Conversation

@KaigeGao1110

Copy link
Copy Markdown

Purpose

The GGUF weight iterators name the synthetic weight_type companion with name.replace("weight", "weight_type"), which rewrites every occurrence. A module whose own name contains weight therefore loses its name.

Qwen3.8-Flash-Next's hyper-connection projection input_mix_weight_down.weight got the companion input_mix_weight_type_down.weight_type. No module owns that name, so loading the GGUF failed with no module or parameter named hyper_connection_mixer.input_mix_weight_type_down.

Changes

  • Add gguf_weight_type_name() in weight_utils.py. It replaces only the last weight (the parameter name) and raises if the name has none.
  • Use it in both the LLM iterator (gguf_quant_weights_iterator_multi) and the diffusion iterator (weights_adapter/diffusion/base.py).

Names with a single weight are unchanged, so existing models map exactly as before.

Test plan

tests/test_weight_type_names.py:

  • the helper keeps the module name for single-weight names, *_weight.weight names and fused w13_weight, and rejects names without weight;
  • both real iterators, run over a written Q8_0 GGUF file, yield the companion name derived from the last weight only.

With the two call sites reverted to str.replace (the helper kept), the iterator tests fail:

FAILED tests/test_weight_type_names.py::test_iterator_names_companions_after_the_mapped_parameter
  'model.hyper_connection_mixer.input_mix_weight_type_down.weight_type' != 'model.hyper_connection_mixer.input_mix_weight_down.weight_type'
FAILED tests/test_weight_type_names.py::test_diffusion_iterator_names_companions_after_the_parameter
  'transformer_blocks.0.attn.gate_weight_type_proj.weight_type' != 'transformer_blocks.0.attn.gate_weight_proj.weight_type'
2 failed, 6 passed

With the fix, all 8 tests pass.

Test results

I ran the full suite on one RTX PRO 6000 Blackwell (SM120) with vLLM nightly 2a02f6ef (0.28.1rc1.dev628) and PyTorch 2.13.0+cu130, with Hugging Face offline. I ran it on this branch and on unmodified main (d4c1f0d), using the same _C_gguf build (csrc/ is untouched):

tree failed passed skipped
main 81 1552 578
this PR 81 1560 578

The new tests pass (8 more passed). The 81 failures are the same test IDs on both trees, and none of them comes from this change:

  • 68 × test_kernels.py::test_moe: triton.runtime.errors.OutOfResources: out of resource: shared memory, because the Triton block size exceeds this GPU's limit.
  • 7 × test_gguf_generation.py::test_models and 5 × test_multimodal_gguf.py: the models cannot be downloaded offline (OSError, LocalEntryNotFoundError, and HFValidationError for repo:quant references).
  • 1 × test_plugin.py::test_register_sets_engine_args_for_gguf_model: HFValidationError for the local path /tmp/model.gguf.

pre-commit run --all-files passes.

Duplicate check: I searched open and closed PRs for weight_type. Nothing touches companion naming.

🤖 Generated with Claude Code

…only

The GGUF iterators built the synthetic companion name with
name.replace("weight", "weight_type"), which rewrites every occurrence.
A module whose own name contains "weight" therefore lost its name:
Qwen3.8-Flash-Next's hyper-connection projection
input_mix_weight_down.weight got the companion
input_mix_weight_type_down.weight_type, which no module owns, and loading
its GGUF failed with "no module or parameter named
hyper_connection_mixer.input_mix_weight_type_down".

Add gguf_weight_type_name(), which replaces only the last "weight", and use
it in both the LLM and the diffusion iterator. Names with a single "weight"
are unchanged. The tests cover the helper and drive both real iterators
over a written Q8_0 GGUF file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Kaige <a825075826@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant