Tthe LLM Compressor export modification in mixed‑precision mode: the original format unifies all layers as Linear, causing vLLM inference to fail during parsing.

[1607662794]

Description

This PR fixes an issue where the LLM Compressor exports all layers as Linear type in mixed‑precision mode, regardless of their original type. This causes vLLM to fail during model parsing and prevents inference from starting.

Type of Change

Bug fix

Checklist Before Submitting

• [√] My code has been tested locally.
• [√] Documentation has been updated as needed.
• New or updated tests are included where applicable.
• The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

[wenhuach21]

@Yi4Liu @WeiweiZhang1 please have a review

[yiliu30]

Hi @1607662794, we’ve handled the mixed-precision case this way. Could you follow the same approach here as well? Thanks!

auto-round/auto_round/export/export_to_llmcompressor/export_to_fp.py

Lines 313 to 322 in 98f2cdb

	scheme_groups = {} # (bits, data_type) -> list of layer names
	for name, cfg in layer_config.items():
	layer_bits = cfg.get("bits", bits)
	layer_dt = cfg.get("data_type", data_type)
	if layer_bits > 8:
	continue
	key = (layer_bits, layer_dt)
	scheme_groups.setdefault(key, []).append(name)
	is_mixed = len(scheme_groups) > 1

[xin3he]

@1607662794 Could you please share the case before and after this fix?
I wonder whether it's already resolved by another fix in vLLM. vllm-project/vllm#49483
We changed the match rule to match the naming first. You can try it by installing the nightly vLLM. (not contained in v0.26.0)

[1607662794]

@1607662794 Could you please share the case before and after this fix? I wonder whether it's already resolved by another fix in vLLM. vllm-project/vllm#49483 We changed the match rule to match the naming first. You can try it by installing the nightly vLLM. (not contained in v0.26.0)

Before the modification, the quantization config’s target was uniformly set to “Linear”, making it impossible to distinguish different layers. After the modification, target is specifically assigned to the actual layers covered by that quantization config. Exported in the llm‑compressor format.

[xin3he]

@1607662794 Thank you for this contribution. It's a known issue and the fix I shared could fix this issue by checking the layer name first instead of the class name.

I worry that your change might not work in vLLM. In my memory, the MOE class name is RoutedExperts in vLLM, while your check is based on transformers model architecture.

If you could share the model tested with this fix and confirm that it loads successfully in v0.26.0 vLLM, I would be happy to approve this PR.

[1607662794]

@1607662794 Thank you for this contribution. It's a known issue and the fix I shared could fix this issue by checking the layer name first instead of the class name.

I worry that your change might not work in vLLM. In my memory, the MOE class name is RoutedExperts in vLLM, while your check is based on transformers model architecture.

If you could share the model tested with this fix and confirm that it loads successfully in v0.26.0 vLLM, I would be happy to approve this PR.

@1607662794 Thank you for this contribution. It's a known issue and the fix I shared could fix this issue by checking the layer name first instead of the class name.

I worry that your change might not work in vLLM. In my memory, the MOE class name is RoutedExperts in vLLM, while your check is based on transformers model architecture.

If you could share the model tested with this fix and confirm that it loads successfully in v0.26.0 vLLM, I would be happy to approve this PR.

Sure, using Qwen3.6-35B-A3B as an example with vLLM 0.26.0, this mixed‑precision configuration can be successfully loaded and inferred.