Summary
MXFP8 and NVFP4 inference quantize transformer weights after merging the constructor-provided LoRA adapter. LoRA lifecycle operations must update those quantized weights before the next inference request.
PR #1796 rejects lifecycle operations that cannot maintain this invariant. This issue tracks implementing those operations.
Review context: #1796 (comment)
Behavior implemented by PR #1796
| Public operation |
MXFP8 |
Default NVFP4 |
NVFP4 with retained BF16 weights |
| unmerge_lora_weights() |
Explicitly rejected |
Explicitly rejected |
Explicitly rejected |
| set_lora_adapter() |
Allowed and requantized |
Explicitly rejected |
Allowed and requantized |
Required work
- Support MXFP8 LoRA unmerge by quantizing the restored BF16 weights before inference.
- Support NVFP4 LoRA unmerge by preserving or restoring the BF16 weights and then quantizing the restored weights.
- Support default NVFP4 adapter changes by preserving or restoring the BF16 weights, merging the requested adapter, and then quantizing the merged weights.
- Keep quantized weights synchronized after every merge, unmerge, adapter change, strength change, and accumulated adapter update.
Tests
- Verify MXFP8 unmerge and remerge behavior.
- Verify NVFP4 unmerge with both removed and retained BF16 weights.
- Verify adapter replacement for MXFP8 and NVFP4.
- Compare lifecycle outputs against the equivalent BF16 LoRA execution.
Summary
MXFP8 and NVFP4 inference quantize transformer weights after merging the constructor-provided LoRA adapter. LoRA lifecycle operations must update those quantized weights before the next inference request.
PR #1796 rejects lifecycle operations that cannot maintain this invariant. This issue tracks implementing those operations.
Review context: #1796 (comment)
Behavior implemented by PR #1796
Required work
Tests