Skip to content

Support LoRA lifecycle operations for MXFP8 and NVFP4 inference #1821

Description

@Davids048

Summary

MXFP8 and NVFP4 inference quantize transformer weights after merging the constructor-provided LoRA adapter. LoRA lifecycle operations must update those quantized weights before the next inference request.

PR #1796 rejects lifecycle operations that cannot maintain this invariant. This issue tracks implementing those operations.

Review context: #1796 (comment)

Behavior implemented by PR #1796

Public operation MXFP8 Default NVFP4 NVFP4 with retained BF16 weights
unmerge_lora_weights() Explicitly rejected Explicitly rejected Explicitly rejected
set_lora_adapter() Allowed and requantized Explicitly rejected Allowed and requantized

Required work

  • Support MXFP8 LoRA unmerge by quantizing the restored BF16 weights before inference.
  • Support NVFP4 LoRA unmerge by preserving or restoring the BF16 weights and then quantizing the restored weights.
  • Support default NVFP4 adapter changes by preserving or restoring the BF16 weights, merging the requested adapter, and then quantizing the merged weights.
  • Keep quantized weights synchronized after every merge, unmerge, adapter change, strength change, and accumulated adapter update.

Tests

  • Verify MXFP8 unmerge and remerge behavior.
  • Verify NVFP4 unmerge with both removed and retained BF16 weights.
  • Verify adapter replacement for MXFP8 and NVFP4.
  • Compare lifecycle outputs against the equivalent BF16 LoRA execution.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    scope: inferenceInference pipeline, serving, CLIscope: trainingTraining pipeline, methods, configs

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions