Skip to content

Fix _Linear8bitLt quantized guards and O(N^2) missing-keys hook in bitsandbytes plugin - #22044

Open
deep1blue wants to merge 1 commit into
Lightning-AI:masterfrom
deep1blue:fix/bitsandbytes-linear8bit-and-missing-keys-hook
Open

deep1blue wants to merge 1 commit into
Lightning-AI:masterfrom
deep1blue:fix/bitsandbytes-linear8bit-and-missing-keys-hook

Conversation

@deep1blue

Copy link
Copy Markdown

What does this PR do?

Fixes two issues in src/lightning/fabric/plugins/precision/bitsandbytes.py:

  1. In-place $O(N)$ filtering in _ignore_missing_weights_hook:
    Previously, _ignore_missing_weights_hook iterated over reversed(incompatible_keys.missing_keys) and called incompatible_keys.missing_keys.remove(key) on each "*weight" entry:

    for key in reversed(incompatible_keys.missing_keys):
        if key.endswith("weight"):
            incompatible_keys.missing_keys.remove(key)

    Because list.remove(key) always searches linearly from index 0, removing elements from the end of missing_keys requires scanning the entire prefix on every removal ($O(N^2)$ time). Replacing this with in-place slice assignment (incompatible_keys.missing_keys[:] = [key for key in incompatible_keys.missing_keys if not key.endswith("weight")]) preserves the missing_keys list object identity in $O(N)$ time (reducing 10,000-key filtering from 401.2 ms to 0.49 ms, an ~825x speedup).

  2. Correct Int8Params / torch.int8 guards in _Linear8bitLt.to_empty() and _Linear8bitLt.reset_parameters():
    _Linear8bitLt.to_empty() and _Linear8bitLt.reset_parameters() checked self.weight.dtype == torch.uint8 and isinstance(self.weight, bnb.nn.Params4bit) (which are the 4-bit invariants from _Linear4bit), whereas _Linear8bitLt uses bnb.nn.Int8Params and torch.int8 storage (as checked in _Linear8bitLt.quantize_()). As a result:

    • Calling to_empty() on an already-quantized _Linear8bitLt layer (dtype == torch.int8) bypassed the NotImplementedError guard.
    • In _Linear8bitLt.reset_parameters(), linear_init_finished = isinstance(self.weight, bnb.nn.Params4bit) was always False, bypassing both the quantized NotImplementedError guard (causing kaiming_uniform_ to fail with RuntimeError: "normal_kernel_cpu" not implemented for 'Char') and the post-init re-quantization / _replace_param branch.

Validation

pytest tests/tests_fabric/plugins/precision/test_bitsandbytes.py -k test_ignore_missing_weights_hook_and_linear8bit_guards

Acknowledgments

Credit and thanks to @carmocca (Carlos Mocholí, #18655, #19150), @awaelchli (Adrian Wälchli, #19886, #20121), @Borda (Jirka Borovec, #20911), @matthewdouglas (Matthew Douglas, #20956), and @bhimrazy (Bhimraj Yadav, #21858) for designing and maintaining the BitsandbytesPrecision plugin in Lightning Fabric.

…tsandbytes plugin

1. In `_ignore_missing_weights_hook`, iterating over
   `reversed(incompatible_keys.missing_keys)` and calling
   `incompatible_keys.missing_keys.remove(key)` on each `"*weight"` entry
   scans from index 0 of the list on every removal, resulting in O(N^2)
   time across models with many quantized Linear layers. Replace with an
   in-place list slice comprehension (`missing_keys[:] = [...]`) to
   filter missing weight keys in O(N) time while preserving the list
   instance.

2. In `_Linear8bitLt.to_empty()` and `_Linear8bitLt.reset_parameters()`,
   the quantized-weight guards checked `self.weight.dtype == torch.uint8`
   and `isinstance(self.weight, bnb.nn.Params4bit)` (copied from
   `_Linear4bit`), even though `_Linear8bitLt` uses `bnb.nn.Int8Params`
   with `torch.int8` storage (`quantize_()` checks `weight.data.dtype ==
   torch.int8` and `isinstance(self.weight, bnb.nn.Int8Params)`). Update
   both methods in `_Linear8bitLt` to check `bnb.nn.Int8Params` and
   `torch.int8`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Parth Vadera <parthvadera@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant