Repository navigation
waifu: Residency::Hybrid keeps the smallest weights on the GPU when offloading; replaces LowVram - #131
Merged
Conversation
Residency::Hybrid { device_weight_bytes } is low-vram with the smallest
weights put on the device and kept there, as many as fit in the budget;
the rest are moved over at each load as before. Weights picks them once
the whole package is read (keep_the_smallest_on_device / smallest_within),
so load() and the IR are unchanged.
Smallest first because a low-vram load costs mostly a wait, not bytes:
norms, biases and scales are hundreds of waits and little to keep.
The budget reaches the engine through new options structs:
- Rust: EngineOptions { device, device_weight_bytes }; Engine::new takes it.
- C: WaifuEngineOptions with struct_size, and
waifu_engine_new_with_options. waifu_engine_new is unchanged, so the ABI
version stays 2.
- CLI: -device-weights GB, for cuda_cpu_offload and metal_disk_offload
(refused beside any other named device; taken beside auto).
- waifu-tools generate and memprofile: hybrid=GB beside lowvram.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aces it
LowVram was Hybrid { device_weight_bytes: 0 }: every weight on the host
and moved over at its load. The enum now has Device and Hybrid alone,
cuda_cpu_offload and metal_disk_offload are Hybrid with 0 bytes until
-device-weights says otherwise, and the tests and docs that named
LowVram name Hybrid { device_weight_bytes: 0 }. The CLI's own Residency
is renamed to match. waifu-tools keeps `lowvram` as the spelling of
hybrid=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fload
The Device picker moves out of the Model card into a Device group of its
own (DeviceCard). Under metal_disk_offload it shows a "vram (GB) ?"
slider: how many gigabytes of the model's smallest weights stay on the
GPU, from 0 to the model's size (this Mac's memory before it is
downloaded), in half gigabytes. The ? opens a popover saying what it
means and that it is the weights alone, not a limit on the run.
- WaifuKit: Engine(device:deviceWeightBytes:) over
waifu_engine_new_with_options.
- Launcher: keyed on Placement { device, deviceWeightBytes } rather than
the device name, so a changed slider reads the model again on the next
job; the Model card's footnote says so until it has.
- Session: deviceWeightGB, kept in UserDefaults; 0 is the old behaviour.
- SliderRow: an optional explanation behind a ? beside the label.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Residency::Hybrid { device_weight_bytes }replacesResidency::LowVram: an offload run keeps that many bytes of the model's smallest weights on the device and moves only the rest at each load (0 is the old low-vram). It is set by-device-weights GBin the CLI,WaifuEngineOptions/waifu_engine_new_with_optionsin the C API, and a "vram (GB)" slider in a new Device group of the Mac app.Why smallest first
A low-vram load costs mostly a wait, not bytes. Norms, biases and scales are hundreds of loads and little memory, so the first few hundred megabytes remove most of the waits; what is left to move is the large projections, which copy at the bus's rate.
What changed
flint/ir.rs):Weightsreads every weight to the host as before, thenkeep_the_smallest_on_devicesorts by size and moves weights to the device until the next does not fit.load()and the IR are unchanged.LowVramis gone;cuda_cpu_offloadandmetal_disk_offloadareHybridwith 0 bytes.Engine::new(EngineOptions { device, device_weight_bytes }). NewWaifuEngineOptions(withstruct_size) andwaifu_engine_new_with_options;waifu_engine_newis unchanged, soWAIFU_ABI_VERSIONstays 2. More than 0 bytes on a non-offload device is refused.-device-weights 6(GB, also6.5GB), for the offload devices orauto.hybrid=GBbesidelowvramingenerateandmemprofile.metal_disk_offloada "vram (GB) ?" slider from 0 to the model's size, with a popover explaining it. Changing it reads the model again on the next job.Tests
unittest, waifu (210), modelmanager, waifu-cli (176), waifu-tools builds,swift test(10).struct_size), CLI parsing, a Metal test that a hybrid package keeps exactly the small weights resident and draws the same, and a WaifuKit test.qwen_image*,indextts_gptwere only compiled), and no speed measurement yet of hybrid against plain offload on a real model.🤖 Generated with Claude Code