Problem
AirLLM streams models layer-by-layer from disk even when the entire model could fit in GPU VRAM. For small models on consumer GPUs this leaves most VRAM unused while disk I/O limits inference to ~0.1 tok/s.
Proposal
Add an environment variable AIRLLM_FULL_LOAD=1. When set, AirLLM estimates the total model GPU footprint from safetensors file sizes. If it fits within 85% of free VRAM, all weights are loaded to GPU at once and streaming hooks are skipped — the model runs at native GPU speed. If it does not fit, normal layer streaming is used with a clear warning.
Test Results (RTX 4060 Laptop 8 GB)
| Model |
Before |
After |
Speedup |
| 1.5B |
0.1 tok/s |
32.2 tok/s |
322× |
| 7B |
0.05 tok/s |
0.05 tok/s (auto fallback) |
unaffected |
Related PR
#324 — implementation with tests
Problem
AirLLM streams models layer-by-layer from disk even when the entire model could fit in GPU VRAM. For small models on consumer GPUs this leaves most VRAM unused while disk I/O limits inference to ~0.1 tok/s.
Proposal
Add an environment variable
AIRLLM_FULL_LOAD=1. When set, AirLLM estimates the total model GPU footprint from safetensors file sizes. If it fits within 85% of free VRAM, all weights are loaded to GPU at once and streaming hooks are skipped — the model runs at native GPU speed. If it does not fit, normal layer streaming is used with a clear warning.Test Results (RTX 4060 Laptop 8 GB)
Related PR
#324 — implementation with tests