vllm-gds-bounded

A vLLM custom model loader that loads weights over GPUDirect Storage (GDS) - directly NVMe -> GPU VRAM - using a small, rotating pool of pre-registered staging buffers. It exists for GPUs/setups where vLLM's stock fastsafetensors GDS path crashes the nvidia-fs driver.

It registers as a vLLM plugin, so vLLM and fastsafetensors stay stock:

pip install -e .
vllm serve /path/to/model --load-format gds_bounded

Why

vLLM's fastsafetensors GDS loader registers the final tensor buffers as GDS targets. That creates many concurrent GPU BAR1 mappings whose lifecycle races the NVMe block-dispatch submit path inside nvidia-fs, and on some hardware that races into a null/stale-pointer dereference (nvfs_get_p2p_dma_mapping) that crashes the driver mid-load. Turning down thread/queue knobs does not help - the trigger is the buffer registration pattern, not concurrency level.

Observed on an unlocked NVIDIA CMP 170HX (GA100) loading a model larger than host RAM.

How it works (bounded staging)

1. Parse the single-file safetensors header (name -> dtype, shape, byte range).
2. Register a small pool of staging buffers in VRAM once (N stable GDS mappings, no churn).
3. Per tensor, in file order: GDS-read its bytes in aligned chunks, each read using the next buffer in the pool (rotation), then device-to-device copy the exact bytes into a fresh CUDA destination tensor and yield (name, tensor). A sub-4KB file tail (O_DIRECT can't read it) is read buffered and copied host-to-device.
4. model.load_weights(iterator) copies each tensor into its parameter, then it frees - so peak extra VRAM is the pool (~256 MB) plus one tensor, never the whole checkpoint.

Rotating the pool is the key detail: each GDS read is a separate driver IO keyed by the buffer's address. With a single reused buffer, back-to-back reads reuse the same key and the previous IO's teardown can still be in flight when the next reuses it - which is exactly the lifecycle race. Rotating N buffers means a buffer isn't reused until N reads later, long past its teardown.

Requirements

• A GDS-capable setup with nvidia-fs + cuFile installed, and GDS actually engaged (cufile.json allow_compat_mode: false); the loader uses libcufile directly.
• The model as a single .safetensors file in the model directory (merge shards first).
• vLLM 0.26.x (the BaseModelLoader / register_model_loader plugin interface).

Verification

Loads are verified byte-for-byte against the source file (a standalone checker compares every tensor's bytes), and end-to-end by serving and confirming coherent generation - wrong weight placement is silent, so both checks matter.

Notes

• Throughput is bounded by your NVMe->GPU P2P link; the value here is reliable GDS loading of models larger than host RAM, not raw speed.
• --load-format gds_bounded is resolved via the vllm.general_plugins entry point in pyproject.toml; installing the package is enough to register it.

License

Apache-2.0.