DGX Spark vLLM Hydra Manager

This is a launcher for running vLLM models on an NVIDIA DGX Spark. It's meant to be the easiest and most reliable way to get one or several models serving on that machine, without fighting the usual problems that come from the Spark's unified memory and its brand-new Blackwell GPU.

You run one command, pick the models you want from a menu, and it launches them, balances the memory between them, and (if you want) puts a web chat UI in front. That's the whole idea. Everything underneath — the memory math, the kernel selection, the patches vLLM needs on this hardware — is handled for you.

There are a few ways it can run the models under the hood, but the recommended one is the DeepDocker backend (menu option 1): it packs the whole environment into a self-contained image so it just works the same way every time. The other backends are there and they're fine, they're just not as bulletproof. More on that below.

The DGX Spark is not a normal GPU box. The CPU and GPU share one 128 GB pool of memory instead of having separate VRAM, and the GB10 chip uses a GPU architecture (sm_121a) that stock vLLM builds and containers don't handle well yet. Most of the work in this project exists to paper over exactly those two facts so that you don't have to.

Not affiliated with NVIDIA or the vLLM project. It just runs their software.

A note on the first start (it compiles, so it's slow the first time)

The very first time you run this, it builds vLLM from source. That means it compiles vLLM's CUDA kernels specifically for the Spark's Blackwell GPU (sm_121a). This takes roughly 45 to 60 minutes and you should not interrupt it. There's no way around it — there are no prebuilt vLLM wheels for this exact chip, so the engine has to be compiled locally to run at full speed. You only pay this cost once (and again if you choose to rebuild for a newer vLLM).

After that, launching is fast, with one small exception: the first time you boot a particular model, vLLM does some just-in-time kernel compilation and CUDA graph capture, which adds a couple of minutes. The result gets cached per model, so the next launch of the same model is quick. If you're impatient or debugging, the b command in the launcher (enforce-eager) skips the graph capture step at the cost of some speed.

What you need

• An NVIDIA DGX Spark (aarch64 / ARM64, GB10, 128 GB unified memory)
• CUDA 13.0 and cuDNN 9
• Python 3.12 (python3.12-dev)
• A few system packages: build-essential, ffmpeg, git, and uv
• Docker, but only if you want to use the container backends

Getting started

git clone https://github.com/EmilHaase/DGX-Spark-VLLM-Hydra-Manager.git
cd DGX-Spark-VLLM-Hydra-Manager
chmod +x run.sh
./run.sh

The first run notices there's no environment yet and runs setup.sh for you, which is the slow compile step described above. Every run after that drops you straight into the menu.

Models are downloaded from Hugging Face. The first time it needs a token it asks you to paste one and saves it to a local .env file, so you only do that once. You can also copy .env.example to .env and fill it in ahead of time. A token is only needed for gated models.

The manual: everything it can do

When you start it you get a numbered menu. Here's what each option is for.

1. Launch DeepDocker Hydra (recommended)

Launches your selected models using the "DeepDocker" backend, which is a self-contained Docker image built specifically for this project. This is the way you should normally run models — it's the most reliable and most reproducible option, because the whole environment is frozen inside the image and can be rolled back to a previous build if an update breaks something. Needs Docker, and needs the image to be built first (option 4).

2. Launch vLLM Hydra (Local or Docker)

The everyday launcher. It asks whether you want the local backend (the vLLM you compiled into .venv) or the docker backend (the community spark-vllm-docker image). Local is the default and the fastest to iterate with. Either way, you land in the same model-selection screen described below.

3. Update vLLM (rebuild the engine and Torch)

Rebuilds the local engine from scratch — fresh PyTorch, fresh vLLM compiled from source, fresh Transformers. Use this when you want newer vLLM features or when something in the environment got into a bad state. It deletes and recreates .venv, so it's the same 45–60 minute compile as the first setup. There's a confirmation prompt so you don't trigger it by accident. It can also do a clean rebuild of the docker backend image instead.

4. Update DeepDocker image (build, roll back, or clean up)

Manages the DeepDocker image lifecycle. You can build or rebuild the image (60–90 minutes the first time), roll back to an earlier timestamped build if a new one misbehaves, and delete old images to reclaim disk. Rollback and the delete safety checks mean you can experiment with builds without risk of losing a known-good one.

5. Download a model

Type a Hugging Face repo id (for example Qwen/Qwen3-8B) and it downloads the model into models/. Anything in that folder shows up automatically in the launcher.

6. Disk Manager

Shows what's taking up space and lets you delete models and clear Hugging Face caches without leaving the tool. Handy because model weights are large and the Spark's storage fills up fast.

7. Show system versions

Prints the installed versions of vLLM, PyTorch, Transformers, and the other key pieces, plus the status of the DeepDocker image. Useful when reporting a problem or checking whether an update actually took.

8. Engine tester and benchmarks

Once engines are running, this scans ports 8000–8007 for them and offers four things:

• Latency — measures time-to-first-token
• Speed — measures raw tokens per second
• Chat — a plain interactive console to talk to a model
• Parallel load — hammers an engine with many simultaneous requests to prove it stays stable under real concurrency

9. LiMatrixLLM

An optional chat-bot frontend that speaks the Matrix protocol. It's described at the bottom of this file. It isn't included in the public repository; if it's not present, this option simply does nothing and everything else works normally.

10. Stop everything

Kills all running engines and the web UI, stops Docker containers, frees the ports, and drops the OS page cache to give the memory back. This is the clean "shut it all down" button, and it's also run automatically before every launch so you never have stale engines fighting for memory.

11. Update Hydra utilities

Updates the small helper environment used by the buffering proxy (explained under "Buffered mode" below). You rarely need to touch this.

12. Exit

Leaves the tool.

The model selection screen

Options 1 and 2 drop you into the launcher, which lists every model in models/ with an estimate of how much memory it will need and whether it fits. You pick models and adjust a few settings, then press r to launch. The commands:

Command	What it does
<id>	Select or deselect a model (just type its number)
l <0.x>	Set the overall memory-utilization target for everything
u <id> <0.x>	Pin one model's memory share; the rest is split among the others
c <id> <len>	Set a model's context length (e.g. c 1 32k)
set_mem <gb>	Tell it how much of the 128 GB pool it's allowed to use
custom <id>	Turn on and edit that model's extra config file (opens nano)
buff <id>	Run the model through the buffering proxy
ab	Switch between auto and forced kernel-backend selection
a <id>	Toggle a reasoning parser
t <id>	Toggle a tool-calling parser
b <id>	Toggle enforce-eager (skips CUDA graph capture)
parj <id> <n>	Set how many compile jobs that model may use
f	Launch using the faster fastsafetensors weight loader
r	Launch the selected models
q	Go back to the main menu

The one rule to remember: the memory shares of everything you select have to add up to no more than the whole pool. If you over-commit, the screen shows a [DANGER] warning, because two engines both trying to grab the same physical memory will crash. The balancer normally keeps you safe automatically; the manual commands are there for when you want to override it.

After you press r, it asks what you want in front of the engines: Odysseus, the Matrix bot, or nothing (headless, just the raw API on the ports).

How it runs models (the three backends)

There are three ways to actually run an engine, and they all end up at the same place — an OpenAI-compatible server on ports 8000 and up. They are not equal in practice, though, so here's the honest ranking.

DeepDocker is the one to use, and it's what I rely on. It runs your models inside a self-contained image built specifically for this project, with everything — the compiled engine, the kernels, the patches — frozen inside it. Because nothing depends on the state of your local environment, it behaves the same way every time, and if an update ever breaks something you can roll straight back to a previous build. It's the most reliable option by a clear margin, and it's menu option 1 for that reason. Build the image once (option 4) and launch from option 1 after that.

The other two work and are worth knowing about, but they're less reliable:

• local runs the vLLM you compiled into .venv. It's the quickest to start and the handiest for development or trying things, but it depends on your local environment staying healthy, so it's more prone to breaking after updates.
• docker runs the community spark-vllm-docker image. A reasonable middle ground if you'd rather use a prebuilt container than compile locally, but it's someone else's image and not tuned to this project the way DeepDocker is.

Whichever you use, output is streamed to log files under logs/, and each engine writes a small manifest under .runtime/engines/ so the other pieces can find it.

The vLLM patches (yes, they're included, and they apply themselves)

Stock vLLM has a few bugs and gaps on this exact hardware — mostly around unified memory accounting and the NVFP4 mixture-of-experts kernels on Blackwell. The fixes live in the vllm_patches/ folder in this repo, as complete drop-in replacement modules. The matching .patch and .orig files show exactly what changed versus upstream, so nothing is hidden.

You do not apply these by hand. It happens automatically every time you launch:

• The local backend copies the patch files over the installed vLLM inside .venv right before starting the engine. (This is also why editing vLLM in .venv directly is pointless — it gets overwritten on the next launch. Edit the files in vllm_patches/ instead.)
• The docker backend mounts the patch files read-only into the container.
• The deepdocker backend doesn't use the patch overlay — its image is built around a pinned vLLM release that already carries the fixes it needs, which is part of why it's the most reproducible backend.

So on local and docker the patches are always in effect and always match what's in this repo, and deepdocker gets equivalent fixes from its pinned engine build.

How the memory math works

The number SYSTEM_TOTAL_GB = 128 in config.py is the physical unified pool. When you set a utilization target, it's measured against the usable part of that pool (which you can adjust with set_mem). For each model it estimates roughly the weight size, plus about 4 GB per 32k of context, plus a couple of GB of baseline. On top of that, the local backend hands vLLM an explicit KV-cache byte budget, because vLLM's own profiler misreads free memory on a unified-memory system (the OS page cache looks like "used" memory and throws it off). Getting this right is the main reason you can run several models at once here without them crashing into each other.

Per-model settings

If a particular model needs extra vLLM flags, put them in a file named after the model's folder: custom_config/<model-folder>.conf for the local and docker backends, or deepdocker_config/<model-folder>.conf for DeepDocker. Turn it on with custom <id> in the launcher, which opens it in an editor. You don't need to set things like the port, the model path, the memory fraction, or context length there — those are always managed for you and are stripped out of the file automatically so they can't conflict. DeepDocker config files can also set container environment variables with -e KEY=VALUE lines.

A few things are decided just from the model's folder name: names containing NVFP4 get the FP4 kernels, and names containing qwen3, minimax, or gemma-4 get sensible reasoning and tool parsers by default.

Buffered mode (for diffusion and block-generating models)

Some newer models (like DiffusionGemma) generate text in blocks rather than one token at a time, and some models stop early. Either way, a normal client can end up seeing only the first chunk. Buffered mode fixes that. When you turn it on with buff, vLLM runs hidden on a private port and a small proxy takes the public port, stitches the blocks together until the model is actually done, and streams the result out normally so tools like Odysseus behave. It also keeps the model's reasoning and final answer separated correctly across those blocks.

Ports

• Engines: 8000–8007, assigned in the order you selected the models
• Odysseus: 7000 (self-hosted AI workspace; cloned on first launch)
• Buffered mode: the real engine hides on the port plus 1000, and the proxy takes the normal port

Running the tests

python3 -m pytest tests/

Run it from the repo root. Tests that depend on the optional Matrix bot are skipped automatically if its dependencies aren't installed.

Optional: the Matrix chat bot (LiMatrixLLM)

Besides Odysseus, the launcher can put a Matrix chat bot in front of your models, with encrypted-at-rest credentials and a vector memory. This part is maintained separately and is not included in this repository — the credentials, encrypted data, and chat history stay on the operator's machine and never get published. If the matrix/ folder isn't there, nothing else is affected; only the Matrix menu option goes quiet.

Credits

This project is a launcher and a set of fixes. The heavy lifting is done by other people's work, and it's worth naming it:

• vLLM — the actual inference engine
• PyTorch and NVIDIA CUDA / CUTLASS — the compute stack
• FlashInfer, Triton, and xgrammar — the accelerated kernels
• Hugging Face Transformers and Hub — models and tokenizers
• spark-vllm-docker by eugr — the community Docker backend
• Odysseus — the web workspace frontend

The vllm_patches/ in this repo are modifications of vLLM source; see the .patch files for the exact changes and vLLM's own license for the terms on that code.

License

This project is released under the MIT License — see LICENSE. In short: do what you want with it, keep the copyright notice, and there's no warranty. The third-party projects listed above keep their own licenses.