Skip to content

README.md

Latest commit

 

History

History
268 lines (197 loc) · 9.65 KB

README.md

File metadata and controls

268 lines (197 loc) · 9.65 KB
Raw

Docker GPU Allocator

Exclusive GPU allocation for ordinary Docker and Compose on one Linux host. --gpus 1 gets an unused GPU by UUID, not Docker's usual race to index 0.

$ docker --context gpu-local run --rm --gpus 1 nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L
GPU 0: NVIDIA ... (UUID: GPU-the-selected-unused-device)

Quickstart

You need Linux, Docker, Go 1.23+, make, an NVIDIA driver, and the NVIDIA Container Toolkit.

1. Verify Docker can use the GPU

The proxy allocates GPUs; the NVIDIA Container Toolkit injects them into containers. Confirm ordinary Docker GPU access works first:

nvidia-smi
sudo docker run --rm --gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi

If Docker reports could not select device driver "nvidia", install the NVIDIA Container Toolkit using its official guide, then configure Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo docker info | grep -i nvidia

Docker's default runtime should remain runc; an nvidia runtime only needs to be listed under docker info. Repeat the Docker GPU test before continuing.

2. Build and install the allocator

From this repository:

make build
sudo ./scripts/install.sh

The installer starts the service and prints its logs if startup fails.

3. Create a Docker context

If your user is not already in the docker group:

sudo usermod -aG docker "$USER"

Log out and back in after changing groups. Then create the context as the user who will run Docker:

docker context create gpu-local --docker host=unix:///run/docker-gpu/docker.sock

Do not create the context with sudo; Docker contexts are per-user.

4. Test allocation

docker --context gpu-local run --rm --gpus 1 nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L
sudo /usr/local/libexec/docker-gpu-proxy status

5. Use Compose

Add a normal Docker GPU reservation to compose.yaml:

services:
  trainer:
    image: nvidia/cuda:13.0.0-base-ubuntu24.04
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
docker --context gpu-local compose up

Always use --context gpu-local or set DOCKER_CONTEXT=gpu-local. Request GPUs with --gpus or Compose reservations—not --runtime=nvidia or NVIDIA_VISIBLE_DEVICES.

How allocation works

For POST /containers/create, the proxy:

  1. discovers physical NVIDIA GPUs with nvidia-smi (UUID identity);
  2. inspects every Docker container (including stopped) plus active host compute processes;
  3. locks allocator state and subtracts claims and durable markers;
  4. rewrites Count: N to unused UUID DeviceIDs and adds lease labels;
  5. rolls back definite create failures; reconciles ambiguous ones from state + labels.

Allocation is all-or-nothing (409 Conflict if capacity is short). GPUs stay claimed for the container's lifetime — including while stopped — because docker start can revive the same HostConfig. Remove the container to free them.

On NUMA hosts, multi-GPU requests prefer one node before spilling. See Topology-aware placement.

Existing non-proxy containers are adopted as claims (DeviceRequests, NVIDIA runtimes, NVIDIA_VISIBLE_DEVICES, CDI, raw /dev/nvidiaN, DRM, privileged, GPU cgroup rules). Unrewritten count requests claim the full inventory (fail-closed). Stale claims for departed UUIDs are ignored.

MIG / fractional sharing / time-slicing are not supported.

Compared to alternatives

Stock Docker and the NVIDIA Container Toolkit provide GPU passthrough, not allocation. --gpus 1 does not mean "pick an unused GPU". Exclusive access is normally an orchestrator concern.

Approach Trade-off
Manual device=GPU-... No automatic unused-GPU selection; races under concurrency
Kubernetes / Nomad / Slurm Real scheduling, but not stock docker run / Compose
Custom GPU queues Clients leave the Docker API
Docker socket ACL proxies No GPU allocation

This project keeps the Engine/SDK/Compose APIs and inserts atomic exclusive allocation by proxying the Engine socket on a single host.

Roadmap

Implemented: NUMA-aware GPU selection (prefer/strict), GPU-local CPU cores and memory admission via request labels, status / explain, optional Prometheus metrics.

Remaining: NVLink/NVSwitch grouping; richer resource classes (model, VRAM, pools).

Non-goals: multi-host scheduling, job queues, deployments, ingress, autoscaling.

Install details

make check && make build
sudo ./scripts/install.sh

The installer copies the binary and configures systemd. The service listens on /run/docker-gpu/docker.sock, forwards to /var/run/docker.sock, and stores leases in /var/lib/docker-gpu. Default socket mode is 0660 / group docker. See /usr/local/libexec/docker-gpu-proxy -h for flags.

State schema is v3 (CPU/memory leases, absolute pending expiry, daemon ID binding, reserved-CPU init). Older binaries reject newer state. Empty legacy state adopts the current daemon; legacy state with leases needs -adopt-unbound-state after operator verification. Rollback means draining managed containers and clearing or migrating the state directory.

Manual run without install:

sudo ./bin/docker-gpu-proxy \
  -listen /run/docker-gpu/docker.sock \
  -upstream /var/run/docker.sock \
  -state-dir /var/lib/docker-gpu

Usage notes

  • Explicit --gpus device=GPU-... is allocated and fails if busy.
  • Rejected bypasses: NVIDIA runtime aliases, manual NVIDIA_VISIBLE_DEVICES / DOCKER_RESOURCE_GPU, opaque OCI annotations, NVIDIA sentinel mounts, CDI in Devices, raw NVIDIA/DRM mappings, privileged creates, GPU cgroup rules.
  • Accepted creates get an authoritative visibility guard (leased UUIDs, or void for non-GPU). Custom Toolkit swarm-resource env names are not supported.
  • Non-CDI Devices require an absolute, canonical, existing char/block node under /dev (no symlinks/directories). Departed /dev/nvidiaN claims by minor only when that minor is gone.

Logs, attach, exec, pulls, and most other API traffic pass through. Topology updates are restricted — see below.

Topology-aware placement

NUMA layout comes from sysfs; GPU→node from PCI bus ID. Multi-GPU count requests prefer a single node (tightest fit), then spill.

docker --context gpu-local run --gpus 1 \
  --label io.github.docker-gpu.request.cpus=4 \
  --label io.github.docker-gpu.request.memory=64g \
  --label io.github.docker-gpu.request.topology=strict \
  IMAGE
Label Effect
request.cpus=N Lease N physical cores (SMT siblings) near the GPUs → CpusetCpus
request.memory=SIZE Admit against one GPU-local NUMA node → Memory + CpusetMems
request.topology=strict Fail instead of cross-node fallback (prefer is default)

Labels need a GPU DeviceRequest and conflict with manual cpuset/memory fields. -reserved-cpus 0-1 keeps housekeeping cores off-limits (immutable per state dir once set). CPU leases coordinate pinned containers; they do not isolate from host processes or containers without cpusets. Memory admission covers CpusetMems-pinned workloads only.

Do not race manual cpuset creates against labeled ones. Topology-changing docker update (CpusetCpus / CpusetMems / Memory) is rejected by default; recreate instead, or pass -allow-uncoordinated-topology-updates (lease-owned fields stay protected).

Status / explain / metrics

sudo docker-gpu-proxy status          # inventory, claims, leases, NUMA
sudo docker-gpu-proxy status -json
sudo docker-gpu-proxy explain -gpu-request count=2 -cpu-cores 8 -memory 64GiB -topology strict

explain is advisory (shared lock only; capacity can change immediately). Exit 0 fit, 2 capacity, 3 invalid, 1 inspection failure.

Metrics are off by default: -metrics-listen 127.0.0.1:9090. Exports docker_gpu_proxy_* aggregates only — no container/lease/UUID labels.

Exclusivity boundary

Clients that create GPU containers must use the allocator socket. Anyone with /var/run/docker.sock can bypass it. For a strong host guarantee, restrict the real socket to the proxy and expose only /run/docker-gpu/docker.sock.

This is exclusive scheduling among cooperating Docker clients, not a security boundary between untrusted users. Outside the guarantee: real-socket clients, non-NVIDIA custom runtimes, host-root /dev tricks, Toolkit policy changes behind a running proxy, and graphics/KMS workloads not visible to nvidia-smi compute queries.

Non-empty HostConfig.Annotations are rejected (CDI prefixes are configurable in the Toolkit). Use labels for app metadata.

Failure behavior

  • Inventory / Docker / corrupt state failures are fail-closed.
  • Runtime config and daemon ID are refreshed; NVIDIA default or refresh failure blocks creates. Create uses a pinned backend connection after /info validation.
  • Backend error → release pending lease. Dropped connection → ambiguous; lease held until label reconcile or pending TTL (default 2m, heartbeated).
  • -docker-create-timeout (default 2m) treats timeout as ambiguous.
  • Confirmed markers last for the labeled container lifetime.
  • Proxies sharing a state dir serialize on a filesystem lock and must share the same Docker daemon and reserved-CPU config.

Development

make check
go test -count=20 ./...

Tests use fake NVIDIA inventory and a fake Docker Engine. Real-host acceptance belongs on a dedicated idle GPU machine.