Docker GPU: llama-server is never installed after downloading a GGUF model

[vickers8]

Docker GPU: Local GGUF models fail to launch (llama-server / llama-cpp-python missing?)

Hi,

I'm new to Odysseus and I'm trying to use the built-in local model support with Docker.

Environment

• Ubuntu 24.04.4 LTS
• Current dev branch (6efc07c5fc7c49f957581504631752fed39587b3)
• docker-compose.gpu-nvidia.yml
• NVIDIA RTX 5070 Ti (16 GB)
• NVIDIA Driver 580.159.03
• Docker 29.6.1
• Docker Compose v5.3.1
• NVIDIA Container Toolkit configured and working

GPU passthrough

GPU passthrough appears to be working correctly:

docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu24.04 nvidia-smi

The GPU is detected successfully.

Steps to reproduce

1. Fresh clone of the current dev branch.
2. Started Odysseus with:

docker compose -f docker-compose.gpu-nvidia.yml up -d --build

3. Opened Cookbook.
4. Downloaded:

lmstudio-community/gemma-4-12B-it-QAT-GGUF

5. Clicked Run.

Expected

The model launches and becomes available for chat.

Actual

Odysseus logs show it attempting to execute:

CUDA_VISIBLE_DEVICES=0 llama-server --model ...

However, every chat request fails with:

Cannot reach http://localhost:8000
HTTP 503

Eventually the model endpoint disappears, and Settings → Added Models is empty.

Additional debugging

Inside the container:

which llama-server

No output.

llama-server --version

Returns:

bash: llama-server: command not found

Searching the filesystem:

find / -name llama-server 2>/dev/null

Returns no results.

Checking for llama-cpp-python:

pip show llama-cpp-python

Returns:

WARNING: Package(s) not found: llama-cpp-python

Also:

pip list | grep -i llama

Returns no output.

Trying to reach the local server:

curl http://localhost:8000/v1/models

Returns:

curl: (7) Failed to connect to localhost port 8000 after 0 ms: Could not connect to server

Notes

Earlier during startup I also saw output similar to:

Native llama-server not found — building from source...
Could NOT find Vulkan (missing: Vulkan_LIBRARY Vulkan_INCLUDE_DIR glslc)
NVIDIA detected but installed llama-cpp-python is CPU-only — reinstalling with CUDA wheel index for GPU offload...

I installed the missing Vulkan development packages (libvulkan-dev, vulkan-tools, glslang-tools), but the behaviour didn't change.

I also looked at issue #5287, but my checkout doesn't contain the INSTALL_CUDA_TOOLKIT changes yet (grep -R "INSTALL_CUDA_TOOLKIT" . returns no matches).

Am I missing a bootstrap step for llama-server / llama-cpp-python, or is this a regression in the current dev branch?

Thanks!

[AteebNoOne]

The reason installing libvulkan-dev on your host didn't change the behavior is because Docker containers are completely isolated. Installing development headers on your Ubuntu 24.04 host does not make them available inside the Docker container where the actual compilation is happening.

The Root Cause

The llama-server: command not found and missing llama-cpp-python errors are caused by a silent build failure inside the container.

When you start the container, the startup script notices you want GPU offload, so it attempts to compile llama-cpp-python and llama-server from source. However, compiling CUDA applications requires the CUDA Toolkit (specifically the nvcc compiler and cmake).

Because the default docker-compose.gpu-nvidia.yml usually pulls a base or runtime NVIDIA image to save bandwidth, it lacks these heavy build tools. The background compilation fails, the packages are never installed, and the script moves on—leaving you with HTTP 503 errors when it tries to start the non-existent server.

The Fix

You need to switch the Docker image to a devel (development) variant so the container has the necessary compilers to build the binaries.

Step 1: Open the Dockerfile that is used by docker-compose.gpu-nvidia.yml (usually named Dockerfile or Dockerfile.gpu).

Step 2: Look at the very first line. It likely says something like:

FROM nvidia/cuda:12.2.2-runtime-ubuntu22.04
# OR
FROM nvidia/cuda:12.x.x-base-ubuntu24.04

Step 3: Change runtime (or base) to devel. Keep the version numbers exactly the same. For example:

FROM nvidia/cuda:12.2.2-devel-ubuntu22.04

Step 4: Rebuild the container from scratch to pull the new image and re-trigger the build:

docker compose -f docker-compose.gpu-nvidia.yml down
docker compose -f docker-compose.gpu-nvidia.yml up -d --build --force-recreate

Once the container boots up with the devel image, watch the logs. The Native llama-server not found — building from source... step will actually succeed this time, and your RTX 5070 Ti will be able to load the GGUF model correctly.

[vickers8]

Thanks! That makes sense about the host packages not affecting the container.

One thing I'm unsure about though: the current dev branch Dockerfile appears to build from python:3.14-slim, not an nvidia/cuda base image.

Earlier during the build I saw:

load metadata for docker.io/library/python:3.14-slim

Also, inside the container I checked the following:

pip show llama-cpp-python

which returns:

WARNING: Package(s) not found: llama-cpp-python

and:

pip list | grep -i llama

returns no output.

Also:

which llama-server

returns no output, and:

find / -name llama-server 2>/dev/null

returns no results.

Given that neither llama-cpp-python nor llama-server appear to exist in the container, do you still think switching to a CUDA devel image is the correct approach for the current dev branch, or does this point to a different bootstrap/install failure?

I'm happy to test anything if it helps.