local-ai-stack

A toolkit for standing up a complete local AI environment on a single CUDA workstation: GPU setup and verification, Hugging Face model serving with quantization so large models fit on one card, a small inference API, and an agent example on top. It takes a fresh high-end machine and turns it into a working, optimised local AI box, with the practical parts (drivers, GPU memory, model loading, throughput) handled deliberately rather than left to guesswork.

Reusable setup toolkit, not a single fixed deployment. Bring up as much or as little of the stack as you need on your own hardware.

What it sets up

• GPU environment - verifies the CUDA toolkit and driver, checks the GPU is visible to the frameworks, and reports memory and compute capability, so problems surface before models load, not after.
• Model serving (Hugging Face) - loads transformer models from the Hub and serves them locally, with quantization (8-bit / 4-bit) so large models fit in available VRAM.
• Inference API - a small local HTTP endpoint for text generation / embeddings, so other tools and agents can call the models over a stable interface.
• Agent example - a worked example of an AI agent (tool use + a multi-step loop) running entirely against the local models.
• Benchmarking - measures tokens/sec, latency, and VRAM use so you can see the effect of quantization and batching and pick settings that fit the card.

Why each piece is there

• Fitting the model on the card. A high-end workstation still has finite VRAM; a model that is too big either fails to load or spills and crawls. Quantization (4-bit / 8-bit) and sensible dtype choices are the difference between "fits and runs" and "out of memory."
• GPU actually being used. It is common to think a model is on the GPU when it is quietly running on CPU. The setup step verifies device placement.
• Memory management across requests. Loading/unloading models, KV-cache growth, and batching move VRAM around; the serving layer keeps this predictable so long sessions don't drift into out-of-memory.
• A stable interface. A small API in front means serving details can change without breaking everything on top.

Stack

CUDA (toolkit + driver verification, device placement, memory reporting), Hugging Face transformers, quantization (4-bit / 8-bit), a small Python serving layer, an agent (tool-calling loop) over the local models, benchmark scripts for tokens/sec, latency, and VRAM. Runs on Linux (and Linux under WSL2 on Windows workstations).

Structure

local-ai-stack/
├── setup/   check_gpu.py (verification + report), environment.md
├── serve/   loader.py (load + quantize + placement), api.py (endpoint), memory.py (VRAM-aware)
├── agent/   example_agent.py (tool-using loop)
├── bench/   benchmark.py (tokens/sec, latency, VRAM)
└── requirements.txt

Bringing it up

# 1. verify the GPU is ready
python setup/check_gpu.py

# 2. serve a model (quantized to fit the card)
python serve/api.py --model <hf-model-id> --quant 4bit

# 3. talk to it
curl localhost:8000/generate -d '{"prompt": "hello", "max_tokens": 128}'

# 4. run the agent against the local model
python agent/example_agent.py

# 5. benchmark
python bench/benchmark.py --model <hf-model-id> --quant 4bit

WSL2: the same steps run under a Linux distro on Windows once the GPU is exposed to WSL and the CUDA toolkit is installed inside it.

What it demonstrates

• CUDA workstation setup - getting the GPU, driver, toolkit, and frameworks aligned so models actually run on the card, and diagnosing when they don't.
• Hugging Face serving - loading and serving Hub models locally with quantization tuned to the available VRAM.
• Optimization - measuring and improving tokens/sec, latency, and memory footprint.
• Integration - a stable local API and a working agent on top.

Notes

• Reusable toolkit for a local AI environment; the specific models, hardware, and which pieces you run are chosen per machine.
• Quantization and batching settings are exposed on purpose, so they can be tuned to the actual card rather than hard-coded.

MIT licensed.