What four 2017-era GTX 1080 Ti still do in 2026 — measured, not guessed.
CUDA 13 dropped compilation support for Pascal. PyTorch 2.8 dropped SM 6.1. vLLM, SGLang, TGI, and TensorRT-LLM all require compute capability ≥ 7.0. By every current compatibility matrix, these cards are finished.
They aren't. This repo documents a working private research assistant — chat, cited web search, RAG over local documents, multi-hour deep research, and a coding agent — running entirely offline on hardware that modern tooling has abandoned.
pp512 (prefill) 1035 tok/s tg128 (decode) 60.5 tok/s
3 concurrent users at 1.91× throughput KV cache for 4×32K ctx: ~500 MiB total
Model: Qwen3.6-35B-A3B (MoE, ~3B active) Q4_K_M across 3x 11 GB
This is a reference implementation and a set of field notes — not a turnkey installer. Paths, usernames, and GPU IDs are specific to one machine. Read it, take what applies, adapt the rest. The reasoning and the measurements are the point; the scripts are the appendix.
Almost every "run LLMs locally" guide assumes a recent GPU and a machine you fully control. This documents the opposite: Pascal-era cards, on a shared multi-user server, with time-limited root access, no firewall control, and a campus network that requires per-host approval. That combination is closer to most institutional reality than a clean cloud VM, and it's badly under-documented.
| If you want to know… | Read |
|---|---|
| Will this run on my old GPU at all? | docs/03-benchmarks.md |
| Why llama.cpp and not vLLM? | docs/01-why-these-choices.md |
| It broke / it's slow / it lies about working | docs/04-troubleshooting.md |
| What do these terms mean? | docs/05-concepts.md |
| I also have a shared box and no root | docs/06-constraints-pattern.md |
| Engine | Status | Notes |
|---|---|---|
| llama.cpp | ✅ Supported | Build with -DCMAKE_CUDA_ARCHITECTURES=61 against CUDA 12.x. The only maintainable option. |
| Ollama | Wraps llama.cpp; hides the KV-cache/flash-attn/batch knobs you need on constrained VRAM. | |
| vLLM / SGLang / TGI | ❌ No | Require compute capability ≥ 7.0; PyTorch 2.8 dropped SM 6.1. |
| TensorRT-LLM | ❌ No | Targets tensor cores, which Pascal lacks. |
| ExLlamaV2 / V3 | ❌ No | Ampere-era assumptions throughout. |
Hard rule: never install a CUDA 13 toolkit on a Pascal host. Binaries built against it cannot target SM 6.1.
browser ──► Open WebUI ──┬──► SearXNG (private web search)
├──► Qdrant (vector store)
└──► llama.cpp ──► Qwen3.6-35B-A3B (GPUs 0,1,3)
llama.cpp ──► BGE-M3 embeddings (GPU 0)
status page (traffic light) ──► /metrics
Prometheus + Grafana ──► GPU / host / inference telemetry
Everything is local. No API keys, no data leaving the machine, no per-token cost.
Low-active-parameter MoE is the right shape for bandwidth-bound hardware. A 35B model with ~3B active parameters per token decodes at roughly the speed of a 3B dense model while reasoning like a much larger one. On Pascal — where memory bandwidth, not compute, is the ceiling — this is the single most important architectural choice.
Prefill is the bottleneck, not decode. No tensor cores means ingesting a long prompt is far slower than generating from it. This is why the practical context is 32K rather than the advertised 256K, and why RAG beats stuffing documents into context.
Concurrency works better than expected. Three simultaneous users get 1.91× the throughput of one, because ~3B active parameters don't saturate memory bandwidth on a single stream. Full table in benchmarks.
KV cache is nearly free on GQA models. Four slots at 32K each cost about 500 MiB total. Context length is not the constraint; prefill time is.
| Runs anywhere | scripts/verify.sh · scripts/status/status.py · benchmarks/run-concurrency-test.sh |
|---|---|
| Reference data | docs/03-benchmarks.md — answers "will this work on my old GPU?" |
See SECURITY.md. Short version: generate your own secrets, decide deliberately which ports are network-visible, and understand that Docker's iptables rules bypass UFW — a published port is exposed regardless of what your firewall says.
This is one person's working lab, documented as built. It is not production-grade, not a supported product, and the numbers come from one specific machine. Where something is an estimate rather than a measurement, it says so.
Known limitations are documented rather than hidden — including a genuine hardware fault (one GPU faulting under multi-GPU load) that took three days to diagnose and is written up in full, because the diagnostic path is more useful than the conclusion.
MIT for the scripts and configuration. The documentation is CC BY 4.0 — reuse it, adapt it, credit it.