Ornith Local 192K on 6 GB VRAM

A fully local Windows coding-agent stack built around OpenCode, llama.cpp CUDA, and Ornith 1.5 9B. This profile is tuned for NVIDIA laptop GPUs with only 6 GB of dedicated VRAM while still exposing a 196,608-token context window.

OpenCode -> OpenAI-compatible llama.cpp server -> Ornith 1.5 9B -> CUDA

No inference request needs to leave the machine.

Why this profile exists

Ornith's Q4_K_M weights nearly fill a 6 GB GPU by themselves. Keeping every model layer and a 192K KV cache on the GPU leaves too little room for Windows. This configuration uses a deliberate split:

• 21 model/output layers on the GPU and 12 in CPU RAM;
• Q4_K_M model weights for stronger coding quality;
• Q4_0 K/V cache on the GPU;
• Flash Attention enabled;
• one 196,608-token slot;
• an 8,192-token OpenCode output/compaction reserve.

The result gives up some model-layer offload to keep long-context attention on the GPU. See BENCHMARK.md for measured data and the rejected faster-but-unsafe alternatives.

Requirements

• Windows 11
• NVIDIA CUDA-capable GPU with 6 GB VRAM
• 24 GB system RAM recommended for the 192K profile
• current NVIDIA driver
• PowerShell 5.1 or newer
• Python 3.11+ for benchmarks
• Node.js and OpenCode

Install OpenCode if needed:

npm install -g opencode-ai

Install

1. Clone the repository and enter it.
2. Download an official Windows CUDA build from the llama.cpp releases page.
3. Extract llama-server.exe and its CUDA DLLs into runtime\.
4. Download Ornith-1.5-9B-Q4_K_M.gguf from the official Ornith GGUF repository into models\.
5. Generate local configuration and add the launcher to your user PATH:

Set-ExecutionPolicy -Scope Process Bypass
.\setup.ps1

The runtime, model, local configuration, logs, and machine inventories are ignored by Git.

Use

Open a new terminal, enter any repository, and launch the agent:

cd path\to\your-project
ornith

The wrapper verifies the local server, starts it when necessary, and launches OpenCode with only ornith-local/ornith-1.5-9b-local enabled.

Useful controls:

.\start-ornith.ps1
.\check-ornith.ps1
.\check-ornith.ps1 -ProbeTools
.\stop-ornith.ps1

Configuration

setup.ps1 creates the ignored files settings.json and opencode.json from their public examples. Relative model and runtime paths are resolved against the repository root, so the checkout can live anywhere.

The default 192K settings are:

Setting	Value
Context	196,608
GPU layers	21
CPU-resident model layers	12
Batch / uBatch	256 / 128
Flash Attention	on
K/V type	Q4_0 / Q4_0
K/V location	GPU
Parallel slots	1

If your machine has less than 24 GB RAM, reduce context or GPU layers. Do not increase GPU layers without measuring peak VRAM under load.

Experimental full 256K profile

Ornith reports a 262,144-token training context. The complete window was also validated on the same 6 GB GPU, but it requires a different memory split:

Setting	Value
Context	262,144
GPU-resident model layers	20 of 33
CPU-resident model layers	13 of 33
Batch / uBatch	256 / 128
K/V type	Q4_0 / Q4_0
K/V location	GPU

A 262,012-token retrieval prompt occupied 99.95% of the slot and recalled a marker from the beginning exactly. The request took about 70 minutes and generation near full occupancy fell to 2.41 tok/s. Minimum measured free VRAM was only 51 MiB, so this is a maximum-capacity experiment rather than the recommended daily profile. See BENCHMARK.md for the complete measurements and comparison with system-RAM KV.

Tool calling and safety

The server enables llama.cpp's Jinja chat template and Ornith reasoning/tool format. The supplied OpenCode policy allows routine repository reading, edits, Git inspection, and common local test commands. Unknown shell commands require approval. Recursive deletion, destructive Git cleanup/reset, force pushes, disk formatting, registry deletion, and obvious credential-extraction commands are denied.

These rules reduce accidents; they are not an operating-system sandbox. Review approval requests before accepting them.

Benchmarks

Install the Python benchmark dependencies, stop the normal server before a sweep, and run:

python -m pip install psutil requests pytest
.\benchmark-ornith.ps1 -Mode quick -Repetitions 2
.\benchmark-ornith.ps1 -Mode tools
.\benchmark-ornith.ps1 -Mode agent

The experimental full-window probe is separate because it uses a different server memory split and can take more than an hour:

.\stop-ornith.ps1
.\benchmark\start_full_context_candidate.ps1 `
  -ServerPath .\runtime\llama-server.exe `
  -ModelPath .\models\Ornith-1.5-9B-Q4_K_M.gguf
python .\benchmark\full_context_probe.py

Generated raw evidence remains local because it can contain absolute paths and machine inventory. Only sanitized aggregate results are committed.

Limits

• A 192K allocation does not make every 192K request fast. Generation measured near 190K active tokens is much slower than generation near an empty cache.
• Q4_0 KV cache is a memory/quality compromise.
• One slot means one active inference stream.
• A 9B local model remains less reliable than frontier cloud agents on large, ambiguous architectural work. Keep tasks scoped and require tests.

Privacy

The server binds to 127.0.0.1, runs llama.cpp in offline mode, disables its web UI, and configures OpenCode with a single local provider. Sharing and web tools are disabled in the example configuration.

License

MIT