llama_kit — local LLM offload toolkit for coding agents

A Qoder-CLI skill + standalone toolkit that routes routine, token-heavy coding work (boilerplate, docstrings, refactors, test scaffolding, translations, file summarization, code edits) to a local llama.cpp server instead of cloud models. Zero cloud tokens for delegated work.

Tested only with Qoder CLI so far — the toolkit itself is agent-agnostic (plain Python + shell), but the Claude Code and GitHub Copilot integrations are untested templates.

Requirements

• Python 3.10+ (standard library only — no pip installs)
• A running (or auto-startable) llama-server with the OpenAI-compatible API

Quick start

1. Copy this folder to ~/.qoder/skills/local/ (or anywhere).

2. Create your machine-private config.json from the template and edit it:

   cp config.example.json config.json

   {
     "url": "http://localhost:8080",
     "workdir": "C:/path/to/llama.cpp",
     "start_command": "build/bin/Release/llama-server.exe -m models/your-model.gguf --port 8080 -ngl 99 -c 80000 -np 1 --cache-prompt",
     "startup_timeout_s": 300
   }

   config.json is gitignored; never commit your own paths/model names.

3. Call the delegate:

   py -3 bin/llama_delegate.py "your task prompt"

   If the server is down it is launched automatically from start_command.

Usage

All commands run through one script: bin/llama_delegate.py (Windows: py -3, Linux/macOS: python3).

# Ask the local model a task
py -3 bin/llama_delegate.py "write a docstring for: def f(x): return x * 2"

# Give it a file — the script reads it itself, cloud context stays empty
py -3 bin/llama_delegate.py "find the bug in this file" --file src/utils.py

# Big file? Take only one symbol instead of the whole file
py -3 bin/llama_delegate.py "explain this" --file big.py --focus _run_training

# Or compress a huge file into a summary first (cached by content hash)
py -3 bin/llama_delegate.py --summarize huge_file.py

# Let the model edit a file (SEARCH/REPLACE blocks, applied atomically)
py -3 bin/llama_delegate.py --patch src/utils.py "rename to snake_case" --dry-run

# ... and make it verify + self-fix: run a command after patching, and if it
# fails, the model repairs the file and re-runs it until it passes
py -3 bin/llama_delegate.py --patch src/utils.py "optimize parse()" --test "py -3 -m pytest tests -q"

# Focused helper modes (read piped stdin, keep the cloud agent out of the loop)
pytest -q 2>&1 | py -3 bin/llama_delegate.py --triage        # FAILING/ERROR/LIKELY_CAUSE/NEXT
git diff | py -3 bin/llama_delegate.py --review               # SUMMARY/RISKS/FOCUS pre-review
git diff | py -3 bin/llama_delegate.py --commit-msg           # Conventional Commits message
some_big_log.txt | py -3 bin/llama_delegate.py --summarize-stdin
py -3 bin/llama_delegate.py --ask "what does parse() return on error?" --file src/utils.py

# Iterate in a multi-turn conversation (one history file per task)
py -3 bin/llama_delegate.py "make it simpler" --session task1.json

# Guaranteed-schema JSON output
py -3 bin/llama_delegate.py "extract the function names" --json-schema '{"type": "object"}'

# Usage report with estimated cloud-cost savings / health check
py -3 bin/llama_delegate.py --report --savings
py -3 bin/llama_delegate.py --health-only

How cloud tokens are saved

• --file: pass paths, not content — the script loads files itself, so the cloud agent never reads them into its own context.
• --focus: with --file, inline only the region defining a symbol instead of the whole file. Recognized: Python def/class; JS/TS function/interface/type (also with export); Rust fn/struct/ enum/impl; Go type. Not recognized: Go receiver methods (func (r *T) Name) and brace-delimited class bodies.
• --grep: with --file, inline only the lines matching a regex (case-insensitive) plus --grep-context lines (default 3), numbered, with ... between non-adjacent regions. Pure keyword lookup — no model call, cheaper than --focus. Falls back to the whole file when nothing matches.
• Auto-summary fallback: files that overflow the token budget are replaced by a locally generated summary automatically.
• Summary cache: summaries are cached by content hash in summary_cache/; unchanged files are never re-summarized.
• --patch: the model emits only SEARCH/REPLACE blocks, not whole files.
• Self-fix loop: --patch ... --test "<cmd>" runs the command after applying; on failure the model gets the error output, fixes the file and re-tests (up to --test-retries, default 3; the pre-patch file is restored if it never passes). Same verify→fix→retest loop an agent would drive, zero cloud tokens. Caveat: "tests pass" is not "the fix is good" — a small quantized model can occasionally patch its way to green, so spot-check the diff before trusting it unattended.
• Text modes: --triage, --review, --commit-msg, --summarize-stdin and --ask give the agent one-call, structured answers for test logs, diffs and repo questions — the raw log/diff never enters cloud context, only the short local answer does.
• Response cache: identical requests (chat, patch, text modes) are served from response_cache/ instead of the GPU — instant repeats, no usage logged. Bypass with --no-cache.
• Honest output: every mode instructs the model to say so when it is unsure or the context is insufficient instead of guessing, and to accept corrective feedback and revise (works across --session turns).
• Session compaction: long --session histories are condensed locally.
• Budget guard: exact /tokenize counting against a 70k prompt cap (override with $LLAMA_KIT_MAX_PROMPT_TOKENS; the legacy $COPILOT_PROVIDER_MAX_PROMPT_TOKENS still works as a fallback).

Measured savings (RTX 5070 Ti, 27B Q3 model)

Technique	Cloud-context cost
Reading a 57 KB source file into the agent	~13,000 tokens
Same file via --file --focus <symbol>	3–15% of that (1,700–8,700 chars)
Same file auto-summarized locally	~1,500 tokens (cached forever)
Prefix caching (--cache-prompt), 20k-token prefix	18.0s → 2.75s (6.5x)
Delegated task itself	0 cloud tokens

Every routine task routed here converts cloud input+output tokens into free local GPU time; --report --savings estimates the dollar value, broken down per mode (sorted by savings) so you can see which delegations pay off most.

Model prices for --savings

By default the estimate assumes $3/$15 per 1M input/output tokens. Price it against a real cloud model instead:

py -3 bin/llama_delegate.py --list-prices                    # show the table
py -3 bin/llama_delegate.py --report --savings --cloud-model sonnet-5
py -3 bin/llama_delegate.py --report --savings --cloud-model "bailian-intl/qwen3.8-max-tp"

The table (in llama_kit/pricing.py) covers Claude Opus/Sonnet/Haiku/Fable, GPT-5.6 sol/terra/luna, GPT-5.5(+pro), GPT-5 nano, Gemini 3.1 Pro / 3 Flash, and Qwen3-Max/3.8-Max, as of 2026-08-30 — vendor prices move, so check --list-prices notes or pass explicit --input-price/--output-price (those always win over --cloud-model).

Exit codes

Code	Meaning
0	success
1	malformed model response / patch failed
2	bad input (empty prompt, missing file, bad schema)
3	PROMPT_TOO_LONG — prompt exceeds the safe budget
4	LOCAL_MODEL_UNAVAILABLE — server unreachable/autostart failed

Files

• llama_kit/ — the package: delegate.py (CLI), completion.py (request + slot lock + retry), summarize.py, patch.py, sessions.py, report.py, pricing.py, telemetry.py, client.py, tokens.py, config.py
• tests/ — pytest suite (run: py -3 -m pytest tests -q), also wired to GitHub Actions (.github/workflows/test.yml)
• bin/llama_delegate.py, bin/usage_report.py — thin entry points (they only import and call llama_kit; all logic lives in the package)
• config.example.json — template; copy to config.json and edit
• integrations/qoder/, integrations/claude/, integrations/copilot/ — optional wiring for Qoder CLI, Claude Code, and GitHub Copilot (see below)
• hooks/read_file_guard.py — optional Qoder hook: warns the agent when it Reads a file >50 KB and suggests the local summary path (see hooks/README.md)
• usage.jsonl, server.log, summary_cache/, response_cache/, config.json — runtime / machine-private data (gitignored)

Qoder CLI integration (optional)

The toolkit works standalone, but it can wire itself into Qoder CLI so your agent offloads routine work automatically. From a fresh clone, one command does the whole install:

git clone https://github.com/flyingriverhorse/llama-kit.git
cd llama-kit
powershell -ExecutionPolicy Bypass -File integrations\qoder\install.ps1   # Windows

git clone https://github.com/flyingriverhorse/llama-kit.git
cd llama-kit
sh integrations/qoder/install.sh                                          # Linux/macOS

The installer (1) copies the repo to ~/.qoder/skills/local/ so the /local skill is live, (2) installs the local-gpu-offloader agent into ~/.qoder/agents/ with all <HOME> placeholders replaced, and (3) creates config.json from the template. Afterwards: edit config.json (your llama.cpp folder + model), restart Qoder CLI, done.

Step-by-step details, optional extras (delegation snippet for your own skills, per-project AGENTS.local.md, the large-file guard hook), and uninstall: integrations/qoder/README.md.

Your own machine-private files (real config.json, your projects' AGENTS.local.md) must never be committed — they contain personal paths.

Integrations for other agents

The toolkit is agent-agnostic; the same delegate CLI works from any coding agent that can run shell commands:

• Claude Code — integrations/claude/: one-command installer (agent + skill into ~/.claude/) plus a CLAUDE.md delegation snippet. See integrations/claude/README.md.
• GitHub Copilot — integrations/copilot/: a custom-agent template for .github/agents/ and a delegation snippet for Copilot instructions. See integrations/copilot/README.md.

Tips

• Start the server with --cache-prompt: repeated prefixes (session history, shared system prompts) are then reused from the KV cache — measured ~6x faster second requests on a 20k-token prefix.
• Thinking/reasoning is OFF by default (enable_thinking: false in the chat template); add --think for complex generation.
• Single slot (-np 1) is enforced via a lock file: parallel callers queue. A lock left behind by a dead process (Ctrl-C, crash, timeout) is detected via its PID and removed automatically, so callers never queue behind a corpse.
• Batch small related tasks into one call ("1) docstring for f, 2) rename x, 3) ...") — one round-trip and the shared prefix stays hot in the cache.
• Explore locally first: before an agent reads 3+ files or any file >50 KB into cloud context, --summarize them (or use --file/--focus).

Options reference

Full option list for the commands shown in Usage (llama_delegate.py [prompt] [options]; reads stdin when no prompt):

Option	Default	Purpose
--url URL	config / http://localhost:8080	llama-server base URL
--system TEXT	terse assistant prompt	system prompt
--max-tokens N	2048	generation budget
--ctx-limit N	80000	server context window (match -c)
--max-prompt-tokens N	env or 70000	hard prompt cap
--temperature T	0.6	sampling temperature
--timeout S	600	request timeout
--think	off	enable reasoning/thinking
--health-only	—	health check and exit
--no-autostart	—	don't launch the server
--summarize F...	—	cached local summaries
--file PATH (repeatable)	—	load file into prompt locally
--focus SYM (repeatable)	—	with --file: only that symbol
--grep PATTERN	—	with --file: only matching lines + context (no model call)
--grep-context N	3	context lines around each --grep match
--patch FILE	—	SEARCH/REPLACE edit mode
--dry-run	—	preview --patch result
--test CMD	—	with --patch: verify + self-fix until CMD passes
--test-retries N	3	max self-fix rounds for --test
--summarize-stdin	—	text mode: dense summary of stdin text/log
--triage	—	text mode: FAILING/ERROR/LIKELY_CAUSE/NEXT from a log
--review	—	text mode: SUMMARY/RISKS/FOCUS review of a diff
--commit-msg	—	text mode: Conventional Commits message from a diff
--ask Q	—	text mode: question about stdin/--file contents
--no-cache	—	bypass the response cache
--session PATH	—	multi-turn history file
--json-schema FILE|JSON	—	schema-constrained output
--chunk-tokens N	12000	summarize chunk size
--report [--savings]	—	usage report (+$ estimate)
--cloud-model NAME	—	price --savings at a real cloud model
--input-price / --output-price	3.0 / 15.0	cloud $/1M for --savings (override)
--list-prices	—	print the known cloud-model price table

Tuning the token budget

Every request is checked against a safe prompt budget before it is sent:

budget = min(ctx_limit − max_tokens, max_prompt_tokens)

• --ctx-limit is the server's context window — it must match the -c value in your start_command (default: both 80000).
• --max-tokens is the room reserved for the model's output; it is subtracted from the context automatically.
• --max-prompt-tokens (or $LLAMA_KIT_MAX_PROMPT_TOKENS) is an extra ceiling on top; the smaller of the two wins.

So raising the output allowance or running a smaller context window shrinks the allowed prompt automatically — nothing to compute by hand. Adjust to your setup:

Your setup	What to pass	Effective prompt budget
Defaults (-c 80000, output 2048)	nothing	70,000
Long outputs (e.g. big files)	--max-tokens 8000	min(72000, cap) = 70,000
Small context server (-c 16000)	--ctx-limit 16000	13,952
Keep prompts extra-cheap	LLAMA_KIT_MAX_PROMPT_TOKENS=40000	40,000

Exact token counts come from the server's /tokenize endpoint; a prompt over the budget is rejected up front (PROMPT_TOO_LONG, exit 3). If you hit that, drop context (--focus, --summarize) rather than raising the caps — or raise --ctx-limit and restart the server with a matching -c.

Troubleshooting

• LOCAL_MODEL_UNAVAILABLE (exit 4) — autostart failed. Check server.log; verify workdir + start_command in config.json; try the command manually. Use --no-autostart if you manage the server yourself.
• PROMPT_TOO_LONG (exit 3) — prompt over the safe budget (see Tuning the token budget). Drop context (--focus, --summarize), or raise --max-prompt-tokens / --ctx-limit (must match the server's -c).
• Port already in use — another llama-server holds 8080. Point --url / config.json at it, or start yours on another port.
• Windows: python not found — use the py -3 launcher (installed with the official Python installer).
• Slow first request after server start — normal: model load + KV warmup. Repeat requests with the same prefix are much faster with --cache-prompt.
• PATCH_FAILED (exit 1) — a SEARCH block didn't match; nothing was written. Retry with clearer instructions, or add --think.

MIT licensed.