A categorized, multi-difficulty capability eval for local LLMs — built to answer one question per request type: can a local model handle this, or should it escalate to a frontier model? The graded results aggregate into an LLM routing table.
Difficulty is unknowable up front, but it's measurable in aggregate. By scoring a local
model across categories and difficulty tiers, we learn its capability frontier — the
(category, difficulty) cells it passes reliably — and route accordingly: serve locally
where it's strong, escalate where it isn't. Every locally-served request is frontier
spend avoided.
| group | categories |
|---|---|
| Coding & agentic | code_generation, bug_fixing, code_comprehension, agentic_tool_use |
| Reasoning & ML/quant | reasoning_logic, instruction_following, structured_output, ml_quant_reasoning |
| Security & infra | prompt_injection_resistance, devops_shell, web_ts_mcp, context_faithfulness |
Each category has easy / medium / hard tiers (see SCHEMA.md); hard is the routing boundary. Categories are chosen to mirror real workloads (agentic coding, structured/MCP output, ML-eval reasoning, prompt-injection robustness, shell/devops).
tests/<category>.jsonl # the graded suite (one case per line)
harness/run_eval.py # runs a suite against an OpenAI-compatible endpoint, grades, emits results
harness/graders.py # deterministic + judge graders
results/<model>__<date>.jsonl # per-run results
results/routing_table.json # derived per-cell route decisions
router/router.py # live router: classify a request -> local model or frontier (see router/README.md)
maintain/ # plan + script for keeping the suite fresh (see below)
python harness/run_eval.py --endpoint http://127.0.0.1:11435/v1 --model ornith-9b
python harness/route_table.py results/ornith-9b__<date>.jsonl --threshold 0.75The harness is endpoint- and model-agnostic, so the same suite scores any local model (Ornith, Qwen-Coder, etc.) — that's how the routing table learns which local model to use for which cell.
The suite is meant to be living. Planned: a local reviewer (a strong local model, or a small script + judge) that periodically (a) flags stale, ambiguous, or duplicate cases, (b) re-grades and surfaces cells where the model's behavior drifted, and (c) proposes new cases for under-covered cells — all as a diff for human review, never auto-committed. See maintain/PLAN.md.
Cases are domain-flavored but scrubbed — every product name, host, and IP is fictional
(e.g. Clipwright, clipwright.example, RFC-5737 doc IPs); no secrets, real hostnames,
tokens, or proprietary internals. Run the secret scan before publishing.