llm-eval-harness

A drop-in evaluation harness for AI take-home cases and interviews.

The thesis: vibecoding made building a demo cheap — anyone can ship a working prototype now. The real filter is whether you can prove it's any good. This repo is the proof: a golden-set runner, deterministic metrics, ragas-style RAG metrics, an LLM-as-judge, and a report generator. Build fast, then drop this in and show judgment.

Runs offline with zero dependencies (a stub judge approximates scores so the pipeline always produces a report). Add an API key to switch to a real LLM judge.

Quickstart

# 1. See it work end-to-end on a toy RAG system (offline, no key needed)
python3 examples/run_example.py

# 2a. Real judge via your CLAUDE SUBSCRIPTION (no API cost; needs `claude` logged in)
EVAL_USE_CLAUDE_CLI=1 python3 examples/run_example.py

# 2b. Or via an API key (pay-per-token)
cp .env.example .env        # add ANTHROPIC_API_KEY or OPENAI_API_KEY
pip install anthropic       # or: pip install openai
set -a; source .env; set +a
python3 examples/run_example.py

# 3. Point it at YOUR system in a take-home
python3 run_eval.py --golden mycases.jsonl --sut app.rag:answer

Three ways to power the judge

Mode	How	Cost
Offline stub (default)	nothing	free, approximate
Your Claude subscription	EVAL_USE_CLAUDE_CLI=1	free* (uses claude -p, your sub)
API key	ANTHROPIC_API_KEY / OPENAI_API_KEY	pay-per-token

* counts toward your normal Claude usage limits, not API billing. Slower per call (spawns the claude CLI), fine for eval-sized batches.

Your system ("SUT") is any callable with this contract:

def answer(question: str) -> dict:
    return {"answer": "...", "retrieved_contexts": ["...", "..."]}

What you get

Layer	File	What it measures
Golden set	harness/dataset.py	Your test cases (JSONL): question, reference answer, reference contexts
Runner	harness/runner.py	Executes your SUT over every case, captures latency + errors
Deterministic metrics	harness/metrics.py	Exact match, token-F1, retrieval precision/recall — free, fast, CI-able
RAG metrics	harness/rag_metrics.py	Faithfulness, answer-relevance, context precision/recall (LLM-judged)
LLM judge	harness/judge.py	Rubric scoring 1-5 (correctness, completeness, grounding, clarity) + reasoning
Report	harness/report.py	Markdown + HTML: aggregate table, worst-cases-first, full per-case table

Why each layer exists (say this in the interview)

• Deterministic first. Free, reproducible, good for CI regression gates. But brittle on phrasing — that's why you also need a judge.
• LLM-as-judge catches nuance deterministic metrics miss, but it's noisy and biased toward verbosity — so you anchor it with a rubric and keep deterministic metrics as a sanity check. Two independent signals beat one.
• Faithfulness is the hallucination metric: is every claim grounded in retrieved context? Case q5 in the example has no supporting doc on purpose — a good system should abstain, and the harness catches when it doesn't.
• Worst-cases-first report. You don't read 100 rows; you read the 5 that failed and explain why and what you'd fix. That's the senior move.

Customizing

• Rubric: pass your own to Judge(rubric={...}) — tune dimensions per domain.
• Match threshold: retrieval_precision_recall(..., thresh=0.5) — raise for stricter context matching.
• Add metrics: each scorer just returns a dict merged into record.scores; add your own (e.g., latency SLA pass/fail, cost-per-query, toxicity) the same way.

See CASE_PLAYBOOK.md for how to use this to pass a take-home with confidence.