Agent Eval Harness

A tool-using text-to-SQL agent, and the evaluation harness that proves how good it is.

Wiring up an agent is easy. Knowing whether it actually works, which version is better, and whether you can trust your own evaluation is the hard part. This repo is built around the second problem. The agent is deliberately small and readable; the eval layer is the real artifact.

What makes it different

Most eval tools stop at "run the agent, print an accuracy number." The interesting work is everything after that, and it is where this harness lives:

• It treats evaluation as a measurement, not a vibe. Accuracy comes with a confidence interval; comparing two models uses a paired significance test. A two-point gap is reported as noise when it is noise.
• It evaluates its own evaluator. The LLM judge is calibrated against human labels (Cohen's kappa) and stress-tested for verbosity bias. An unaudited judge is just another unverified model.
• It diagnoses, not just scores. Every failure is classified into a SQL-specific category that points at the fix.
• It is a gate, not just a report. One flag (--min-accuracy) makes it exit non-zero on a regression, so it drops into CI to block a bad agent change rather than merely describing it.
• It is small enough to read in one sitting. Provider-agnostic over one interface, zero runtime dependencies, the whole thing is standard-library Python you can audit top to bottom.

Grading runs on three levels: execution accuracy (the agent's SQL is run and its result set compared to a reference query, so correct-but-different SQL still passes), an LLM-as-judge for the prose answer (calibrated and bias-audited), and regression tracking that diffs any two runs question-by-question.

---

The scorecard

$ python -m harness demo --variant flawed

# Eval scorecard — `demo-flawed`

| Metric | Value |
|---|---|
| Execution accuracy | **83.3% (95% CI 55.2-95.3%)** |
| Total cost | $0.0000 |
| Tokens (in/out) | 1,800 / 720 |
| Mean latency | 0.00s |
| Hit max steps | 0 |
| Agent errors | 0 |

## Failures
Breakdown: wrong_values ×2

| # | Question | Why | Agent SQL |
|---|---|---|---|
| q05 | What is the total revenue from completed orders? | wrong_values | `SELECT sum(p.price*oi.quantity) FROM order_items oi JOIN products p ...` |
| q09 | List revenue by product category for completed orders ... | wrong_values | `SELECT p.category, sum(p.price*oi.quantity) AS rev FROM order_items ...` |

## By category (weakest first)
| Category | Accuracy | n |
|---|---|---|
| group | 0.0% (95% CI 0.0-79.3%) | 1 |
| join | 50.0% (95% CI 15.0-85.0%) | 4 |
| sort | 75.0% (95% CI 30.1-95.4%) | 4 |
| aggregate | 83.3% (95% CI 43.6-97.0%) | 6 |
| filter | 100.0% (95% CI 56.6-100.0%) | 5 |

Both failures are wrong_values: the SQL ran and returned the right shape, just wrong numbers (the flawed agent dropped a WHERE status='completed' filter). That category points straight at a missing predicate rather than a schema or syntax problem, which is the whole point of diagnosing failures instead of just counting them.

The per-category slice shows where the agent is weak (here, group and join queries) and leads with the weakest. Each slice carries its own Wilson interval, so a 0% built on a single question reads as the near-useless signal it is, not a damning headline.

The demo command runs the whole pipeline with a mock agent, so it works offline and in CI with no API key. --variant oracle is a perfect agent (100%); --variant flawed makes two deliberate mistakes so failures and regressions are visible. The full output of a flawed run is committed at examples/scorecard-flawed.md.

Stored runs become a leaderboard:

$ python -m harness runs
run_id                   accuracy (95% CI)                 cost  n
demo-oracle              100.0% (95% CI 75.7-100.0%)  $  0.0000  12
demo-flawed              83.3% (95% CI 55.2-95.3%)    $  0.0000  12

---

A real run: a five-model Gemini leaderboard

Five Gemini models put through the identical 12 questions as the agent, all graded by the same fixed judge (gemini-2.5-flash) so the columns are comparable. Execution accuracy grades the query the agent commits as its answer; judge accuracy grades the prose answer. These are single runs on 12 questions, so the confidence intervals (in the per-model scorecards) are wide and small gaps are within noise — read the ranks, not the decimals.

Model	Execution acc	Judge acc	Cost	Latency
gemini-2.5-pro	100.0%	100.0%	$0.0541	15.9s
gemini-3.5-flash	91.7%	100.0%	$0.0017	6.5s
gemini-3-flash-preview	91.7%	91.7%	$0.0016	8.1s
gemini-2.5-flash	83.3%	91.7%	$0.0112	4.4s
gemini-2.5-flash-lite	50.0%	16.7%	$0.0016	1.7s

Two things worth pulling out:

1. The metric caught a bug in its own harness — and that is the best demonstration of why it exists. An earlier version graded the agent's last run_sql query. That quietly punished exploratory agents: gemini-3.5-flash ran SELECT * to look at the data and computed answers in prose, so it scored 25% execution / 100% judge — a 75-point gap that made a perfectly correct model look broken. The divergence between the two metrics is what surfaced it. The fix was to change the agent's contract: it now commits a final_answer with the single query it stands behind, which is what gets graded. After the fix the same model scores 91.7%. (Spider/BIRD grade the query for the same reason.)

2. After the fix, the ranking is sane and the cheap models are remarkable. Pro tops the board, but gemini-3.5-flash and 3-flash-preview match or beat 2.5-flash at ~6x lower cost than 2.5-flash and ~30x lower than pro. flash-lite is the genuine weak link: 50% execution, and a 16.7% judge score because it frequently fills the query but leaves the prose answer blank — a failure the dual metrics pin down precisely. Whether pro's edge over the flash tier is real or noise is a question for a bigger question set; on 12 it is not significant.

---

Quickstart

No install needed for the offline demo:

git clone https://github.com/alvinekelund/agent-eval-harness
cd agent-eval-harness

python -m harness demo --variant oracle    # perfect agent  -> 100%
python -m harness demo --variant flawed     # flawed agent   -> 83.3%, failures listed
python -m harness runs                       # leaderboard: accuracy (+CI) and cost per run
python -m harness report demo-flawed         # re-render any stored run's full scorecard
python -m harness diff demo-oracle demo-flawed

## Regression diff — `demo-oracle` → `demo-flawed`
**0 fixed · 2 regressed**

McNemar exact test on 2 discordant pairs: **p = 0.500** — the net change is not significant (could be noise).

### ⚠️ Regressions (were correct, now wrong)
- `q05` — What is the total revenue from completed orders?
- `q09` — List revenue by product category for completed orders, highest first.

That p-value is the point: two regressions out of twelve is not yet evidence of a real change. The harness says so instead of letting you over-read a noisy delta.

To run a real model, set the relevant API key and point eval at a provider:

export ANTHROPIC_API_KEY=...
python -m harness eval --provider anthropic:claude-sonnet-4-6 \
                       --judge    anthropic:claude-haiku-4-5

The same harness runs across providers unchanged, so you can put two models through identical questions and compare them honestly:

python -m harness eval --provider anthropic:claude-sonnet-4-6
python -m harness eval --provider openai:gpt-4o-mini
python -m harness eval --provider gemini:gemini-2.5-flash
python -m harness diff "anthropic:..." "openai:..."

---

Running on a real benchmark (Spider / BIRD)

The 12 built-in questions are for fast iteration. For a credible number, point the same harness at a public text-to-SQL benchmark. Spider and BIRD share one format (a <split>.json of questions + gold SQL, and a database/<db_id>/<db_id>.sqlite per schema), and the loader maps it straight onto the harness:

# download + extract Spider from https://yale-lily.github.io/spider, then:
python -m harness eval --provider anthropic:claude-sonnet-4-6 \
                       --spider-dir ./spider --split dev --limit 200

Each question runs against its own schema: the loader hands the runner a db_factory that backs each database into a private in-memory copy, so the agent can never mutate the benchmark and questions stay isolated from each other. Grading is execution accuracy (Spider ships no gold natural-language answer, so the judge is skipped automatically). The datasets are large and not redistributed here; the loader is tested against a synthetic fixture so CI needs no download.

---

Use it as a gate, not just a report

--min-accuracy makes the harness exit non-zero when execution accuracy drops below a bar, and --report-file writes the scorecard for CI to upload or post. Together they turn the harness into a check that blocks a bad agent change instead of letting it merge:

# .github/workflows/agent-eval.yml
- run: |
    python -m harness eval \
      --provider anthropic:claude-sonnet-4-6 \
      --report-file scorecard.md \
      --min-accuracy 0.85          # fails the build if the agent regresses
- uses: actions/upload-artifact@v4
  with: { name: scorecard, path: scorecard.md }

$ python -m harness demo --variant flawed --min-accuracy 0.9
...
FAIL: execution accuracy 83.3% is below the required 90.0%.   # exit code 1

---

How the agent works

A hand-rolled ReAct-style loop (no agent framework, so you can read exactly what is sent and returned). Given a question it:

1. calls describe_schema to see the tables,
2. explores with run_sql (read-only; a bad query returns an error string it can read and retry),
3. commits a final_answer: the single query it stands behind (graded by execution accuracy) plus its plain-language answer (graded by the judge).

That final_answer step matters: grading the committed query rather than whatever the agent ran last is what makes execution accuracy fair for an agent that explores before answering.

run_sql is guarded twice: the statement is validated to be a single SELECT/WITH, and the SQLite connection is flipped to query_only for the call, so the agent can never mutate the database it is graded on.

The agent runs against a small, self-contained SQLite store (customers, products, orders, order_items). Because the seed data is fixed, every execution-accuracy check compares result sets over the exact same rows.

---

Can you trust the judge?

Using an LLM to grade answers is convenient and completely worthless if the judge is unreliable. So the harness measures its judge two ways before believing it.

Calibration against humans. calibrate() runs the judge over answers a human has already labeled and reports agreement and Cohen's kappa. Below substantial agreement (kappa > 0.6), judge-based accuracy is not trustworthy and the harness says so.

Verbosity-bias audit. audit_verbosity() re-judges each answer after padding it with confident, fact-free filler. The verdict should not move; the rate at which it does is a bias score, and the dangerous direction (a wrong answer marked correct once dressed up) is reported separately.

## Judge robustness audit (verbosity bias)

Re-judged 20 answers after meaning-preserving padding.

| Metric | Value |
|---|---|
| Verdict flip rate | 15.0% |
| Leniency flips (wrong → correct) | 3 |
| Verdict | BIASED — inflating scores |

Calibration on plain answers can look perfect while this is happening, which is exactly why both checks exist. Together they turn "I used an LLM judge" into "I measured my LLM judge."

---

Why these choices

• Execution accuracy over string match. There are many correct SQL queries for one question. Comparing result sets is the honest signal, and it is the standard used by text-to-SQL benchmarks (Spider, BIRD).
• Diagnosed failures, not just counted ones. Every miss is classified from its result set into one of a few SQL-specific categories (no_sql, execution_error, empty_result, wrong_cardinality, wrong_values). Each maps to a different fix, so the scorecard tells you why the agent is wrong, not only how often.
• Sliced by query type, weakest first. Accuracy is broken down by the tags on each question (join, aggregate, subquery, ...), each with its own confidence interval, so you see where the agent actually struggles instead of an average that hides it.
• A measured judge, not a trusted one. An LLM judge that has not been checked is just another unverified model. harness/eval/calibration.py scores it against human labels (Cohen's kappa) and harness/eval/judge_audit.py stress-tests it for verbosity bias. Treat judge-based accuracy as meaningful only when both pass.
• Per-item regression diff. Overall accuracy can stay flat while a prompt change quietly breaks three questions and fixes three others. The diff surfaces exactly what moved.
• Intervals and significance, not point estimates. Accuracy is reported with a Wilson 95% confidence interval, and two runs are compared with an exact McNemar test, because the models answer the same questions (a paired design). A two-point gap on 200 examples is usually noise, and the harness says so rather than letting you over-read it. Both are derived in harness/eval/stats.py from standard-library math, so nothing is a black box.
• Provider-agnostic, stdlib-only core. One neutral message/tool format; thin adapters translate to each vendor over urllib. Adding a provider is one function. Tests need no network and no key.
• Resilient to transient API errors. Every request goes through one retry-with-backoff helper, so a stray 503 or rate-limit is retried instead of silently scoring a question as wrong. This is not optional polish: an early run was corrupted by exactly one such error, which is what motivated it.

---

Tests

python -m venv .venv && .venv/bin/pip install -r requirements-dev.txt
.venv/bin/pytest -q

69 tests, all offline via the mock provider: the read-only guard, the agent loop (including the final_answer contract, tool-error retry, max-steps, and provider failure), execution-accuracy edge cases and the failure taxonomy, per-tag slicing, judge parsing, Cohen's kappa, the verbosity-bias audit, the Wilson interval and exact McNemar test, the HTTP retry/backoff logic, the Gemini 3 thought-signature round-trip, the multi-database benchmark loader, the CLI commands end-to-end including the accuracy gate, the runner, store round-trip, the regression diff, and the Anthropic/OpenAI/Gemini translation layers.

---

Layout

Path	What it is
harness/llm/	Provider-neutral types + Anthropic / OpenAI / Gemini / mock adapters + registry + retrying HTTP
harness/db/	Seeded SQLite sandbox and read-only SQL execution
harness/agent/	ReAct loop and the SQL tools it calls
harness/eval/	Dataset, execution-accuracy metric + failure taxonomy, LLM judge, judge calibration + bias audit, statistics (Wilson CI + McNemar), run store, regression diff, scorecard
harness/eval/benchmarks/	Spider / BIRD loader (multi-database, isolated in-memory copies)
harness/cli.py	demo, eval, runs (leaderboard), report, diff
data/golden.jsonl	12 hand-built questions, each with a reference SQL query
examples/	A committed sample scorecard, so the full output is visible without running
tests/	Offline pytest suite

---

Roadmap

• pass@k and seed-variance reporting for non-deterministic runs
• Per-tag accuracy slices in the scorecard (join / aggregate / subquery)
• Post the scorecard as a PR comment in CI

---

References

• Yu et al. 2018 — Spider: A Large-Scale Human-Labeled Dataset for Text-to-SQL
• Li et al. 2023 — BIRD: Can LLM Already Serve as a Database Interface?
• Yao et al. 2022 — ReAct: Synergizing Reasoning and Acting in Language Models
• Zheng et al. 2023 — Judging LLM-as-a-Judge (MT-Bench)