fred: Harbor eval benchmark and the merge gate

[walkerhughes]

The design doc currently says a Harbor benchmark is "worth adding once the tool surface has settled". It has settled: five tools, all merged in the #30 stack. This adds the agent-loop evals and makes them a merge gate, following plugins/tastytrade/evals/ rather than inventing a second pattern.

Why the unit tests are not enough

The existing 284 tests prove the shaping is right and the correction table is right. They cannot prove an agent can drive the server, and that is a different question with its own failure modes. Tastytrade's harness was built by hitting them:

• An agent produced a perfect answer without ever calling a tool: it read the mock's source off disk and drove the backend directly.
• Four trials opened with ToolSearch, loaded a schema, and then never called the tool, one of them trying to invoke it as a shell command until it timed out.
• Two delegated the call to a subagent, which harbor's trajectory.json does not record at all.

None of that is visible to a unit test, and all of it would be invisible to an eval that scored only the answer.

Two rewards per task

Same split as tastytrade, computed by rewardkit from the subdirectories of tests/:

• outcome the answer is right.
• process it came through mcp__fred__*, and nothing went round the server.

process reads both trajectory.json and the raw session transcripts under /logs/agent/sessions, so a delegated call counts and a delegated bypass is still caught. Both checks fail closed: no trajectory means no evidence, so process is 0.

Bypass vectors specific to this plugin

Tastytrade's mock listens on a port; this one has three ways round:

1. The local mock's port. Matched on the port, not on a hostname list. The mock binds every interface, so a two-hostname enumeration lets the rest through.
2. The fixture module on disk. tests/fixtures/fred_api.py holds every expected answer in plain Python. An agent that reads it can answer everything without a single tool call.
3. The real FRED API. network_mode: public is required for the agent to reach the Anthropic API, so api.stlouisfed.org is reachable from inside the container. This one is new relative to tastytrade and gets both a check and a hard stop.

Prerequisites, both real bugs

• FRED_BASE_URL is documented in .env.example and never read by client.py. Shipped that way in feat(fred): plugin scaffold, API client, guided errors, CI #36. The eval container has to point the server at the local mock, so this gets fixed rather than worked around.
• The mock is an httpx transport, not a server. The eval container needs it over HTTP. The routing gets extracted into a plain route(path, params) -> (status, body) function with the httpx transport and a small stdlib HTTP server as two thin callers, so unit tests, integration tests, and the benchmark all score against exactly the same fixture behaviour.

A require-local-api guard refuses to start the server unless FRED_BASE_URL points at localhost. That is the hard stop that keeps a benchmark run from ever spending the developer's real API key.

Tasks

Ten, covering all five tools. Every expected answer is computed by generate_tasks.py from the fixtures using the server's own shaping code, so a task can never disagree with the data the agent sees.

Task	Tool exercised	Why it earns a slot
unemployment-latest	get_observations	the base case
series-units	get_series	units are a string, not a number
find-series-id	search_series	popularity ordering; the wrong order returns a regional variant
inflation-yoy	get_observations	the units="yoy" correction, against a series long enough for it
rate-history-max	get_observations	the important one, below
gdp-peak-unemployment	get_observations	two series on one date index
initial-print	get_revisions	the real-time window, which is the whole reason the tool exists
revision-count	get_revisions	initial against current across a range
next-release	get_release_calendar	the released/upcoming split
release-series-count	search_series	release_id plus a frequency tag

rate-history-max is the one worth calling out. It asks for the maximum of a long daily series whose peak sits in the interior, so it is dropped by downsampling. An agent that reads the returned points gets the wrong answer; one that reads the summary gets the right one. That is the central claim of get_observations turned into a pass/fail.

Deliverables

evals/
  environment/Dockerfile, scripts/{require-local-api,start-mock,mcp-server}
  tasks/<10>/{task.toml,instruction.md,tests/,solution/,environment/}
  job.yaml, generate_tasks.py, check_reward.py, explain_trials.py
  validate_local.sh, validate_in_container.sh, README.md

Plus make evals / make validate-tasks / make benchmark-*, an evals job in .github/workflows/fred.yml gated behind check, and the design doc updated to describe the benchmark rather than defer it.

Two gates, two questions

• make validate-tasks scores every verifier against synthetic trajectories and asserts the whole reward matrix (solved, empty, bypassed, bypassed-alt, delegated, delegated-bypass). No model, no key, no network, so it runs on every PR. It catches a broken verifier.
• make evals drives all ten tasks with the real claude-code agent and fails unless every reward is 1.0. Only this catches a server an agent cannot drive.

Hub results upload as ci-evals-fred, following the repo-wide ci-evals-<plugin> convention.

Done when

Both gates pass, the reward matrix asserts that each of the three bypass routes scores outcome=1, process=0, and a real agent clears 1.0 on all ten tasks.