feat(fred): Harbor eval benchmark and the merge gate

[walkerhughes]

Closes #41. On top of #40.

The design doc said a benchmark was worth adding once the tool surface had settled. It has: five tools, all merged in the #30 stack. This adds the agent-loop evals and makes them a merge gate, following plugins/tastytrade/evals/ rather than inventing a second pattern.

Note: this adds a token-spending CI job (evals), gated behind check and restricted to same-repo PRs, billing to the Claude subscription exactly as tastytrade's does.

Why the 291 unit tests were not enough

They prove the shaping is right and the correction table is right. They cannot prove an agent can drive the server, which is a different question with its own failure modes. Tastytrade's harness was built by hitting them: an agent that produced a perfect answer without ever calling a tool, four trials that loaded a schema and then never called it, two that delegated to a subagent whose calls trajectory.json does not record at all.

Two rewards per task

outcome is the answer. process is whether it came through mcp__fred__*.

This plugin has three ways round the server, not tastytrade's two:

Route	Why it is reachable	How it is caught
The local mock's port	the server needs a backend	matched on :8080, not a hostname. The mock binds every interface, so it answers on localhost, 127.0.0.1, 0.0.0.0, [::1] and the container name; naming two lets the other three through
The fixtures on disk	they hold every expected answer in plain Python	matched on the module name
The real FRED API	network_mode: public is required for the agent to reach its own model	matched on the API hostname

The third is new. Tastytrade's mock-only setup never had to consider it. It is also why require-local-api exists: the server refuses to start unless FRED_BASE_URL points at localhost, verified to exit 1 against the real API host, so a gate run cannot spend a developer's real key even if one leaked into the environment.

The task that earns its keep

rate-history-max asks for the highest value a long daily series ever reached. The fixture puts that maximum on a single day at an index the downsampler does not sample:

summary max                     300.0
highest point actually returned 199.96
tolerance                         0.01

An agent reading the returned points is wrong by a hundred. This is the central claim of get_observations (the summary covers every observation, the points are only a sample) turned into pass or fail.

The first version of this task proved nothing. The fixture had a broad triangular peak and the sampling grid landed exactly on it, so reading the points gave the right answer too. Caught by checking rather than assuming, and test_the_extremes_are_not_in_the_returned_points now fails if it ever regresses.

Two prerequisite bugs, both real

• FRED_BASE_URL was documented in .env.example, never read by client.py, and absent from .mcp.json. Shipped that way in feat(fred): plugin scaffold, API client, guided errors, CI #36. All three now agree, and a test pins the passthrough set, because an env var the code honours but the plugin config drops does nothing once installed, which is the same class of bug.
• The mock was an httpx transport with no way to serve it. Routing is now a plain route(path, params) -> (status, body) with the transport and a stdlib HTTP server as thin callers. No new dependency, and the benchmark and the test suite cannot disagree about what FRED returns.

Fixture changes

CPIAUCSL gains a real 26-month series, because year-over-year needs an observation twelve months back and a six-point series has none, so units="yoy" would otherwise have been indistinguishable from passing no units at all. The mock now applies the units transforms for real rather than echoing the code back.

Release dates are generated relative to a clock rather than written down. A calendar whose whole job is "what came out and what is next" stops straddling today the moment a hard-coded date ages, and CI runs on whatever day it runs. Tests pin the clock; the container uses the real one. next-release's check and oracle both compute the date at run time, and the check accepts today's and yesterday's answer since a trial starting before midnight is graded after it, which the container run actually exercised.

Verification

Everything below was run, not assumed.

make validate-tasks   ->  70 passed, 0 failed
                          (10 tasks x solved/empty/3 bypasses/delegated/delegated-bypass)
make check            ->  clean
make coverage         ->  291 tests, 98%

The bench image, driven over stdio through the real mcp-server wrapper against the mock served over HTTP:

init:  {'name': 'fred', 'version': '0.6.0'}
tools: [get_observations, get_release_calendar, get_revisions, get_series, search_series]

unemployment-latest    4.3
inflation-yoy          2.90557
rate-history-max       (300.0, 199.9574)   <- summary vs best sampled point
initial-print          22900.0
revision-count         1
release-series-count   3
series-units           Billions of Chained 2017 Dollars
find-series-id         UNRATE
next-release           2026-08-15

Safety guard:

FRED_BASE_URL=https://api.stlouisfed.org/fred  ->  refuses, exit 1
FRED_BASE_URL unset                            ->  refuses, exit 1
FRED_BASE_URL=http://localhost:8080/fred       ->  allowed

The evals gate itself has not been run locally, since it spends subscription tokens; the CI job on this PR is its first real run.

Also noted, not fixed

start="5y" means five years ago, and there is no spelling for "the next 60 days", so a forward window needs an absolute end date. The calendar's default window already straddles today, which covers the common question. Recorded in the design doc's deferred work rather than expanded into this PR.