feat(fred): Harbor eval benchmark and the merge gate

.github/workflows/fred.yml

@@ -42,3 +42,100 @@ jobs:
       # in-process over an injected httpx transport.
       - name: Coverage (unit + integration)
         run: make coverage
+
+      # Scores every task's real verifier against its oracle, an empty run, three
+      # bypass routes, and a delegated call. No model involved.
+      - name: Validate eval task verifiers
+        run: make validate-tasks
+
+  evals:
+    # THE MERGE GATE on the tool surface: drives every task with the real claude-code
+    # agent against the mock FRED API, and fails unless the rewards clear the threshold.
+    #
+    # `check` above runs make validate-tasks, which is a different question: it proves
+    # each verifier accepts its oracle and rejects a bypass, with no model involved.
+    # That catches a broken verifier. Only this catches a server a real agent cannot
+    # actually drive.
+    #
+    # Unlike every other step here this one spends model tokens, so it needs `check` to
+    # pass first and is guarded to same-repo PRs (a fork PR gets no secrets).
+    #
+    # It spends them against a Claude subscription rather than API credits.
+    # CLAUDE_FORCE_OAUTH makes harbor's claude-code agent blank ANTHROPIC_API_KEY before
+    # building the container env, so only CLAUDE_CODE_OAUTH_TOKEN survives and the CLI
+    # falls back to subscription auth. Never add ANTHROPIC_API_KEY back: with both
+    # present and the flag unset the CLI silently prefers the key and the run is on
+    # credits again, with only a debug log to say so.
+    #
+    # Serialized on purpose. Subscription rate limits are per account and shared with
+    # interactive use.
+    runs-on: ubuntu-latest
+    needs: [check]
+    if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }}
+    concurrency:
+      group: fred-evals
+      cancel-in-progress: false
+    env:
+      CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+      CLAUDE_FORCE_OAUTH: "1"
+      # For --upload. Results land on the hub as `ci-evals-fred`, following the repo-wide
+      # `ci-evals-<plugin>` convention so every plugin's CI history is searchable together.
+      HARBOR_API_KEY: ${{ secrets.HARBOR_API_KEY }}
+      # One attempt per task. A task that only passes sometimes is a signal to fix the
+      # task, not to average it away.
+      EVAL_ATTEMPTS: "1"
+      EVAL_MIN_MEAN: "1.0"
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.13"
+
+      # A setup-token credential is long-lived but not permanent. Without this probe an
+      # expired one surfaces as "the gate did not clear 1.0", which reads as a server
+      # regression and isn't. Fails rather than skips on a missing token: a
+      # secret-dependent job silently going green is the worse failure.
+      - name: Check the subscription token
+        run: |
+          set -euo pipefail
+          if [ -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]; then
+            echo "::error::CLAUDE_CODE_OAUTH_TOKEN is not set. Mint one with 'claude setup-token' and add it as a repository secret."
+            exit 1
+          fi
+          status="$(curl -sS -o /tmp/auth-probe.json -w '%{http_code}' \
+            https://api.anthropic.com/v1/messages \
+            -H "Authorization: Bearer $CLAUDE_CODE_OAUTH_TOKEN" \
+            -H "anthropic-version: 2023-06-01" \
+            -H "anthropic-beta: oauth-2025-04-20" \
+            -H "content-type: application/json" \
+            -d '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"hi"}]}')"
+          if [ "$status" = "401" ]; then
+            echo "::error::CLAUDE_CODE_OAUTH_TOKEN is expired or revoked. Mint a new one with 'claude setup-token' and update the repository secret."
+            exit 1
+          fi
+          if [ "$status" != "200" ]; then
+            echo "::error::Auth probe returned HTTP $status, expected 200. The gate would likely fail after spending a rate-limit window; response body:"
+            cat /tmp/auth-probe.json
+            exit 1
+          fi
+          echo "Auth probe returned HTTP 200; proceeding."
+
+      - name: Run the eval gate (claude-code)
+        env:
+          EVALS_OUT_DIR: ${{ runner.temp }}/eval-trials
+          # Every gate run uploads, PR included: gating uploads to main left a PR gate
+          # with nothing on the hub, so the only record was an artifact expiring in 7 days.
+          EVALS_UPLOAD: "1"
+        run: make evals
+
+      # always(), not failure(): harbor draws progress as a live TUI, so a run that
+      # passes slowly logs almost nothing. The trials are the only record a PR run
+      # produces, and a failure is only diagnosable from the trajectory.
+      - name: Upload trials for diagnosis
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: fred-eval-trials
+          path: ${{ runner.temp }}/eval-trials
+          retention-days: 7

plugins/fred/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "fred",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "description": "MCP server for the FRED API: economic time series from the St. Louis Fed, with search, aligned multi-series observations, revision history, and the release calendar.",
   "author": {
     "name": "Walker Hughes"

plugins/fred/.gitignore

@@ -14,3 +14,6 @@ wheels/
 
 # Coverage
 .coverage
+
+# Harbor job output
+evals/jobs/

plugins/fred/.mcp.json

@@ -6,6 +6,7 @@
       "args": ["${CLAUDE_PLUGIN_ROOT}/scripts/start-server.sh"],
       "env": {
         "FRED_API_KEY": "${FRED_API_KEY:-}",
+        "FRED_BASE_URL": "${FRED_BASE_URL:-}",
         "FRED_LOG_LEVEL": "${FRED_LOG_LEVEL:-}"
       }
     }

plugins/fred/Makefile

@@ -1,4 +1,5 @@
-.PHONY: lint lint-fix format typecheck check test test-unit test-integration coverage
+.PHONY: lint lint-fix format typecheck check test test-unit test-integration coverage \
+        validate-tasks evals mock-api benchmark-build benchmark benchmark-view
 
 lint:
 	uv run ruff check .
@@ -25,3 +26,37 @@ test-integration:
 
 coverage:
 	uv run pytest --cov --cov-report=term-missing
+
+# Check every eval task verifier against its oracle and against each bypass route. No
+# model and no API key, so it runs in CI. Needs Docker: rewardkit scores these checks
+# and does not build on macOS.
+validate-tasks:
+	uv run python evals/generate_tasks.py
+	bash evals/validate_local.sh
+	python3 evals/check_reward.py --selftest
+
+# THE MERGE GATE: drives the tasks with a real agent. Spends model tokens, needs Docker
+# and CLAUDE_CODE_OAUTH_TOKEN. validate-tasks proves the verifiers work; only this
+# proves an agent can actually drive the server.
+evals:
+	bash evals/run_gate.sh
+
+# Run the mock FRED API on its own, the way the eval benchmark runs it.
+mock-api:
+	uv run python -m tests.fixtures.fred_api 8080
+
+# Benchmark targets (need Docker running and CLAUDE_CODE_OAUTH_TOKEN set). These cd into
+# evals/ so Harbor resolves the task path correctly regardless of where you invoke make,
+# and call Harbor through uv so it does not need to be on PATH. Pinned to the version
+# the tasks were validated against.
+HARBOR ?= uv tool run --from "harbor==0.18.0" harbor
+
+# Context is the plugin root so the image can copy the working tree it is testing.
+benchmark-build:
+	docker build -f evals/environment/Dockerfile -t fred-bench .
+
+benchmark:
+	cd evals && $(HARBOR) run -c job.yaml
+
+benchmark-view:
+	cd evals && $(HARBOR) view jobs

plugins/fred/README.md

@@ -92,16 +92,21 @@ What economic data just came out, and what is scheduled next, split around today
 
 ```bash
 uv sync
-make check      # lint, typecheck, unit tests
-make test       # everything
-make coverage   # with a report, 80% floor
+make check           # lint, typecheck, unit tests
+make test            # everything
+make coverage        # with a report, 80% floor
+make validate-tasks  # score every eval verifier against its oracle and each bypass (needs Docker)
+make evals           # THE MERGE GATE: drive the tasks with a real agent (needs Docker + a token)
 ```
 
 Integration tests drive the registered MCP server against a mock FRED built from trimmed real captures. No network and no API key, so the whole suite runs anywhere.
 
+The agent-loop benchmark lives in [`evals/`](evals/): 10 tasks over all five tools, each scoring both whether the answer is right and whether it came through the MCP server rather than round it. See [evals/README.md](evals/README.md).
+
 ## Not here
 
 - **Maps / GeoFRED.** A different product with a different shape.
 - **Tag and category tree browsing.** `search_series` covers the reachable ground; the tree is a UI affordance.
 - **Sources.** Metadata about metadata.
 - **A response cache.** FRED allows 120 requests a minute and the data moves slowly, so nothing is under pressure. See the design doc's deferred work.
+- **Forward date spans.** `"5y"` means five years ago; a forward window needs an absolute end date.

plugins/fred/docs/design.md

@@ -164,16 +164,36 @@ leave a model holding three IDs with no idea which to fix.
 
 ## Evals
 
-Unit-level misuse tests (`tests/unit/test_schemas.py`, `test_dates.py`) feed realistic model
+Three layers, answering three different questions.
+
+**Unit-level misuse tests** (`tests/unit/test_schemas.py`, `test_dates.py`) feed realistic model
 mistakes through correction and validation and assert the corrections and the suggestion-bearing
-errors. Integration tests drive the registered MCP server against a mock FRED built from trimmed
-real captures, so the shaping is asserted against FRED's shapes rather than invented ones. All of
-it runs in CI with no key and no network.
+errors.
+
+**Integration tests** drive the registered MCP server against a mock FRED built from trimmed real
+captures, so the shaping is asserted against FRED's shapes rather than invented ones.
+
+**The Harbor benchmark** (`evals/`) drives Claude Code over 10 tasks covering all five tools and
+fails unless every reward is 1.0. This is the only layer that can catch a server an agent cannot
+drive, which is a different failure from a server that computes the wrong thing. Every task scores
+two rewards: `outcome` (the answer is right) and `process` (it came through `mcp__fred__*`, and
+nothing went round the server).
+
+The split exists because a tastytrade gate run produced a perfect answer without calling a single
+tool: it read the mock's source off disk and drove the backend directly. This plugin has three such
+routes rather than two, since the benchmark runs with the network up and `api.stlouisfed.org` is
+therefore reachable, so `process` matches on the mock's port, the fixture module name, and the real
+API's hostname. `require-local-api` is the hard stop behind that: the server refuses to start
+unless `FRED_BASE_URL` points at localhost, so a benchmark run cannot spend a real key.
+
+One task is worth naming. `rate-history-max` asks for the highest value a long daily series ever
+reached, and the fixture puts that maximum on a single day at an index the downsampler does not
+sample: the summary says 300.0 and the best point actually returned is 199.96, against a tolerance
+of 0.01. It is the summary-before-downsampling claim above, turned into pass or fail.
 
-There is no Harbor agent-loop benchmark, unlike the tastytrade server. That one exists because
-order placement makes a wrong answer expensive; FRED is read-only. A benchmark is worth adding once
-the tool surface has settled, and it would measure the table above at the agent loop rather than at
-the payload.
+All three layers use the same fixtures through the same `route()` function, so the benchmark and
+the test suite cannot disagree about what FRED returns. The first two run in CI on every PR with no
+key and no network; `evals/README.md` covers the third.
 
 ## Deferred work
 
@@ -184,3 +204,6 @@ the payload.
 - **Maps/GeoFRED.** Regional data by shape rather than by series.
 - **Paging.** Every tool returns a bounded page with FRED's own total. No cursor, because no
   question so far has needed the second page.
+- **Forward date spans.** `start="5y"` means five years *ago*; there is no spelling for "the next
+  60 days", so a forward window is written out as an absolute date. The calendar's default window
+  already straddles today, which is what the common question needs.