fix(tastytrade): make the eval benchmark buildable again, and cover the skill

.github/workflows/tastytrade.yml

@@ -43,3 +43,102 @@ jobs:
 
       - name: Validate eval task verifiers
         run: make validate-tasks
+
+  evals:
+    # THE MERGE GATE on the tool surface and the earnings-calendars skill: drives
+    # every task with the real claude-code agent against the mock API, and fails
+    # unless the rewards clear the threshold.
+    #
+    # `check` above runs make validate-tasks, which is a different question: it
+    # proves each verifier accepts its oracle and rejects an empty answer, with no
+    # model involved. That catches a broken verifier. Only this catches a server or
+    # skill that a real agent cannot actually drive.
+    #
+    # Unlike every other step here this one spends model tokens, so it needs `check`
+    # to pass first and is guarded to same-repo PRs (a fork PR gets no secrets).
+    #
+    # It spends them against a Claude subscription rather than API credits.
+    # CLAUDE_FORCE_OAUTH makes harbor's claude-code agent blank ANTHROPIC_API_KEY
+    # before building the container env, so only CLAUDE_CODE_OAUTH_TOKEN survives
+    # and the CLI falls back to subscription auth. Never add ANTHROPIC_API_KEY back:
+    # with both present and the flag unset the CLI silently prefers the key and the
+    # run is on credits again, with only a debug log to say so.
+    #
+    # Serialized on purpose. Subscription rate limits are per account and shared
+    # with interactive use. Note this group does not serialize against harbor-hub's
+    # gate, which draws on the same account; if the two start colliding, give both
+    # workflows one shared group name.
+    runs-on: ubuntu-latest
+    needs: [check]
+    if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }}
+    concurrency:
+      group: tastytrade-evals
+      cancel-in-progress: false
+    env:
+      CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+      CLAUDE_FORCE_OAUTH: "1"
+      # For --upload on pushes to main. Results land on the hub as
+      # `ci-evals-tastytrade`, following the repo-wide `ci-evals-<plugin>`
+      # convention so every plugin's CI history is searchable together.
+      HARBOR_API_KEY: ${{ secrets.HARBOR_API_KEY }}
+      # One attempt per task. 13 tasks is already the bulk of a gate run, and a
+      # task that only passes sometimes is a signal to fix the task, not to
+      # average it away.
+      EVAL_ATTEMPTS: "1"
+      EVAL_MIN_MEAN: "1.0"
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.13"
+
+      # A setup-token credential is long-lived but not permanent. Without this
+      # probe an expired one surfaces as "the gate did not clear 1.0", which reads
+      # as a server regression and isn't. Fails rather than skips on a missing
+      # token: a secret-dependent job silently going green is the worse failure.
+      - name: Check the subscription token
+        run: |
+          set -euo pipefail
+          if [ -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]; then
+            echo "::error::CLAUDE_CODE_OAUTH_TOKEN is not set. Mint one with 'claude setup-token' and add it as a repository secret."
+            exit 1
+          fi
+          status="$(curl -sS -o /tmp/auth-probe.json -w '%{http_code}' \
+            https://api.anthropic.com/v1/messages \
+            -H "Authorization: Bearer $CLAUDE_CODE_OAUTH_TOKEN" \
+            -H "anthropic-version: 2023-06-01" \
+            -H "anthropic-beta: oauth-2025-04-20" \
+            -H "content-type: application/json" \
+            -d '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"hi"}]}')"
+          if [ "$status" = "401" ]; then
+            echo "::error::CLAUDE_CODE_OAUTH_TOKEN is expired or revoked. Mint a new one with 'claude setup-token' and update the repository secret."
+            exit 1
+          fi
+          if [ "$status" != "200" ]; then
+            echo "::error::Auth probe returned HTTP $status, expected 200. The gate would likely fail after spending a rate-limit window; response body:"
+            cat /tmp/auth-probe.json
+            exit 1
+          fi
+          echo "Auth probe returned HTTP 200; proceeding."
+
+      - name: Run the eval gate (claude-code)
+        env:
+          EVALS_OUT_DIR: ${{ runner.temp }}/eval-trials
+          # Every gate run uploads, PR included. Gating this to pushes on main
+          # meant a PR gate left nothing on the hub, so the only record of a run
+          # was a CI artifact that expires in 7 days. One job per run is cheap;
+          # a result you cannot find later is not.
+          EVALS_UPLOAD: "1"
+        run: make evals
+
+      # always(), not failure(): harbor draws progress as a live TUI, so a run
+      # that passes slowly logs almost nothing. The trials are the only record a
+      # PR run produces, and a failure is only diagnosable from the trajectory.
+      - name: Upload trials for diagnosis
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: tastytrade-eval-trials
+          path: ${{ runner.temp }}/eval-trials
+          retention-days: 7

plugins/harbor-hub/evals/run_evals.sh

@@ -32,6 +32,9 @@ set -euo pipefail
 
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 HARBOR_TEST_ENV="${HARBOR_TEST_ENV:-docker}"
+# Repo-wide convention: ci-evals-<plugin>, so every plugin's CI runs are
+# searchable together on the hub instead of hiding behind a generic name.
+JOB_NAME="ci-evals-harbor-hub"
 EVAL_TASK_REF="${EVAL_TASK_REF:-hello-world/hello-world@1}"
 # Git ref the eval images pip-install harbor-mcp from. The default branch is
 # only right for a run of main: as a PR gate the images must be built from the
@@ -115,7 +118,7 @@ run_args=(
     -a claude-code
     -e "$HARBOR_TEST_ENV"
     -o "$JOBS_DIR"
-    --job-name evals-gate
+    --job-name "$JOB_NAME"
     --ae HARBOR_API_KEY="$HARBOR_API_KEY"
     --ae EVAL_READ_JOB_ID="$READ_JOB_ID"
     --ae EVAL_DELETE_JOB_ID="$DELETE_JOB_ID"
@@ -135,9 +138,9 @@ harbor run "${run_args[@]}"
 
 # harbor run exits 0 regardless of reward; gate on a perfect result so CI
 # (and `make evals`) fails the moment any eval regresses.
-if ! python3 "$REPO_ROOT/evals/check_reward.py" "$JOBS_DIR/evals-gate/result.json" evals-gate; then
+if ! python3 "$REPO_ROOT/evals/check_reward.py" "$JOBS_DIR/$JOB_NAME/result.json" "$JOB_NAME"; then
     echo "--- verifier output ---" >&2
-    cat "$JOBS_DIR/evals-gate"/*/verifier/test-stdout.txt >&2 2>/dev/null || true
+    cat "$JOBS_DIR/$JOB_NAME"/*/verifier/test-stdout.txt >&2 2>/dev/null || true
     die "the evals did not all reach reward 1.0"
 fi
 

plugins/tastytrade/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tastytrade",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "description": "MCP server for the TastyTrade Open API: brokerage accounts, positions, market data, option chains, transactions, and order preview.",
   "author": {
     "name": "Walker Hughes"

plugins/tastytrade/.dockerignore

@@ -0,0 +1,15 @@
+# The benchmark image copies this directory, so keep out anything large, local, or
+# secret. .env especially: the image ships throwaway credentials on purpose and must
+# never pick up real ones.
+.env
+.venv
+.git
+.githooks
+evals/jobs
+__pycache__
+*.pyc
+.pytest_cache
+.ruff_cache
+.mypy_cache
+.coverage
+htmlcov

plugins/tastytrade/Makefile

@@ -1,4 +1,4 @@
-.PHONY: lint lint-fix format typecheck check selftest test test-unit test-integration coverage install-hooks validate-tasks mock-api benchmark-build benchmark benchmark-view
+.PHONY: lint lint-fix format typecheck check selftest evals test test-unit test-integration coverage install-hooks validate-tasks mock-api benchmark-build benchmark benchmark-view
 
 lint:
 	uv run ruff check .
@@ -38,19 +38,28 @@ install-hooks:
 validate-tasks:
 	uv run python evals/generate_tasks.py
 	bash evals/validate_local.sh
+	python3 evals/check_reward.py --selftest
+
+# THE MERGE GATE: drives the tasks with a real agent. Spends model tokens, needs
+# Docker and CLAUDE_CODE_OAUTH_TOKEN. validate-tasks proves the verifiers work;
+# only this proves an agent can actually drive the server and the skill.
+evals:
+	bash evals/run_gate.sh
 
 # Run the mock Tastytrade API on its own, the way the eval benchmark runs it.
 mock-api:
 	uv run uvicorn tests.fixtures.mock_api.app:app --host 0.0.0.0 --port 8080
 
 # Benchmark targets (need Docker running and ANTHROPIC_API_KEY set). These cd into evals/ so
 # Harbor resolves the task path correctly regardless of where you invoke make, and call Harbor
-# through uv so it does not need to be on PATH. The version is pinned to the one the tasks were
-# validated against. Override with e.g. HARBOR=harbor to use a different harbor.
-HARBOR ?= uv tool run --from "harbor==0.13.2" harbor
+# through uv so it does not need to be on PATH. Pinned to 0.18.0, matching harbor-hub: 0.13.2's
+# --upload wanted a `harbor auth login` credentials file and ignored HARBOR_API_KEY, so the CI
+# gate could not upload. Override with e.g. HARBOR=harbor to use a different harbor.
+HARBOR ?= uv tool run --from "harbor==0.18.0" harbor
 
+# Context is the plugin root so the image can copy the working tree it is testing.
 benchmark-build:
-	docker build -t tastytrade-bench evals/environment
+	docker build -f evals/environment/Dockerfile -t tastytrade-bench .
 
 benchmark:
 	cd evals && $(HARBOR) run -c job.yaml

plugins/tastytrade/evals/README.md

@@ -1,10 +1,13 @@
 # Evals
 
-The server is evaluated at the agent-loop level. Claude Code drives the tools over a set of
-tasks, once against the baseline server (`main`) and once against the candidate server
-(`mcp-server-refactor`), using the [Harbor](https://github.com/laude-institute/harbor)
-framework. The agent is the same in both runs and only the MCP server changes, so any
-difference in success rate, tool calls, tokens, or latency comes from the server itself.
+The server is evaluated at the agent-loop level: Claude Code drives the tools over a set of
+tasks using the [Harbor](https://github.com/laude-institute/harbor) framework, and each trial
+records reward, tool calls, tokens, and latency.
+
+This used to run two agents, a baseline ref and a candidate ref, to show the refactored server
+beat the plain endpoint-wrapper baseline. That refactor landed on main and the candidate branch
+was deleted, which left the image cloning a ref that no longer existed. There is no second side
+to compare against now, so the job runs one server and the evals serve as a regression gate.
 
 Every task runs against the mock Tastytrade API in `tests/fixtures/mock_api`, so the answers
 are fixed and reproducible and no run touches a real account or live market data. The fast
@@ -16,30 +19,39 @@ in `tests/unit/test_misuse_evals.py`.
 ```
 evals/
   environment/
-    Dockerfile            # python and uv, both server checkouts, the mock API
-    scripts/              # require-local-api, start-mock, mcp-baseline, mcp-candidate
+    Dockerfile            # python, uv, the server checkout, the skill, the mock API
+    scripts/              # require-local-api, start-mock, mcp-server
   tasks/<name>/
     task.toml             # task config
     instruction.md        # the prompt the agent sees
     tests/test.sh         # verifier, writes a reward to /logs/verifier/reward.txt
     solution/solve.sh     # oracle, writes the known-correct answer
-  job.yaml                # runs the baseline and candidate agents over every task
+  job.yaml                # runs the agent over every task
   generate_tasks.py       # regenerates the tasks from the fixtures
   validate_local.sh       # checks every verifier without Harbor or Docker
 ```
 
-## Tasks (12)
+## Tasks (13)
 
 Ten tasks ask for a single number (portfolio P/L, largest drawdown, ATM strike, IV rank,
 total fees, net cash, latest dividend, net liquidating value, position count, and the fees on
-a previewed vertical spread). One asks for the symbols in a watchlist. The last asks the agent
+a previewed vertical spread). One asks for the symbols in a watchlist. One asks the agent
 to place an order, and its verifier reads the order the mock recorded rather than a file the
 agent wrote.
 
+The last, `earnings-implied-move`, is the only one that exercises a skill rather than the MCP
+tools. It points the agent at the option chain that `earnings-calendars` ships and asks for the
+implied expected earnings move, which is 10.52% for that chain. The prompt names neither the
+skill nor the script, so it measures whether the agent recognises an earnings-vol question and
+reaches for the right tool. The image installs the skill at `/root/.claude/skills` so the normal
+trigger path is live.
+
 Nothing in the tasks is hand-typed. `generate_tasks.py` computes every expected answer from
 the mock fixtures by running the same shaping code the server uses, so a task can never
-disagree with the data the agent sees. The two anchor values are a total unrealized P/L of
-+$700 and an SPY ATM strike of 200. Regenerate after changing a fixture:
+disagree with the data the agent sees. The skill task follows the same rule: its answer comes
+from importing `scripts/calendars.py` and fitting the shipped chain. The two anchor values are
+a total unrealized P/L of +$700 and an SPY ATM strike of 200. Regenerate after changing a
+fixture:
 
 ```bash
 python evals/generate_tasks.py
@@ -53,7 +65,7 @@ repo root:
 
 ```bash
 export ANTHROPIC_API_KEY=...
-make benchmark-build    # docker build -t tastytrade-bench evals/environment
+make benchmark-build    # docker build -f evals/environment/Dockerfile -t tastytrade-bench .
 make benchmark          # cd evals && harbor run -c job.yaml
 make benchmark-view     # cd evals && harbor view jobs
 ```
@@ -62,10 +74,10 @@ Or run Harbor directly, from inside `evals/`. Call it through `uv` and pin the v
 tasks were validated against, so it doesn't depend on what's on your PATH:
 
 ```bash
+docker build -f evals/environment/Dockerfile -t tastytrade-bench .
 cd evals
-docker build -t tastytrade-bench environment
-uv tool run --from "harbor==0.13.2" harbor run -c job.yaml
-uv tool run --from "harbor==0.13.2" harbor view jobs
+uv tool run --from "harbor==0.18.0" harbor run -c job.yaml
+uv tool run --from "harbor==0.18.0" harbor view jobs
 ```
 
 Each task carries a one-line `environment/Dockerfile` (`FROM tastytrade-bench`). Harbor only
@@ -75,8 +87,37 @@ they inherit.
 
 The tasks reference the image by name (`docker_image = "tastytrade-bench"`), so build it
 before the first run. Each trial's `result.json` records the reward, the phase timings, and
-the token and cost totals, so success rate, tokens, tool calls, and latency per server come
-straight out of the job directory.
+the token and cost totals, so success rate, tokens, tool calls, and latency come straight out
+of the job directory.
+
+The image copies the working tree rather than cloning a ref, so a run measures the code you
+have checked out. Rebuild after changing the server or the skill.
+
+## The merge gate
+
+`make validate-tasks` and `make evals` answer different questions, and CI runs both.
+`validate-tasks` proves each verifier accepts its oracle and rejects an empty answer, with no
+model involved, which catches a broken verifier. `make evals` drives all 13 tasks with the
+real claude-code agent and fails unless every reward is 1.0, which is the only thing that
+catches a server or skill an agent cannot actually drive.
+
+The gate authenticates with `CLAUDE_CODE_OAUTH_TOKEN` so runs bill to a Claude subscription
+rather than API credits. It deliberately does **not** accept `ANTHROPIC_API_KEY`: with a key
+present the CLI prefers it over the token, which either moves the run onto credits silently or,
+if the key is empty, 401s every trial before spending a token. `job.yaml` used to declare the
+key for exactly this reason and no longer does.
+
+Every gate run uploads to the Harbor hub as **`ci-evals-tastytrade`**, one job per CI run
+holding all 13 tasks as trials. That follows the repo-wide `ci-evals-<plugin>` convention, so
+every plugin's CI history is searchable together instead of hiding behind a generic job name.
+
+PR runs upload too. Restricting uploads to main left a PR gate with nothing on the hub, so the
+only record was a CI artifact that expires after 7 days.
+
+```bash
+export CLAUDE_CODE_OAUTH_TOKEN=...   # claude setup-token
+make evals                           # add HARBOR_API_KEY and EVALS_UPLOAD=1 to upload
+```
 
 ## Check the verifiers without Harbor
 
@@ -86,7 +127,7 @@ reward is 0. This is the local stand-in for `harbor run -a oracle`:
 
 ```bash
 bash evals/validate_local.sh
-# 12 passed, 0 failed
+# 13 passed, 0 failed
 ```
 
 ## Safety