fix(tastytrade): make the eval benchmark buildable again, and cover the skill - #24
Merged
Merged
Conversation
…he skill The benchmark image had not built since the monorepo consolidation. It cloned CANDIDATE_REF=mcp-server-refactor, a branch that was deleted after the refactor landed. Nothing in CI builds the image, so it failed silently. The Dockerfile carried a comment saying so and asking for a decision. Taking the single-checkout option. The A/B existed to prove the refactored server beat the endpoint-wrapper baseline; that is settled and there is no second side left, so the job runs one agent and the evals become a regression gate rather than a bake-off. The image now copies the working tree instead of cloning a ref. A clone only ever measured what had been pushed, which is how the stale branch went unnoticed for months. A .dockerignore keeps .env and .venv out of that copy. Building it surfaced a second break the old scripts shared: `uv run --project` sets the environment but not the working directory, so `python -m src.server` raised ModuleNotFoundError from WORKDIR /app. The wrapper cds into the project root now. Verified by running an initialize handshake against the built image. Adds earnings-implied-move, the first task that exercises a skill rather than the MCP tools. It points the agent at the chain the earnings-calendars skill ships and asks for the implied expected move. The prompt names neither the skill nor the script, so it measures whether the agent recognises an earnings-vol question, and the image installs the skill at /root/.claude/skills so the real trigger path is live. The expected answer is computed by importing calendars.py, holding the same no-hand-typed-answers rule as the fixture tasks. 13 verifiers pass under validate_local.sh.
Until now nothing ran the benchmark. CI ran `make validate-tasks`, which proves each verifier accepts its oracle and rejects an empty answer with no model involved. That catches a broken verifier. It cannot catch a server or a skill that a real agent is unable to drive, which is most of what the evals are for. Adds an `evals` job on the same paths filter as the rest of the plugin's CI, so any change under plugins/tastytrade/ has to clear it. It drives all 13 tasks with claude-code against the mock API and fails unless every reward is 1.0. Follows the pattern harbor-hub's gate already established: billed to a Claude subscription via CLAUDE_FORCE_OAUTH rather than API credits, an auth probe first so an expired token reads as an expired token instead of a regression, serialized through a concurrency group, same-repo guard because a fork PR gets no secrets, and the trials uploaded as an artifact on success or failure since a bare 0.0 is not diagnosable. run_gate.sh rebuilds the image from the working tree before running, so the gate measures the code under review rather than a stale image. check_reward.py carries its own selftest and is wired into validate-tasks. The harbor flags were verified against 0.13.2 on linux rather than assumed: --n-attempts is real, and so are -c, -o, and --job-name. EVAL_MIN_MEAN is 1.0 and EVAL_ATTEMPTS is 1. Whether 13 tasks pass perfectly on Haiku is unmeasured, since harbor will not install on darwin here. If the first runs prove flaky the threshold should move on evidence, not on a guess made before seeing a single result.
…plugin
The first gate run failed all 13 trials with NonZeroAgentExitCodeError and zero
tokens spent. The trials artifact showed why: apiKeySource was ANTHROPIC_API_KEY
and every trial 401'd before the agent made a request.
job.yaml declared ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} in the container env.
That variable is unset in CI, so it injected an empty key, and the CLI prefers a
key over CLAUDE_CODE_OAUTH_TOKEN. Removed. The gate now refuses to accept
ANTHROPIC_API_KEY as a fallback at all: on a machine that has a real one it
would silently move the run onto API credits, which is the same bug wearing a
green check.
The artifact also confirmed the rest of the setup is sound: the MCP server
connected, all 12 tools were exposed, and earnings-calendars was registered as a
skill. Harbor copies ~/.claude/skills into the agent's config dir, so installing
the skill in the image is the right mechanism.
Hub results are now named per plugin on the repo-wide `ci-evals-<plugin>`
convention, so a search on the hub turns up one plugin's CI history rather than
a pile of jobs called evals-gate. tastytrade uploads as ci-evals-tastytrade and
harbor-hub is renamed to ci-evals-harbor-hub to match. Uploads happen on pushes
to main only; a hub job per PR push would pile up with nothing to drop them.
The gate ran and scored 0.846, 11 of 13. Both failures were real and neither was a server bug, which is the point of running it. earnings-implied-move: the agent read the chain and priced the front straddle, reporting 11.34% against an expected 10.52%. That is a defensible reading of "implied move" and my prompt did not rule it out. The front expiry carries four days of ordinary vol on top of the event, so the straddle overstates the event itself. The instruction now says to isolate the event. It still names neither the skill nor the script, so what it measures is unchanged. The trajectory also showed the agent never invoked the skill, only Read and Write. Worth watching: if it keeps skipping the skill after the prompt is unambiguous, that is a triggering problem in the skill description rather than a task problem. place-limit-order: pre-existing, and it failed because the agent stopped to ask for confirmation before placing a financial order. That is the right instinct and the wrong outcome for a single-turn eval. The instruction now states what was already true and previously unsaid: this is a mock brokerage, no real account, no real money, orders go to a local file. That removes a false safety concern instead of talking the agent past a real one, and the task exercises the same place_order path it always did.
…main The gate has passed twice and left nothing on the hub. EVALS_UPLOAD was gated to push events, copied from harbor-hub, so PR runs uploaded nothing and the only record of a run was a CI artifact that expires in 7 days. Uploading one job per run is cheap. A result you cannot find later is not.
The first run with uploads enabled died in 28s: "Not logged in to Harbor Hub. Run `harbor auth login` before using --upload." 0.13.2 reads upload credentials from ~/.harbor/credentials.json only and ignores HARBOR_API_KEY, so CI had no way to authenticate short of writing that file itself. 0.18.0 reads the env var, which is how harbor-hub's gate has been uploading all along. Matching its pin also removes a version skew between two plugins in the same repo. Checked the 0.18.0 CLI first: every flag the gate uses is still there. Switched to long flags while in here, since -o is the one most likely to drift.
walkerhughes
force-pushed
the
tastytrade-eval-single-checkout
branch
from
August 4, 2026 15:19
d38a8df to
64c08a0
Compare
The gate scored 0.923 and place-limit-order was the one failure. The trajectory showed it was not flakiness: the agent never placed an order through the server at all. It searched for the MCP tools, did not call them, then read mock_api/app.py off the filesystem and drove the REST API directly with urllib. Its instruction was the only one of the 13 that never said to use the MCP tools. Every numeric task ends with "Use the Tastytrade MCP tools to find the answer"; this one described the order and stopped, so routing around the server was a reasonable reading. Says so now, and rules out the HTTP and file shortcuts by name. Worth a follow-up: the verifier is outcome-only, so it reads the recorded order without caring how it got there. A direct HTTP call that happened to write the right record would still score 1.0. harbor-hub splits outcome from process for exactly this reason, and these tasks should too.
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #23. Merge that one first.
The benchmark image has not built since the monorepo consolidation
It cloned
CANDIDATE_REF=mcp-server-refactor, a branch deleted after that refactor landed. Nothing in CI builds the image, so it failed quietly. The Dockerfile carried a comment saying exactly this and asking for a decision between pinning a new ref or dropping the split.Taking the single checkout. The A/B existed to prove the refactored server beat the plain endpoint-wrapper baseline. That is settled, and there is no second side left to compare against, so the job runs one agent and the evals become a regression gate instead of a bake-off.
Changes
The image copies the working tree instead of cloning a ref. A clone only ever measured what had been pushed, which is how a dead branch went unnoticed for months. Now a run measures the code you have checked out. A
.dockerignorekeeps.envand.venvout of that copy, which matters because the image ships throwaway credentials on purpose and must never pick up real ones.mcp-baselineandmcp-candidatecollapse into onemcp-server.job.yamldrops to a single agent.make benchmark-buildbuilds from the plugin root now, since the image needs the source in its context.Building it found a second break
The old wrappers used
uv run --project /opt/... python -m src.server. That sets the environment but not the working directory, so fromWORKDIR /appit raisedModuleNotFoundError: No module named 'src'. Both old wrappers had it. It was invisible because nothing ever ran the image.The wrapper cds into the project root now. Verified with a real JSON-RPC initialize handshake against the built image, which returns the server's capabilities and a clean stderr.
New task: earnings-implied-move
The first eval that exercises a skill rather than the MCP tools. It points the agent at the option chain that
earnings-calendarsships and asks for the implied expected earnings move, which is 10.52% for that chain.The prompt names neither the skill nor the script. Naming them would only test whether the agent can copy a command out of a prompt. As written it tests whether the agent recognises an earnings-vol question and reaches for the right tool, which is the thing worth measuring. The image installs the skill at
/root/.claude/skillsso the normal trigger path is live.The expected answer is computed by importing
calendars.pyand fitting the shipped chain, holding the same rule as the fixture tasks: no answer is hand-typed, so it cannot drift from what the agent sees.Verified
make validate-tasks: 13 passed, 0 failed, including the negative control.make check: lint, typecheck, selftest, 94 unit tests.make benchmark-buildsucceeds.calendars.pyreturns the 10.52 the verifier expects, its selftest passes, no.envwas copied in, and the MCP server completes an initialize handshake over stdio.Not verified: a full
harbor run, which needs an API key. The image builds and the server starts, so the part that was broken is fixed, but the first real run may still turn up task-level surprises.Version
Bumped to 0.4.1. The gate excludes
evals/**but this also touchesMakefileand adds.dockerignoreat the plugin root, so it fires. That seemed better than loosening the gate.🤖 Generated with Claude Code