feat(evals): split the reward into outcome and process

plugins/tastytrade/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tastytrade",
-  "version": "0.4.1",
+  "version": "0.4.2",
   "description": "MCP server for the TastyTrade Open API: brokerage accounts, positions, market data, option chains, transactions, and order preview.",
   "author": {
     "name": "Walker Hughes"

plugins/tastytrade/evals/README.md

@@ -24,11 +24,16 @@ evals/
   tasks/<name>/
     task.toml             # task config
     instruction.md        # the prompt the agent sees
-    tests/test.sh         # verifier, writes a reward to /logs/verifier/reward.txt
+    tests/test.sh         # verifier: `rewardkit /tests`
+    tests/outcome/check.py  # reward 1: the answer is right
+    tests/process/check.py  # reward 2: it came through the MCP server (or the skill)
     solution/solve.sh     # oracle, writes the known-correct answer
   job.yaml                # runs the agent over every task
   generate_tasks.py       # regenerates the tasks from the fixtures
-  validate_local.sh       # checks every verifier without Harbor or Docker
+  check_reward.py         # gates a harbor result.json on its rewards
+  explain_trials.py       # names each trial and dumps the tool calls behind a failure
+  validate_local.sh       # scores every verifier without Harbor or a model
+  validate_in_container.sh  # the reward matrix it asserts
 ```
 
 ## Tasks (13)
@@ -93,6 +98,67 @@ of the job directory.
 The image copies the working tree rather than cloning a ref, so a run measures the code you
 have checked out. Rebuild after changing the server or the skill.
 
+## Two rewards per task
+
+Every task scores `outcome` and `process`, both computed by
+[rewardkit](https://pypi.org/project/harbor-rewardkit/) from the subdirectories of
+`tests/`.
+
+`outcome` is the answer. `process` is whether it came through the plugin.
+
+The split is not theoretical. The mock brokerage listens on `localhost:8080` inside the
+container and its source sits in the checkout, so an agent can produce a perfect answer
+without ever calling a tool, and a real gate run did exactly that: it searched for the MCP
+tools, never called them, read `mock_api/app.py` off disk, and drove the REST API with
+`urllib`. `outcome` alone scored that 1.0. `process` is what makes these MCP evals rather
+than answer-matching.
+
+For the twelve tool tasks, `process` asks that a `mcp__tastytrade__*` tool was called and
+that nothing reached the mock brokerage directly. "Directly" is matched on the mock's port
+rather than on a list of hostnames: it binds every interface, so it answers on `localhost`,
+`127.0.0.1`, `0.0.0.0`, `[::1]`, and the container's own name, and naming two of those let
+the other three through. Nothing else in the image listens on that port.
+
+For `earnings-implied-move` `process` asks that the skill or its script was used, since an
+agent that eyeballs the straddle can land close enough to pass `outcome` without loading
+the skill, and an early run did.
+
+A delegated call counts, and seeing it takes a second source. `trajectory.json` holds the
+main session only: harbor's session scan drops any jsonl whose path contains a `subagents/`
+component, which is exactly where Claude Code writes a subagent's transcript. So a call the
+agent hands to a delegate leaves an `Agent` entry and no tool, and two trials fetched the
+right answer that way. The checks therefore read the raw session transcripts under
+`/logs/agent/sessions` as well, and stop caring who placed the call: whether the top-level
+agent called the tool or routed it through a delegate is the harness's decision, not a fact
+about this plugin, and the gate's question is whether a real agent can drive the server.
+
+That has to cut both ways. Crediting a delegated MCP call while missing a delegated `curl`
+would turn "ask a subagent" into an invisible bypass, so the bypass criterion reads the same
+union, and `validate_local.sh` asserts both directions.
+
+Every task prompt states the rule `process` scores, and each paragraph of it is there
+because a gate run failed without it. The first run under the split scored `outcome` 1.0
+on all thirteen tasks and lost `process` on six, with every prompt saying only "use the
+Tastytrade MCP tools" and putting nothing out of bounds. Saying it explicitly fixed two of
+the six. The remaining four all failed the same way, and it was not the way anyone
+expected: each opened with `ToolSearch {"query": "tastytrade"}` and then never called a
+tool. One ran `Bash: mcp__tastytrade__get_option_chain --symbol SPY ...` as a shell
+command. Two delegated. `preview-vertical-spread` spent 44 calls trying to reach the MCP
+over a socket, a subprocess, and an SDK import, having already loaded the schema, and hit
+the agent timeout -- which took `outcome` down with it, 1.0 to 0.0.
+
+So the confusion was never about which tool or which arguments. It was about how to invoke
+an MCP tool at all when its schema is deferred rather than listed, and the prompt now says:
+load it with `ToolSearch`, then call it like any other tool, and no shell command or Python
+import can substitute.
+
+The wording avoids the host, port, and module name the bypass check greps for, so that an
+agent echoing its instructions into a shell comment cannot fail the check by quoting it.
+
+Both checks fail closed. No trajectory means no evidence the intended route was taken, so
+`process` is 0. That is why the oracle scores `outcome=1, process=0`: it is a shell script,
+not an agent, and cannot call tools.
+
 ## The merge gate
 
 `make validate-tasks` and `make evals` answer different questions, and CI runs both.
@@ -119,17 +185,62 @@ export CLAUDE_CODE_OAUTH_TOKEN=...   # claude setup-token
 make evals                           # add HARBOR_API_KEY and EVALS_UPLOAD=1 to upload
 ```
 
+### When it fails
+
+The gate prints each trial by name with its rewards, and for any trial that lost `process`,
+the tool calls the agent made, MCP ones unmarked and everything else flagged `!`. That list
+*is* the `process` score, so it is usually the whole diagnosis:
+
+```
+== dividend-lookup: outcome=1.0, process=0.5
+   3 tool call(s), 1 through the MCP server:
+   ! Bash {"command": "curl -s http://0.0.0.0:8080/market-metrics"}
+     mcp__tastytrade__get_market_data {"symbols": ["AAPL"], "include": ["dividends"]}
+```
+
+It used to `cat` the verifier output instead, which printed thirteen anonymous pairs of
+numbers: you could see that six tasks lost `process` and not which six. Recovering that
+meant downloading the CI artifact, which expires after seven days.
+
 ## Check the verifiers without Harbor
 
-`validate_local.sh` runs each task's oracle (`solve.sh`), then its verifier (`test.sh`), and
-confirms the verifier awards a reward of 1. It then feeds an empty answer and confirms the
-reward is 0. This is the local stand-in for `harbor run -a oracle`:
+`validate_local.sh` scores every task's real verifier against six synthetic trajectories
+and asserts the whole reward matrix. No model, no Harbor, no API key:
+
+| case | answer | trajectory | outcome | process |
+|---|---|---|---|---|
+| solved | oracle | took the intended route | 1 | 1 |
+| empty | none | none | 0 | 0 |
+| bypassed | oracle | went round the server | 1 | 0 |
+| bypassed-alt | oracle | went round it another way | 1 | 0 |
+| delegated | oracle | subagent took the intended route | 1 | 1 |
+| delegated-bypass | oracle | subagent went round the server | 1 | 0 |
+
+The bypass rows are the point, and they are what `harbor run -a oracle` cannot tell you.
+`bypassed-alt` exists because one spelling of a bypass proves only that one spelling is
+caught: it reaches the mock on an address the first check's hostname list did not name, and
+for the skill task it does the arithmetic inline, which leaves no distinctive string at all.
+
+The `delegated` pair covers the blind spot behind it, and both halves have to be asserted
+together: crediting a delegated MCP call while missing a delegated `curl` would make "ask a
+subagent" an invisible bypass, which is worse than not looking at all.
 
 ```bash
-bash evals/validate_local.sh
-# 13 passed, 0 failed
+make validate-tasks
+# 78 passed, 0 failed
 ```
 
+It runs in the bench image rather than on the host, because rewardkit scores these checks
+and does not build on macOS (its litellm dependency wants a newer rustc than ships there).
+Using the same image CI uses also means the verifier under test is the one that will really
+grade a gate run, so **this needs Docker**.
+
+Writing it paid for itself immediately: it caught two bugs in the first draft of the process
+checks. An empty run scored 0.5 because "did not bypass the server" is vacuously true when
+there are no tool calls at all, and the skill check matched the reference chain's own file
+path, which contains the skill's name, so merely reading the input counted as using the
+skill.
+
 ## Safety
 
 The agent never sees real credentials. The image sets `API_BASE_URL` to the local mock and

plugins/tastytrade/evals/check_reward.py

@@ -5,10 +5,16 @@
 result itself. Every tastytrade task reports one reward in [0, 1], so the mean
 over a task is its pass rate across attempts.
 
-    python3 check_reward.py <result.json> <name>                # every reward 1.0
-    python3 check_reward.py <result.json> <name> --min-mean 0.9 # allow some slack
+    python3 check_reward.py <result.json> <name>                 # every reward 1.0
+    python3 check_reward.py <result.json> <name> --min-mean 0.9  # allow some slack
+    python3 check_reward.py <result.json> <name> --only outcome  # one reward only
     python3 check_reward.py --selftest
 
+Each task reports two rewards, ``outcome`` and ``process`` (see evals/README.md).
+``--only`` restricts the gate to one of them, which is how the local validator
+holds the oracle to ``outcome``: the oracle is a shell script, not an agent, so
+it cannot call MCP tools and cannot score on ``process``.
+
 ``--min-mean`` exists because this gate drives a real agent over 13 tasks, not
 3. A single flaky task should not be indistinguishable from a broken server,
 but the threshold is an explicit number in the workflow rather than a silent
@@ -20,23 +26,29 @@
 from pathlib import Path
 
 
-def rewards(stats: dict) -> tuple[list | None, str]:
-    """Every reward in the run, or (None, reason) if it did not complete cleanly."""
+def rewards(stats: dict, only: str | None = None) -> tuple[list | None, str]:
+    """Every reward in the run, or (None, reason) if it did not complete cleanly.
+
+    A task with one reward reports it under the metric name ({"mean": 1.0}); a
+    task with several reports them under the reward names ({"outcome": 1.0,
+    "process": 1.0}). `only` filters to one of those names.
+    """
     if stats.get("n_errored_trials") or not stats.get("n_completed_trials"):
         return None, f"run did not complete cleanly (stats={stats})"
     found = [
         value
         for eval_stats in stats.get("evals", {}).values()
         for metric in eval_stats.get("metrics", [])
-        for value in metric.values()
+        for name, value in metric.items()
+        if only is None or name == only
     ]
     if not found:
-        return None, "no rewards reported"
+        return None, f"no {only or ''} rewards reported".replace("  ", " ")
     return found, ""
 
 
-def gate(stats: dict, min_mean: float = 1.0) -> tuple[bool, str]:
-    found, reason = rewards(stats)
+def gate(stats: dict, min_mean: float = 1.0, only: str | None = None) -> tuple[bool, str]:
+    found, reason = rewards(stats, only)
     if found is None:
         return False, reason
     mean = sum(found) / len(found)
@@ -64,6 +76,17 @@ def _selftest() -> None:
     assert not gate(mixed)[0], "one zero must fail a perfect gate"
     assert not gate(mixed, min_mean=0.9)[0], "mean 0.5 is below 0.9"
     assert gate(mixed, min_mean=0.5)[0], "mean 0.5 meets a 0.5 threshold"
+
+    # Two named rewards per task. The oracle solves the answer but cannot call
+    # MCP tools, so it is outcome=1, process=0 and only `--only outcome` passes.
+    split = {
+        "n_completed_trials": 1,
+        "evals": {"a": {"metrics": [{"outcome": 1.0, "process": 0.0}]}},
+    }
+    assert not gate(split)[0], "a zero process reward must fail the full gate"
+    assert gate(split, only="outcome")[0]
+    assert not gate(split, only="process")[0]
+    assert rewards(split, only="nope")[0] is None, "unknown reward name -> None"
     print("check_reward selftest ok")
 
 
@@ -80,7 +103,8 @@ def main(argv: list[str]) -> int:
     except FileNotFoundError:
         print(f"{name}: no result.json at {result_path}", file=sys.stderr)
         return 1
-    ok, msg = gate(stats, min_mean)
+    only = argv[argv.index("--only") + 1] if "--only" in argv else None
+    ok, msg = gate(stats, min_mean, only)
     print(f"{name}: {msg}", file=sys.stdout if ok else sys.stderr)
     return 0 if ok else 1
 

plugins/tastytrade/evals/environment/Dockerfile

@@ -34,6 +34,13 @@ RUN cd /opt/tastytrade && (uv sync --frozen || uv sync)
 RUN mkdir -p /root/.claude/skills \
     && cp -r /opt/tastytrade/skills/earnings-calendars /root/.claude/skills/earnings-calendars
 
+# rewardkit scores the verifiers (each tests/<name>/ becomes a named reward).
+# Baked in rather than fetched at verify time so verification needs no network,
+# and in its own venv so it cannot disturb the server's resolved dependencies.
+RUN python -m venv /opt/rewardkit \
+    && /opt/rewardkit/bin/pip install --no-cache-dir "harbor-rewardkit==0.1.*" \
+    && ln -s /opt/rewardkit/bin/rewardkit /usr/local/bin/rewardkit
+
 COPY evals/environment/scripts/ /usr/local/bin/
 RUN chmod +x /usr/local/bin/require-local-api /usr/local/bin/start-mock /usr/local/bin/mcp-server
 

plugins/tastytrade/evals/explain_trials.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""Say which trial scored what, and what the agent actually did.
+
+The gate used to dump every trial's verifier output with `cat <job>/*/verifier/
+test-stdout.txt`, which prints thirteen anonymous pairs of numbers:
+
+    outcome: 1.0
+    process: 0.5
+
+That says six tasks lost `process` and nothing about which six, let alone why.
+Recovering it meant downloading the CI artifact, which needs credentials the
+person reading the log may not have and which expires after seven days.
+
+So this prints the trial name with its rewards, and for any trial that did not
+score a perfect `process`, the tool calls it made. `process` is entirely a
+function of that list -- whether an `mcp__tastytrade__*` call is in it, and
+whether anything else went round the server -- so the list is the diagnosis.
+
+    python3 explain_trials.py <job-dir>
+
+Deliberately does not re-implement the bypass regex. Duplicating it here would
+give the log a second opinion that could drift from the checks, and the raw
+calls are what a reader needs anyway.
+"""
+
+import json
+import sys
+from pathlib import Path
+
+MCP_PREFIX = "mcp__tastytrade__"
+ARG_WIDTH = 160
+
+
+def _rewards(trial: Path) -> dict[str, float]:
+    """The trial's rewards, read from whatever the verifier left behind.
+
+    rewardkit writes reward.json; its stdout carries the same numbers as
+    `name: value` lines. Harbor's layout for these has moved before, so both are
+    tried rather than pinning one path.
+    """
+    for path in sorted(trial.rglob("reward.json")):
+        try:
+            data = json.loads(path.read_text())
+        except (OSError, json.JSONDecodeError):
+            continue
+        found = {k: float(v) for k, v in data.items() if isinstance(v, int | float)}
+        if found:
+            return found
+
+    found = {}
+    for path in sorted(trial.rglob("test-stdout.txt")):
+        try:
+            text = path.read_text()
+        except OSError:
+            continue
+        for line in text.splitlines():
+            name, _, value = line.partition(":")
+            try:
+                found[name.strip()] = float(value)
+            except ValueError:
+                continue
+    return found
+
+
+def _calls(trial: Path) -> list[tuple[str, str]]:
+    """(tool name, arguments) for every call in the trial's trajectory."""
+    for path in sorted(trial.rglob("trajectory.json")):
+        try:
+            data = json.loads(path.read_text())
+        except (OSError, json.JSONDecodeError):
+            continue
+        steps = data.get("steps") or []
+        return [
+            (str(call.get("function_name") or "?"), json.dumps(call.get("arguments") or {}))
+            for step in steps
+            for call in step.get("tool_calls") or []
+        ]
+    return []
+
+
+def explain(job_dir: Path) -> None:
+    trials = sorted(p for p in job_dir.iterdir() if p.is_dir())
+    if not trials:
+        print(f"no trial directories under {job_dir}")
+        return
+
+    for trial in trials:
+        rewards = _rewards(trial)
+        if not rewards:
+            continue
+        summary = ", ".join(f"{name}={value}" for name, value in sorted(rewards.items()))
+        print(f"\n== {trial.name}: {summary}")
+
+        if rewards.get("process", 0.0) >= 1.0:
+            continue
+
+        calls = _calls(trial)
+        if not calls:
+            print("   no trajectory recorded, so `process` fails closed at 0")
+            continue
+        mcp = sum(1 for name, _ in calls if name.startswith(MCP_PREFIX))
+        print(f"   {len(calls)} tool call(s), {mcp} through the MCP server:")
+        for name, args in calls:
+            marker = "  " if name.startswith(MCP_PREFIX) else "! "
+            print(f"   {marker}{name} {args[:ARG_WIDTH]}")
+
+
+if __name__ == "__main__":
+    explain(Path(sys.argv[1]))