feat: Terminal-Bench adapter for the release canary (release evaluation PR 2)

[noodlemind]

Summary

PR 2 of the release-evaluation plan: the Terminal-Bench adapter for the pinned cobol-modernization canary, built on Harbor rather than re-implementing Terminal-Bench execution. Everything is deterministic and provider-free in CI; the Harbor CLI, sandbox, and provider are injected boundaries exercised for real at release time.

• task-lock.json — pins terminal-bench@2.0 / cobol-modernization with a verifier contract (reward files, passing reward) and a task-tree checksum slot. stampTaskLock pins the downloaded task; verifyTaskAgainstLock fails closed on an unstamped lock or any tampered/drifted task file.
• harbor-adapter.mjs — validates the lock, builds harbor run argv using only documented flags (-d <dataset@version> --task-name <task> --agent <ref> --model <m> --env <docker|daytona> -n 1), runs the CLI behind an injected spawn with timeout, discovers the newest job directory, reads trial results, and classifies failures: infrastructure (spawn error/timeout) vs provider vs verifier vs a validly graded trial (reward 0 is a fail, not a failure).
• verifier.mjs — reads Harbor's official verifier artifacts (reward.json preferred, reward.txt fallback), grades against the lock's passing reward, extracts pytest assertion counts, and fingerprints the trial artifact tree (sha256) for auditable end-state evidence. A missing reward stays null, never coerced to 0.
• generic-condition.mjs / harness-condition.mjs — the A/B arms. The generic baseline keeps the original instruction and a fair neutral prompt (encourages exploring/testing/verifying) with zero harness vocabulary — enforced by test. The harness treatment starts from the byte-identical baseline and layers the engineer contract, loaded-skill guidance, and CLI activation commands; both arms get identical, independent limit copies.
• agent.mjs — the stdio bridge: the PR 1 driver/budget/telemetry stack decides actions in Node; a line-delimited JSON protocol (exec/result/done) carries commands to the sandbox. Explicit stop reasons on every exit path: model_finish, budget_exhausted, provider_error (with kind/billed classification), max_steps, protocol_error.
• harbor_agent.py — the thin Harbor BaseAgent wrapper that spawns the Node bridge and executes each command in the Harbor environment. Defensive against exec-surface differences across Harbor releases; syntax-checked in CI, exercised live at release time.

Test coverage

30 new deterministic tests (eval-tb-adapter, eval-tb-verifier, eval-tb-conditions, eval-tb-agent): lock validation and fail-closed pinning, tamper detection, argv construction, spawn/timeout classification, reward parsing precedence and rejection, tree-hash determinism/sensitivity, condition parity and isolation invariants, and the full bridge protocol including provider-error and protocol-error paths.

• Full suite: 725/725 pass
• Deterministic evals: 17 pass / 2 skipped (unchanged baseline)

Stack

Based on #38 (PR 1: driver telemetry and budgets). PR 3 adds host adapters; PR 4 adds release orchestration.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b1cfec51-52d6-4bf7-a5aa-4533278cacdc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[noodlemind]

Commits from this stacked PR are included in #38, which is now the single consolidated PR.