Harden benchmark integrity: mispriced tokens, false-certainty stat, COI + parity disclosure - #23
Closed
macanderson wants to merge 18 commits into
Closed
Harden benchmark integrity: mispriced tokens, false-certainty stat, COI + parity disclosure#23macanderson wants to merge 18 commits into
macanderson wants to merge 18 commits into
Conversation
Adapters for Claude Code, Gemini CLI, Oxagen, Stella (+ mock for CI); held-out verification the agent can never author; normalized token and uniform cost accounting; matched-config enforcement with unmatched-model warnings; agent-error quarantine; Wilson/McNemar/seeded-bootstrap stats; per-run manifests, transcripts, diffs, and reproduce commands.
Repo moved to github.com/oxageninc/arena. Updates the package name, repository URL, README title + clone command, harness self-name in the run manifest and CLI banner, and the Stella link (now oxageninc/stella).
Adds harbor/ — a Python Harbor adapter (BaseInstalledAgent) that runs your coding agent through Harbor's containerized SWE-bench / Terminal-Bench verifier head-to-head against the agents Harbor already ships (Claude Code, Gemini, Codex, Cursor, Aider, ...), all scored by the same repo-native test suite. - ByoAgent: wire up your own agent from a TOML/JSON spec — no Python. - OxagenAgent / StellaAgent: specs for agents Harbor lacks; worked examples. - Spec-driven install (binary upload or script) + one-shot run + best-effort token/cost parsing with the same cache-aware normalization as the TS side. - A non-zero agent exit never aborts scoring — the verifier decides pass/fail. - Pure logic (spec/metrics/command) is Harbor-free and unit-tested; agent tests run against real Harbor 0.6.1 (39 tests green) and verify Harbor's factory loads each agent by import path. - Docs + run.sh + a CI job (pip install -e .; pytest). Main README + METHODOLOGY now point at this adapter as the scale-up path.
Harbor requires Python >=3.12, so the CI job used 3.11 and could not resolve harbor. Bump CI to 3.12, set requires-python accordingly, and pin the dep to the tested 0.6.x line.
Snapshot a run's per-agent metrics as a committed baseline, then fail CI when a later run regresses: - arena baseline save <runDir> [--agent A] -> arena-baseline.json - arena baseline show - arena gate <runDir> [--require-significant] [--allow-task-mismatch] [...] exits non-zero on regression. Checks per agent: resolve-rate drop (strict, or only when it clears 95% CI noise with --require-significant), median token/cost increase past thresholds, wall-clock (reported; enforced with --speed-increase). Refuses to compare across different task sets unless --allow-task-mismatch. Thresholds via flags or arena-gate.json. src/summary.ts factors the per-agent aggregation into one source of truth shared by report + baseline + gate. 14 new unit tests (51 total); example CI workflow + gate config under examples/. README + METHODOLOGY document it.
…, dependency review Quality: - Add Biome (lint + format), pinned devDep, as a CI gate (pnpm lint). Normalized formatting across src/test; removed dead code Biome caught (unused pricing local in orchestrator; unused test locals). Supply chain / security (standard for popular OSS repos): - .github/dependabot.yml — weekly npm + pip + github-actions updates. - dependency-review workflow — blocks PRs adding high-severity vulns. - OpenSSF Scorecard workflow — security-posture scoring to the Security tab. - SECURITY.md — private vulnerability reporting policy. - CODEOWNERS — routes review to the maintainer. - Least-privilege 'permissions: contents: read' on CI. typecheck + 51 tests + biome ci all green.
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 25.9.5 to 26.1.1. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 26.1.1 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on [harbor](https://github.com/harbor-framework/harbor-cookbook) to permit the latest version. - [Commits](https://github.com/harbor-framework/harbor-cookbook/commits) --- updated-dependencies: - dependency-name: harbor dependency-version: 0.19.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mac Anderson <mac@oxagen.sh>
Bumps [typescript](https://github.com/microsoft/TypeScript) from 5.9.3 to 7.0.2. - [Release notes](https://github.com/microsoft/TypeScript/releases) - [Commits](https://github.com/microsoft/TypeScript/commits) --- updated-dependencies: - dependency-name: typescript dependency-version: 7.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mac Anderson <mac@oxagen.sh>
Bumps the actions group with 7 updates: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4` | `7` | | [pnpm/action-setup](https://github.com/pnpm/action-setup) | `4` | `6` | | [actions/setup-node](https://github.com/actions/setup-node) | `4` | `6` | | [actions/setup-python](https://github.com/actions/setup-python) | `5` | `6` | | [actions/dependency-review-action](https://github.com/actions/dependency-review-action) | `4` | `5` | | [ossf/scorecard-action](https://github.com/ossf/scorecard-action) | `2.4.0` | `2.4.3` | | [github/codeql-action](https://github.com/github/codeql-action) | `3` | `4` | Updates `actions/checkout` from 4 to 7 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v7) Updates `pnpm/action-setup` from 4 to 6 - [Release notes](https://github.com/pnpm/action-setup/releases) - [Commits](pnpm/action-setup@v4...v6) Updates `actions/setup-node` from 4 to 6 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4...v6) Updates `actions/setup-python` from 5 to 6 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5...v6) Updates `actions/dependency-review-action` from 4 to 5 - [Release notes](https://github.com/actions/dependency-review-action/releases) - [Commits](actions/dependency-review-action@v4...v5) Updates `ossf/scorecard-action` from 2.4.0 to 2.4.3 - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](ossf/scorecard-action@v2.4.0...v2.4.3) Updates `github/codeql-action` from 3 to 4 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pnpm/action-setup dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/dependency-review-action dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: ossf/scorecard-action dependency-version: 2.4.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/codeql-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mac Anderson <mac@oxagen.sh>
* chore(deps-dev): Bump vitest from 2.1.9 to 3.2.6 Bumps [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) from 2.1.9 to 3.2.6. - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v3.2.6/packages/vitest) --- updated-dependencies: - dependency-name: vitest dependency-version: 3.2.6 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> * fix * fix --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mac Anderson <mac@oxagen.sh>
Replaces #18 — the Dependabot branch accumulated lockfile conflicts after the vitest-3.x and typescript-7 merges landed on main. This reapplies the same two patch bumps (biome 2.5.2->2.5.4, tsx 4.19.2->4.23.1) on a clean base with a freshly-regenerated lockfile.
…end (#20) Harness (TypeScript): - parse: brace-matched envelope scan so pretty-printed JSON (gemini-cli) parses; line-based JSONL scan kept as the first pass - adapters/base: kill the whole process group on timeout (agents spawn subprocesses that held the stdio pipes open past the cap); settle on "exit" with a drain grace; cache version (was two spawns per trial) - orchestrator: drop dead loadPricing call and module-level pricing cache (load once, pass down); timeout now takes precedence over a post-kill test pass; pass taskDir through AdapterRunArgs instead of the MockAdapter.currentTaskDir mutable static - workspace: async fs/child_process APIs (no event-loop blocking inside the async run loop); disable commit.gpgsign in seeded repos - report: aggregate via perAgentSummary (was duplicated inline math that contradicted summary.ts's single-source-of-truth contract); no more NaN cells when an agent has zero scored trials - cli: validate numeric flags instead of silently running with NaN - METHODOLOGY: document timeout precedence Harbor adapter (Python): - remove dead tomli fallback (package requires Python >= 3.12) - frozen dataclasses for specs and parsed metrics; validate metrics.kind - collections.abc.Mapping; class-level _agent_output/_spec defaults; tolerate float ARENA_TIMEOUT values - add ruff (lint config in pyproject, run in CI); fix all findings Branding + site: - black & white stadium mark (nested geometric stadium shapes), IBM Plex type, all-neutral palette; static site under web/ for arena.oxagen.sh - logo assets in assets/, logo in README
The arena and stella repositories moved from the oxageninc org to the macanderson account; point all badges, links, and package metadata at the new home.
…ch paper (#22) * fix: verification integrity, metric honesty, and CLI hardening Highest-severity first: - Timeouts kill the agent's whole process tree (POSIX process groups); trials can no longer hang on pipes held by orphaned grandchildren, and survivors can no longer tamper with held-out verification. - Zero-token (unparseable-envelope) trials are excluded from token/cost medians and deltas instead of dragging them toward zero past the gate. - claude-code preserves full model ids (no floating "sonnet" alias), so version pinning, matchedModels, and pricing lookups hold. - Gemini normalization counts thoughts (reasoning) and tool tokens. - Pretty-printed JSON envelopes parse (TS + Harbor Python mirror). - Per-trial error containment + incremental results.json; wall clock measures spawn-to-exit only; git-diff failure is "unknown", never "empty"; report table shares perAgentSummary with the gate (no NaN). - CLI: -o short flag (as documented), strict numeric flags, unknown verify ids error, duplicate agent specs rejected, typo'd gate thresholds error instead of silently disabling the check. - Cost with unpriced cache writes is null, never guessed. - Harbor: float-tolerant ARENA_TIMEOUT (never silently unbounded), loud empty-{budget} failure, metrics.kind validated at load. * feat(site): arena.oxagen.sh landing page + Agent Benchmark Protocol draft Static site (site/) for arena.oxagen.sh: landing page with the matched-model scoreboard, and the ABP design spec. Spec source lives at docs/agent-benchmark-protocol.md: four interchange schemas (task, run, trace, verdict), the live GitHub-issue task pipeline with golden traces, smallest-normalized-diff scoring, conformance levels, and roadmap. * docs: The State of Agent Benchmarking (paper) + site rendering Research paper surveying the July 2026 agent-benchmark landscape through the engine-vs-model lens: 28 cited references, gap analysis of four capabilities (matched-model isolation, efficiency grading, statistical CI gating, live tasks with golden traces), COI disclosed. Markdown source in docs/, rendered at arena.oxagen.sh/paper/. README links the site, paper, and spec. * docs: architecture map (module graph, trial data flow, contracts)
…t, disclose COI + parity gaps Pre-launch fairness/methodology audit fixes for the benchmark. Statistics - wilsonInterval(0, n=0) returned [0,0] — false certainty of 0% from zero data. A run with zero scored trials (e.g. every trial excluded as agent-error) would make `arena gate --require-significant` certify a "100% resolve-rate drop". Now returns [0,1] (maximal uncertainty); test updated. - Paired-bootstrap CI used floor(0.025N)/ceil(0.975N) — asymmetric tails (trims 2.5% below, 2.45% above). Switched to the type-1 empirical quantile (ceil(q·N)-1) on both bounds. Pricing - pricing.json priced Anthropic Opus 4.8 at $15/$75 — 3x the real $5/$25 — and Haiku 4.5 at $0.8/$4 vs $1/$5. Every computed cost involving those models (and any unmatched run against a cheaper competitor) was fiction. Corrected to published rates; added source URL + asOf date + a cache-write-TTL note to each Anthropic entry, matching the "verify before publishing" discipline the third-party entries already carried. Claims (README + web) — disclose what the code actually does - Added a prominent conflict-of-interest note: Arena is built by the team behind Oxagen and Stella, two of the agents it benchmarks. METHODOLOGY already required this of publishers; the project now meets its own bar. - "held-out tests the agent can never see or author" was false for local runs (the verify suites sit on disk and are public in this repo). Reworded to "copied in only after the agent exits / not in the workspace", with an explicit not-network-sandboxed caveat pointing at Harbor. - "Autonomy levels are matched as closely as each CLI allows" was contradicted by the adapters: claude-code/gemini are capped at edits-only while oxagen/stella run one-shot ungated (shell included) — an advantage for the author's agents on test-graded tasks. Disclosed the asymmetry and the matched-autonomy flags, and noted the full-env-passthrough (every CLI sees every vendor's keys). Not changed here (needs an owner decision, see the audit report): pinning the adapters to a matched autonomy level, nulling tokens/cost on envelope-parse failure, decoupling the agent-error classifier from parsed token counts, and env-allowlisting spawned agents.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
macanderson
marked this pull request as ready for review
July 30, 2026 04:08
macanderson
enabled auto-merge (squash)
July 30, 2026 04:08
macanderson
disabled auto-merge
July 30, 2026 04:08
macanderson
enabled auto-merge (squash)
July 30, 2026 04:09
auto-merge was automatically disabled
July 30, 2026 21:58
Pull request was closed
macanderson
force-pushed
the
launch-hardening
branch
from
July 30, 2026 21:58
9648785 to
3f6fb41
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-launch adversarial audit of the benchmark's fairness and methodology. A benchmark that ranks the author's own agents against competitors gets the harshest scrutiny in software — these are the things a hostile expert would lead with. All fixes verified:
tscclean, 53/53 vitest, biome clean.📉 Statistics
wilsonInterval(0, n=0)returned[0,0]— false certainty of 0% from zero data. A run where every trial is excluded asagent-error(scoredTrials = 0) would makearena gate --require-significantcertify a "100% resolve-rate drop" — the exact case the agent-error category exists to prevent. Now returns[0,1].floor(0.025N)/ceil(0.975N)— trims 2.5% below but 2.45% above). Switched to the type-1 empirical quantileceil(q·N)-1on both bounds.💵 Pricing (
pricing.json)computedUsdinvolving those models was fiction, and any unmatched run against a cheaper competitor inflated the competitor's cost gap ~3×. Corrected to published rates, and addedsource+asOf+ a cache-write-TTL note to each Anthropic entry (the third-party entries already carried "verify before publishing" notes; the wrong ones didn't).🏷️ Claims vs. reality (README + web)
claude-code/geminiare capped at edits-only whileoxagen/stellarun one-shot ungated (shell included) — a real edge for the author's agents on test-graded tasks. Disclosed, with the matched-autonomy flags and the full-env-passthrough caveat.🧭 Deliberately left for an owner decision (documented in the audit report, not silently changed here)
These change measured behavior or need a judgment call:
--dangerously-skip-permissions/--approval-mode yolofor all).0 tokens / $0.00(today one CLI JSON-shape drift silently hands that vendor the cost column).agent-errorclassifier from parsed token counts (currently vendor-asymmetric).PWD, don't hand every CLI every vendor's keys).Part of a two-repo launch audit (Arena + Stella); the companion report ranks the top 25 pitfalls across both.