Skip to content

feat(benchmark): add the autonomousqa member [roadmap:qa-benchmark] - #13

Merged
tcballard merged 8 commits into
mainfrom
claude/autonomousqa-benchmark-member-j2d79y
Jul 2, 2026
Merged

feat(benchmark): add the autonomousqa member [roadmap:qa-benchmark]#13
tcballard merged 8 commits into
mainfrom
claude/autonomousqa-benchmark-member-j2d79y

Conversation

@tcballard

@tcballard tcballard commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper): a public, deterministic, agent-agnostic benchmark for autonomous-QA agents, landing as the autonomousqa/ member alongside decisiongrounding/ per ADR-092.

What's in the member

  • Four frozen sample apps, one per drive modality, all Python-stdlib served with zero dependencies to drift: browser-notes (browser flow), api-ledger (API service), cli-tally (CLI tool), ext-wordbadge (MV3 browser extension). Each seeds a Lore corpus that validates clean, grading capabilities easy → hard, including negative paths and deliberately ambiguous capabilities whose honest outcome is unverified.
  • Harness — takes (app, corpus, agent config); exports the corpus over the published rac export --graph contract; invokes the reference agent over the published @itsthelore/proofkeeper npm CLI (pinned 2026.7.1 in workspace/), never repo internals; and writes append-only records with verified/unverified per capability, fidelity pass-rates, tokens in/out, and wall-clock. The agent invocation is a pluggable seam (agents.base.AgentAdapter) so a second agent is one adapter away.
  • Token metering and the CI stub are one component: an OpenAI-compatible proxy on the agent's documented BYOK seam (OPENAI_BASE_URL). Forward mode meters real provider usage per call; scripted mode replays canned per-capability flows so CI runs the whole pipeline — drive, compile, fidelity gate, scoring — with no key and no token spend.
  • Deterministic scoring (ADR-066 lineage): verdicts re-derive from recorded raw evidence (stdout + exit code) with a pure parser — no embeddings, no LLM judge; recorded results re-score offline without re-spending tokens. Exit 0 without fidelity evidence scores as an error, never a verification. Honesty is scored: verifying a capability seeded ambiguous counts as a false verification.
  • Results page generated from records: rates by app, by modality, by model, by tier; the exact harness config on every row; run-to-run variance for repeated configs. The shipped result set is a scripted-run illustration, labelled as such.
  • CI (.github/workflows/autonomousqa.yml, at repo root with path filters since workflows don't execute from member subdirs): offline pytest matrix (3.11–3.13), rac corpus gates over the member corpus and every seeded app corpus, and the smoke job — one cheap capability against one app with the scripted model, through to a deterministic double re-score.
  • Member corpus (autonomousqa/rac/): the requirement, the harness design, and two ADRs (metering/stub over the BYOK seam; strict evidence-based scoring), gated in CI.

Evidence

  • All 19 capabilities seeded verifiable verify end-to-end through the real pipeline with the scripted model, across all four modalities (make fixtures).
  • 47 offline tests; rac gate rac passes; the workflow is green on this branch (run 28569902233).

Notes for review

  • The published CLI's npm bin shim was observed to silently no-op (exit 0) when invoked via symlink; the adapter therefore invokes dist/cli.js by real path, and scoring refuses to count exit 0 without fidelity evidence — both recorded in ADR-0002.
  • Apps and corpora freeze on publication; behaviour changes land as new apps (freeze policy in the member README).

tcballard added 8 commits July 2, 2026 06:25
…s [roadmap:qa-benchmark]

Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

Adds the member's own Lore corpus:
- requirement AQA-KWGQJ4B7MDRK — the benchmark member's contract
- design AQA-KWGQJ4PHZE3H — contracts in, evidence out
- ADR-0001 — scripted model and token metering ride the BYOK seam
- ADR-0002 — verdicts re-derive from recorded evidence, strictly

Signed-off-by: Tom Ballard <tom@armytage.co>
Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

A self-contained member alongside decisiongrounding/: stdlib-only Python
project (extras only for pytest/jsonschema), Apache-2.0, its own .rac
repository key, and a pinned npm workspace the reference agent runs from —
@itsthelore/proofkeeper 2026.7.1 consumed strictly over its published CLI.

Signed-off-by: Tom Ballard <tom@armytage.co>
…qa-benchmark]

Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

One app per drive modality, all Python-stdlib served with zero dependencies
to drift, frozen 2026-07:
- browser-notes — vanilla-JS notes board (browser flow)
- api-ledger — in-memory JSON ledger with browsable docs (API service)
- cli-tally — stateless number cruncher behind a served man page (CLI tool)
- ext-wordbadge — MV3 word-count badge over fixed target pages (extension)

Each seeds a Lore corpus that validates clean, grading capabilities from
easy to hard, including negative paths and deliberately ambiguous cases
whose honest outcome is unverified. Every verifiable capability ships a
scripted flow proving it verifiable through the real pipeline.

Signed-off-by: Tom Ballard <tom@armytage.co>
…p:qa-benchmark]

Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

The runner takes (sample app, corpus, agent config): exports the corpus over
the published rac export --graph contract, serves the app on a fresh port,
routes the agent's model traffic through a local OpenAI-compatible proxy —
forward mode meters real provider usage per call; scripted mode replays a
canned flow for keyless CI — and writes append-only run records carrying the
exact config, raw evidence, tokens in/out, and wall-clock, schema'd in
schema/run_record.schema.json.

The agent invocation is a pluggable seam (agents.base.AgentAdapter); the
reference adapter shells the pinned Proofkeeper npm CLI by real path and
parses its stable exit codes plus fidelity evidence, refusing to count a
bare exit 0 as verified.

Signed-off-by: Tom Ballard <tom@armytage.co>
…p:qa-benchmark]

Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

Verdicts re-derive from recorded raw evidence with the same pure parser used
at run time — no embeddings, no LLM judge, no token spend on re-score.
Aggregates report verified rate (over capabilities seeded verifiable),
negative-path rate, honesty on the deliberately ambiguous capabilities,
token totals, wall-clock, and run-to-run variance for repeated configs.

The results page (Markdown + static HTML) states the exact harness config on
every row and labels scripted-model runs as harness illustrations; the
shipped result set is one such illustration covering all four apps.

Signed-off-by: Tom Ballard <tom@armytage.co>
Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

47 tests, no browser, no agent, no key: manifest and fleet invariants (every
modality covered, every app seeds a negative path and an ambiguous case,
manifests match corpora), strict evidence parsing (exit 0 without fidelity
evidence never verifies), scoring honesty and variance, proxy determinism,
append-only record writing, schema conformance of shipped results, and the
frozen apps' server-side behaviour matching their corpora.

Signed-off-by: Tom Ballard <tom@armytage.co>
…tics [roadmap:qa-benchmark]

Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

README states the claim under test, the falsifier, the deterministic-scoring
contract, the two published contracts consumed, the app fleet, and the
freeze policy; EXPLAINER is the plain-language companion; CONTRIBUTING
records the credibility rules (frozen apps, proven-verifiable seeds,
scored honesty, deterministic scoring, append-only results, contracts only).

Signed-off-by: Tom Ballard <tom@armytage.co>
… [roadmap:qa-benchmark]

Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md
in itsthelore/proofkeeper).

Workflows only run from the repository root, so the member's battery lives
in .github/workflows/autonomousqa.yml scoped by path filters: the offline
pytest matrix, the rac corpus gates (member corpus plus every seeded app
corpus), and the smoke job — one cheap capability against one app with the
scripted model, drive to deterministic re-score, no key and no token spend —
so the harness itself cannot rot. Lists the member in the root README.

Signed-off-by: Tom Ballard <tom@armytage.co>
@tcballard
tcballard force-pushed the claude/autonomousqa-benchmark-member-j2d79y branch from 1f2fabf to c28dce9 Compare July 2, 2026 06:28
@tcballard
tcballard merged commit 332d4f2 into main Jul 2, 2026
13 checks passed
@tcballard
tcballard deleted the claude/autonomousqa-benchmark-member-j2d79y branch July 2, 2026 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant