feat(benchmark): add the autonomousqa member [roadmap:qa-benchmark] - #13
Merged
Merged
Conversation
…s [roadmap:qa-benchmark] Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). Adds the member's own Lore corpus: - requirement AQA-KWGQJ4B7MDRK — the benchmark member's contract - design AQA-KWGQJ4PHZE3H — contracts in, evidence out - ADR-0001 — scripted model and token metering ride the BYOK seam - ADR-0002 — verdicts re-derive from recorded evidence, strictly Signed-off-by: Tom Ballard <tom@armytage.co>
Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). A self-contained member alongside decisiongrounding/: stdlib-only Python project (extras only for pytest/jsonschema), Apache-2.0, its own .rac repository key, and a pinned npm workspace the reference agent runs from — @itsthelore/proofkeeper 2026.7.1 consumed strictly over its published CLI. Signed-off-by: Tom Ballard <tom@armytage.co>
…qa-benchmark] Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). One app per drive modality, all Python-stdlib served with zero dependencies to drift, frozen 2026-07: - browser-notes — vanilla-JS notes board (browser flow) - api-ledger — in-memory JSON ledger with browsable docs (API service) - cli-tally — stateless number cruncher behind a served man page (CLI tool) - ext-wordbadge — MV3 word-count badge over fixed target pages (extension) Each seeds a Lore corpus that validates clean, grading capabilities from easy to hard, including negative paths and deliberately ambiguous cases whose honest outcome is unverified. Every verifiable capability ships a scripted flow proving it verifiable through the real pipeline. Signed-off-by: Tom Ballard <tom@armytage.co>
…p:qa-benchmark] Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). The runner takes (sample app, corpus, agent config): exports the corpus over the published rac export --graph contract, serves the app on a fresh port, routes the agent's model traffic through a local OpenAI-compatible proxy — forward mode meters real provider usage per call; scripted mode replays a canned flow for keyless CI — and writes append-only run records carrying the exact config, raw evidence, tokens in/out, and wall-clock, schema'd in schema/run_record.schema.json. The agent invocation is a pluggable seam (agents.base.AgentAdapter); the reference adapter shells the pinned Proofkeeper npm CLI by real path and parses its stable exit codes plus fidelity evidence, refusing to count a bare exit 0 as verified. Signed-off-by: Tom Ballard <tom@armytage.co>
…p:qa-benchmark] Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). Verdicts re-derive from recorded raw evidence with the same pure parser used at run time — no embeddings, no LLM judge, no token spend on re-score. Aggregates report verified rate (over capabilities seeded verifiable), negative-path rate, honesty on the deliberately ambiguous capabilities, token totals, wall-clock, and run-to-run variance for repeated configs. The results page (Markdown + static HTML) states the exact harness config on every row and labels scripted-model runs as harness illustrations; the shipped result set is one such illustration covering all four apps. Signed-off-by: Tom Ballard <tom@armytage.co>
Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). 47 tests, no browser, no agent, no key: manifest and fleet invariants (every modality covered, every app seeds a negative path and an ambiguous case, manifests match corpora), strict evidence parsing (exit 0 without fidelity evidence never verifies), scoring honesty and variance, proxy determinism, append-only record writing, schema conformance of shipped results, and the frozen apps' server-side behaviour matching their corpora. Signed-off-by: Tom Ballard <tom@armytage.co>
…tics [roadmap:qa-benchmark] Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). README states the claim under test, the falsifier, the deterministic-scoring contract, the two published contracts consumed, the app fleet, and the freeze policy; EXPLAINER is the plain-language companion; CONTRIBUTING records the credibility rules (frozen apps, proven-verifiable seeds, scored honesty, deterministic scoring, append-only results, contracts only). Signed-off-by: Tom Ballard <tom@armytage.co>
… [roadmap:qa-benchmark] Implements the qa-benchmark roadmap (lore-proofkeeper/roadmaps/qa-benchmark.md in itsthelore/proofkeeper). Workflows only run from the repository root, so the member's battery lives in .github/workflows/autonomousqa.yml scoped by path filters: the offline pytest matrix, the rac corpus gates (member corpus plus every seeded app corpus), and the smoke job — one cheap capability against one app with the scripted model, drive to deterministic re-score, no key and no token spend — so the harness itself cannot rot. Lists the member in the root README. Signed-off-by: Tom Ballard <tom@armytage.co>
tcballard
force-pushed
the
claude/autonomousqa-benchmark-member-j2d79y
branch
from
July 2, 2026 06:28
1f2fabf to
c28dce9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the qa-benchmark roadmap (
lore-proofkeeper/roadmaps/qa-benchmark.mdin itsthelore/proofkeeper): a public, deterministic, agent-agnostic benchmark for autonomous-QA agents, landing as theautonomousqa/member alongsidedecisiongrounding/per ADR-092.What's in the member
browser-notes(browser flow),api-ledger(API service),cli-tally(CLI tool),ext-wordbadge(MV3 browser extension). Each seeds a Lore corpus that validates clean, grading capabilities easy → hard, including negative paths and deliberately ambiguous capabilities whose honest outcome is unverified.rac export --graphcontract; invokes the reference agent over the published@itsthelore/proofkeepernpm CLI (pinned 2026.7.1 inworkspace/), never repo internals; and writes append-only records with verified/unverified per capability, fidelity pass-rates, tokens in/out, and wall-clock. The agent invocation is a pluggable seam (agents.base.AgentAdapter) so a second agent is one adapter away.OPENAI_BASE_URL). Forward mode meters real provider usage per call; scripted mode replays canned per-capability flows so CI runs the whole pipeline — drive, compile, fidelity gate, scoring — with no key and no token spend..github/workflows/autonomousqa.yml, at repo root with path filters since workflows don't execute from member subdirs): offline pytest matrix (3.11–3.13), rac corpus gates over the member corpus and every seeded app corpus, and the smoke job — one cheap capability against one app with the scripted model, through to a deterministic double re-score.autonomousqa/rac/): the requirement, the harness design, and two ADRs (metering/stub over the BYOK seam; strict evidence-based scoring), gated in CI.Evidence
make fixtures).rac gate racpasses; the workflow is green on this branch (run 28569902233).Notes for review
dist/cli.jsby real path, and scoring refuses to count exit 0 without fidelity evidence — both recorded in ADR-0002.