docs(bench): cross-repo Q&A benchmark vs Codex/Claude Code (2026-05-07) - #70
docs(bench): cross-repo Q&A benchmark vs Codex/Claude Code (2026-05-07)#70study8677 wants to merge 1 commit into
Conversation
…Code 15 questions per tool against fastapi/requests/sqlmodel measure ag-refresh+ag-ask versus codex exec (gpt-5.5) and Claude Code's Explore subagent (Opus 4.7). Antigravity ran on Ollama qwen3:0.6b (only model that finished downloading on the test box), so the comparison is asymmetric — but the pipeline ran end-to-end on every repo and per-question latency was the lowest of the three. Aggregate composite (0-12): - Antigravity 0.73 / 12 (model-bottlenecked: 8 empty/refusal, 3 fabricated, 4 partial) - Codex 11.93 / 12 (median 106 s/q) - Claude Code 11.87 / 12 (median 42 s/q) Engine + CLI internal pytest still 188/188 green. Full grades, raw answers, and reproducible runner scripts under artifacts/benchmark-2026-05-07/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e05fcb978b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| local rc=$? | ||
| local t1=$(date +%s) | ||
| echo "$((t1-t0))" > "$timefile" | ||
| echo "[codex] $qid done in $((t1-t0))s rc=$rc" |
There was a problem hiding this comment.
The runner captures rc from codex exec but never acts on it, so transient failures (auth/rate-limit/network/tool crash) are treated as successful runs and the benchmark continues with potentially empty or stale outputs. That can silently corrupt scores and latency summaries because downstream scoring has no failure marker for that question.
Useful? React with 👍 / 👎.
| ROOT = pathlib.Path("/tmp/ag-bench") | ||
| QFILE = ROOT / "questions.json" | ||
| GFILE = ROOT / "answers" / "_meta" / "grades.json" |
There was a problem hiding this comment.
The scorer is committed under artifacts/benchmark-2026-05-07/meta/ but hardcodes /tmp/ag-bench for questions.json and grades, so running python3 meta/score.py against the checked-in artifact set fails unless an external mirror exists at exactly that absolute path. This breaks the "reproducible aggregator" behavior and makes local re-scoring non-portable.
Useful? React with 👍 / 👎.
|
Closing — re-running benchmark with Antigravity wired to gpt-5.5 instead of qwen3:0.6b. Will open a fresh PR after the new run. |
1 participant
Summary
Adds a head-to-head benchmark of
ag-refresh+ag-askagainst Codex CLI and Claude Code's Explore subagent on three real-world repos (fastapi,requests,sqlmodel). 15 questions per tool × 3 tools = 45 graded answers.Headline
Caveat
The comparison is deliberately asymmetric: Antigravity ran on Ollama
qwen3:0.6bbecauseqwen2.5-coder:7bstalled at 50 % twice on the test box's network. The frontier models on the other side were unconstrained. The benchmark therefore proves Antigravity's pipeline is robust and fast — but does not evaluate its architecture against a frontier synthesizer. Recommendations for follow-up runs are in §10 of the report.What's in this PR
artifacts/benchmark-2026-05-07/REPORT.md— full write-up (~14 KB)artifacts/benchmark-2026-05-07/answers/{antigravity,codex,claude}/{qid}.{md,log}— every raw answer + transcriptartifacts/benchmark-2026-05-07/meta/grades.json— per-cell 0-3 scores with notesartifacts/benchmark-2026-05-07/meta/score.py— reproducible aggregatorartifacts/benchmark-2026-05-07/meta/run_codex.sh/run_antigravity.sh— runner scriptsartifacts/benchmark-2026-05-07/questions.json— the 15 promptsInternal tests
engine/tests173 ✓ •cli/tests15 ✓ — both green on this branch.Test plan
qwen2.5-coder:7bonce the slow CDN edge clears (expected 5-10× lift on Antigravity quality)ag-askwith a clearer error when the synthesizer returns empty stdout (one of the §10 recommendations)🤖 Generated with Claude Code