Awesome Agentic Benchmarks

A task-level, evidence-backed map of discovery environments for autonomous agents, AI R&D, and recursive self-improvement

Discovery Tasks · Interactive Discovery · Task JSON · 2026 Frontier Tracker · Benchmark Dimensions · Model Coverage · Suite JSON · Contributing

---

Agent benchmark scores are properties of a complete evaluation system—not a bare model:

model + scaffold + tools + environment + compute budget + evaluator

This repository primarily tracks executable discovery tasks in which agents either optimize scientific, mathematical, engineering, or AI-system artifacts, or discover unfamiliar environment rules through interaction. Benchmark suites are indexed as sources of tasks; methods and agent frameworks are recorded only as systems evaluated on those tasks. The broader catalog also covers agent benchmarks for browsing, computer use, coding, post-training, and safety. It is designed as shared evaluation infrastructure for AI4AI and recursive self-improvement (RSI) research.

🎯 Mission: Evaluation Infrastructure for RSI

Our goal is to help advance recursive self-improvement (RSI) by making progress measurable, comparable, and reproducible. An AI system cannot improve itself reliably without knowing which capabilities changed, whether an apparent gain survives controlled evaluation, and what resources or scaffolding produced that gain.

The primary unit is a task such as Cassini gravity-assist trajectory optimization or Erdős minimum overlap—not a method such as SimpleTES or TTT-Discover. To support this goal, the repository collects and organizes:

• task-level discovery datasets with the question, agent-visible input, required artifact, objective direction, evaluator, environment, reference result, and reported discovery result;
• benchmark suites as task sources spanning quantum compilation, astrodynamics, scientific algorithms, AI foundations, mathematics, coding, machine learning engineering, post-training, paper reproduction, tool use, and safety;
• evaluation methods and protocols, including task construction, inputs and outputs, scoring rules, judges, pass criteria, baselines, uncertainty, contamination controls, and reproducibility procedures;
• evaluation environments and resource envelopes, such as tools, agent scaffolds, sandboxes, network access, time limits, token budgets, compute budgets, and hardware;
• cross-model evidence for GPT, Claude, GLM, Kimi, Qwen, and other model families, with explicit distinctions between the agent model, target model, and judge model;
• machine-readable records that future AI4AI systems can search, compare, validate, and use to select evaluations for their own improvement cycles.

The long-term aim is not merely to maintain a leaderboard. It is to build an open evaluation layer for the AI4AI/RSI loop:

propose an improvement → implement it → evaluate across dimensions → analyze regressions and trade-offs → retain verified gains → repeat

We therefore treat every score as a result of a complete experimental configuration—not as an intrinsic property of a model—and prioritize primary sources, transparent protocols, reproducible environments, and dated evidence.

🔥 News

• 2026-08-25 — Added an interactive-discovery layer with ARC-AGI-3 and DiG-bench: 24 individually verified public environments, exact action/feedback/score contracts, private-split safeguards, and official frontier-model coverage.
• 2026-08-25 — Refocused the repository on executable discovery tasks and added a validated 28-task SimpleTES registry across quantum compilation, astrodynamics, scientific algorithms, AI foundations, and mathematics discovery.
• 2026-08-24 — Added a dated 2026 frontier tracker covering TB3/continuous Terminal-Bench, both RSI Bench projects, AI4AI-Bench, TerminalWorld, and new safety benchmarks; unresolved names are now kept separate from verified releases.
• 2026-08-24 — Added role-aware coverage for GPT, Claude, GLM, Kimi, and Qwen, distinguishing agent, target, and judge models.
• 2026-08-24 — Added a 31-benchmark evidence matrix and a broader primary-source verification backlog.
• 2026-08-24 — Released the initial machine-readable registry with MLE-bench, NatureBench, and PostTrainBench.
• 2026-08-24 — Added JSON Schema, dependency-free validation, and GitHub Actions checks.

📑 Table of Contents

• Why this repository
• Mission: Evaluation Infrastructure for RSI
• Benchmark dimensions
• Discovery task registry
• Featured benchmark suites
• 2026 frontier tracker
• Benchmark suite catalog
  • AI R&D and scientific discovery
  • Coding and software engineering
  • Tool use and interaction
  • Computer use and daily life
  • General, professional, and safety agents
• Model-family coverage
• Machine-readable registries
• Inclusion criteria
• Contributing
• Related repositories

✨ Why this repository

Agentic benchmarks are difficult to compare. Two results using the same underlying model can differ substantially because of the scaffold, available tools, time limit, GPU budget, feedback channel, network policy, inference configuration, or integrity rules.

Awesome Agentic Benchmarks provides:

• a task-first scientific-discovery landscape;
• machine-readable task and benchmark-suite registries for evaluation infrastructure and research agents;
• evidence-backed coverage of GPT, Claude, GLM, Kimi, and Qwen;
• dated score snapshots tied to the model, scaffold, and resource envelope;
• explicit agent-model, target-model, and judge-model roles;
• an AI4AI/RSI taxonomy based on the object and persistence of improvement;
• benchmark integrity notes covering contamination, artifact substitution, evaluator access, and reward hacking.

🧭 Benchmark dimensions

Dimensions describe what the benchmark evaluates. SFT, DPO, GRPO, RLHF, LoRA, and distillation are training methods—not top-level benchmark dimensions.

Dimension	What it evaluates	Example benchmarks
Coding & Software Engineering	Repository editing, issue resolution, debugging, tests, and terminal work	SWE-bench, Terminal-Bench, SWE Refactor Bench
Machine Learning Engineering	Building and optimizing ML systems against a defined objective	MLE-bench, MLAgentBench, ML-Dev-Bench
Post-Training	Improving a provided base model under a bounded compute budget	PostTrainBench, RSI Bench
Open-Ended AI R&D	Proposing, implementing, and validating AI research improvements	AI4AI-Bench, RSI Bench, RE-Bench
Agent & Harness Improvement	Modifying an agent's scaffold, memory, skills, or reusable procedures and testing transfer	RSIBench, LongWoF-Bench
Scientific Discovery	Solving research problems in scientific domains	NatureBench, EarthVerse, SciAgentArena
Interactive World Discovery	Inferring unfamiliar dynamics, rules, goals, and strategies from action-conditioned feedback	ARC-AGI-3, DiG-bench, EdgeBench
Paper Reproduction	Reconstructing research code, environments, and results	PaperBench, CORE-Bench, SUPER
Web Research & Browsing	Locating and synthesizing hard-to-find information	BrowseComp, WebArena, GAIA
Computer Use & Daily Life	Completing workflows in desktop, mobile, and browser environments	OSWorld, TUA-Bench, MobilePA-Bench
Tool Use	Selecting APIs and tools while following stateful policies	BFCL, tau2-bench, MCP-Atlas
Professional Work	Completing realistic office and knowledge-work deliverables	GDPval-AA, SpreadsheetBench 2, WorkArena
Multi-Agent Coordination	Delegation, collaboration, negotiation, and competition	MultiAgentBench, GAMA-Bench, SOTOPIA
Safety & Security	Harmful actions, permissions, prompt injection, reward hacking, and cyber capability	SkillSafetyBench, TAMAS, HVTB
Evaluation Integrity	Verifier exploitation, reward hacking, leakage, and judge reliability	HVTB, CatchBench
General Agents	Broad planning, reasoning, tool use, and long-horizon execution	OmniaBench, AGENCYBENCH, AgentBench

See the complete definitions and classification rules in Benchmark Dimensions.

🔬 Discovery task registry

The catalog separates three layers:

Layer	Example	How it is stored
Discovery task	Mariner 10 trajectory optimization, ARC-AGI-3 ls20, DiG-bench P-19	Primary registry row with question, input, output, metric, evaluator, environment, and results
Task suite/source	SimpleTES, ARC-AGI-3, DiG-bench	Provenance grouping that exposes one or more tasks
Method or evaluated system	SimpleTES search, TTT-Discover, EFT, AlphaEvolve	Attached to a result or configuration; not treated as the task itself

The Discovery Task Registry contains 52 individually verified tasks from three source suites:

Domain	Tasks	Examples
Quantum compilation	2	Superconducting qubit routing; zoned neutral-atom compilation
Astrodynamics	5	Mariner 10, Voyager 2, Galileo, Cassini, and Rosetta trajectory design
Scientific algorithms	7	LASSO path solving; five ZAPBench horizons; single-cell RNA-seq denoising
AI foundations	7	Three GPU kernels and four scaling-law discovery problems
Mathematics discovery	7	Erdős minimum overlap, autocorrelation inequalities, sum-difference, circle packing, and Hadamard determinant
Interactive world discovery	24	ARC-AGI-3's three anonymous public games and DiG-bench P-1 through P-21

Each task is available in machine-readable JSON. The full source suites are larger—ARC-AGI-3 has 135 environments and DiG-bench has 70—but private environments and public IDs that cannot yet be authenticated are not fabricated as task rows. See Interactive Discovery Tasks for the protocol and extraction policy.

🌟 Featured benchmark suites

These are the first fully documented suite-level entries. Their internal evaluation units should be extracted into the task registry when executable artifacts are available.

Benchmark	Primary dimension	Evaluation unit	Output	Environment	Current headline snapshot
MLE-bench	Machine Learning Engineering	One offline Kaggle competition	Prediction submission	24h, A10 GPU	64.44% Any Medal for the leading comparable entry
NatureBench	Scientific Discovery	One Nature-family scientific ML problem	Executable pipeline and predictions	4h, task-dependent GPU	23.3% Surpass-SOTA for the leading entry
PostTrainBench	Post-Training	One base-model × target-benchmark run	Post-trained model checkpoint	10h, one H100	41.79% weighted average for the leading listed entry, with a fallback caveat

MLE-bench
  Agent builds task-specific ML systems
        ↓
NatureBench
  Agent attempts to exceed published scientific ML systems
        ↓
PostTrainBench
  Agent directly changes another language model's weights

These benchmarks measure increasingly direct forms of AI-assisted AI development. None is, by itself, a complete recursive self-improvement benchmark: the evaluated agent does not repeatedly improve successor versions of its own improvement mechanism under a controlled causal protocol.

🛰️ 2026 frontier tracker

Fast-moving 2026 work is tracked separately from the stable catalog so that maturity remains visible.

Benchmark	Dimension	Maturity	Why it matters
RSI Bench	Open-Ended AI R&D / Post-Training	Live preview	Long-horizon research tasks with controlled compute and executable verifiers; current public runs include GPT and Claude systems.
ARC-AGI-3	Interactive World Discovery	Live / Released protocol	Agents discover visual-world mechanics and goals without language instructions; official semi-private frontier results remain below 1% RHAE.
DiG-bench	Interactive World Discovery	Released	70 unknown-rule text games, including 21 public tasks; its official leaderboard covers GPT, Claude, GLM, Kimi, and Qwen.
EdgeBench	Environment Learning	Released / extraction queue	134 real-world tasks with 12–72+ hour learning curves; 51 task artifacts are initially released.
RSIBench	Agent/Harness Improvement	Released dataset	Counterfactual test of whether coding-agent self-modifications transfer to held-out tasks.
AI4AI-Bench	Open-Ended AI R&D	Emerging	Agents rewrite training algorithms in frozen research repositories under a B300 compute envelope.
Terminal-Bench	Coding / Terminal	Live, continuous	The former TB3 effort now publishes tagged continuous task sets through Harbor; a score needs an exact release tag.
TerminalWorld	Coding / Terminal	Live	Its human-verified leaderboard currently contains GPT, Claude, GLM, Kimi, and Qwen results in one shared environment.
EarthVerse	Scientific Discovery	Emerging	405 reproducible Earth-science tasks with executable answer units and process rubrics.
SWE Refactor Bench	Coding	Emerging	Whole-repository migrations evaluated by audits, behavior tests, and independently generated hidden tests.
HVTB	Safety / Evaluation Integrity	Emerging	Detects reward hacking in terminal-agent evaluation rather than trusting verifier success alone.
SkillSafetyBench	Safety	Released	Tests malicious skills and local artifacts across six risk domains.

See 2026 Frontier Agentic Benchmarks for task inputs, evaluators, environments, reported scores, status definitions, and the explicit unresolved-name audit for DeTrustAgent.

📚 Benchmark suite catalog

Legend: Detailed = complete registry entry; Tracked = included in the model-coverage and verification pipeline; Live/Preview/Emerging = status is defined in the 2026 tracker; Discovery queue = primary artifacts still need review.

AI R&D and scientific discovery

Benchmark	Dimension	Status	Paper / Code / Leaderboard
MLE-bench	Machine Learning Engineering	Detailed	Paper · Code · Leaderboard
NatureBench	Scientific Discovery	Detailed	Paper · Code · Leaderboard
PostTrainBench	Post-Training	Detailed	Paper · Code · Leaderboard
ARC-AGI-3	Interactive World Discovery	Detailed	Report · Toolkit · Docs
DiG-bench	Interactive World Discovery	Detailed	Paper · Code · Leaderboard
EdgeBench	Environment Learning	Extraction queue	Project · Code
RSI Bench	Open-Ended AI R&D / Post-Training	Live / Preview	Project · Tasks · Runs
RSIBench	Agent/Harness Improvement	Released dataset	Dataset · Code
AI4AI-Bench	Open-Ended AI R&D	Emerging	Paper
RE-Bench	Open-Ended AI R&D	Tracked	Paper · Code
MLR-Bench	Open-Ended AI R&D	Tracked	Paper
PaperBench	Paper Reproduction	Tracked	Paper · Project
InferenceBench	AI Systems Optimization	Tracked	Code
AgentHPOBench	Machine Learning Engineering	Emerging	Paper
SciAgentArena	Scientific Discovery	Emerging	Paper
EarthVerse	Scientific Discovery	Emerging	Paper
ScienceAgentBench	Scientific Discovery	Discovery queue	Paper · Code
EXP-Bench	Open-Ended AI R&D	Discovery queue	Paper · Code

Coding and software engineering

Benchmark	Scope	Status	Project
SWE-bench Verified	Real GitHub issue resolution	Tracked	Code
SWE-bench Pro	Harder professional repository tasks	Tracked	Project
SWE-bench Multilingual	Repository tasks across programming languages	Tracked	Code
SWE-bench Multimodal	UI-facing repository issues with visual context	Tracked	Code
Terminal-Bench	Continuous long-horizon terminal tasks; TB3 lineage	Live	Code
TerminalWorld	Reproduced real-world terminal environments	Live	Project / Leaderboard
TUA-Bench	General-purpose terminal use	Tracked	Code
SWE Refactor Bench	Whole-repository migrations and technical-debt removal	Emerging	Paper
NetConfArena	Closed-loop multi-device network configuration	Emerging	Paper
SWE-Lancer	Paid freelance software-engineering tasks	Discovery queue	Code

Tool use and interaction

Benchmark	Scope	Status	Project
tau-bench	Stateful customer-service tool use	Tracked	Code
tau2-bench	Dual-control tool-agent-user interaction	Tracked	Code
BFCL	Function calling and API selection	Tracked	Code
MCP-Atlas	Real MCP-server tool use	Tracked	Code · Leaderboard
MCP-Bench	MCP discovery, selection, and execution	Tracked	Code
Toolathlon	Diverse, realistic, long-horizon tool execution	Tracked	Project
ACEBench	Normal, special, and agent function calling	Tracked	Results

Computer use and daily life

Benchmark	Scope	Status	Project
OSWorld	Real desktop applications	Tracked	Code
OSWorld-Verified	Reliability-focused OSWorld subset	Tracked	Project
ScreenSpot-Pro	Professional high-resolution GUI grounding	Tracked	Code
AndroidWorld	Real Android application workflows	Tracked	Code
WindowsAgentArena	Windows application workflows	Tracked	Code
MobilePA-Bench	Stateful mobile personal-assistant workflows	Emerging	Paper
TravelPlanner	Constrained travel planning	Discovery queue	Code

General, professional, and safety agents

Benchmark	Dimension	Status	Project
OmniaBench	General Agent	Tracked	Code
AGENCYBENCH	General Agent	Tracked	Paper
GDPval-AA v2	Professional Work	Tracked	Leaderboard
Claw-Eval-Live	Enterprise Agent Workflows	Tracked	Code
Agent3Sigma	Agent Safety	Tracked	Code
SkillSafetyBench	Skill and Local-Artifact Safety	Released	Paper · Code
TRUST-Bench	Compromised-Tool Robustness	Emerging	Paper
AgentLAB	Long-Horizon Agent Security	Emerging	Paper
TAMAS	Multi-Agent System Safety	Released	Paper · Code
ST-WebAgentBench	Web-Agent Safety and Trust	Emerging	Project
AgentFairBench	Fairness in Stateful Agent Decisions	Emerging	Paper
HVTB	Reward-Hacking Detection	Emerging	Paper
CatchBench	Evaluation Auditing	Work in progress	Paper
TheAgentCompany	Simulated Knowledge Work	Discovery queue	Code
AgentDojo	Prompt-Injection Safety	Discovery queue	Code
AgentHarm	Harmful Agent Behavior	Discovery queue	Code

🤖 Model-family coverage

The target families are GPT, Claude, GLM, Kimi, and Qwen. Coverage is role-aware:

• Agent model — drives the evaluated agent.
• Target model — is trained, modified, or optimized by another agent.
• Judge model — scores outputs but does not perform the task.

This distinction prevents false claims. For example, PostTrainBench uses Qwen3 base checkpoints as target models; that does not establish that a Qwen-powered agent completed PostTrainBench.

Benchmarks with reported results for all five target families already include:

Benchmark	Dimension	GPT	Claude	GLM	Kimi	Qwen
NatureBench	Scientific Discovery	✓	✓	✓	✓	✓
DiG-bench	Interactive World Discovery	✓	✓	✓	✓	✓
SWE-bench Verified	Coding	✓	✓	✓	✓	✓
SWE-bench Pro	Coding	✓	✓	✓	✓	✓
Terminal-Bench	Coding	✓	✓	✓	✓	✓
TerminalWorld	Coding	✓	✓	✓	✓	✓
tau2-bench	Tool Use	✓	✓	✓	✓	✓
MCP-Atlas	Tool Use	✓	✓	✓	✓	✓
MCP-Bench	Tool Use	✓	✓	✓	✓	✓
Toolathlon	Tool Use	✓	✓	✓	✓	✓
OmniaBench	General Agent	✓	✓	✓	✓	✓
AGENCYBENCH	General Agent	✓	✓	✓	✓	✓
Agent3Sigma	Safety	✓	✓	✓	✓	✓
Claw-Eval-Live	Professional Workflows	✓	✓	✓	✓	✓
GDPval-AA v2	Professional Work	✓	✓	✓	✓	✓

The evidence source, exact model version, role, and remaining gaps are recorded in Model Coverage. A checkmark means that a result has been reported; it does not imply that scores produced with different scaffolds are directly comparable.

🧱 Machine-readable registries

The task-level registry is data/discovery-tasks.json, validated against schema/discovery-task.schema.json. The suite-level registry is data/benchmarks.json, validated against schema/benchmark.schema.json.

Each detailed task or suite entry records:

• task type and agent-visible input;
• required output artifact;
• evaluator and aggregation method;
• container, VM, hardware, time, network, and scaffold policy;
• dated score snapshots and caveats;
• integrity risks and controls;
• AI4AI/RSI relevance;
• GPT, Claude, GLM, Kimi, and Qwen evidence by model role.

import json
from pathlib import Path

registry = json.loads(Path("data/benchmarks.json").read_text(encoding="utf-8"))

for benchmark in registry["benchmarks"]:
    coverage = benchmark["model_coverage"]["families"]
    tested_agents = [
        family
        for family, record in coverage.items()
        if record["agent_models"]
    ]
    print(benchmark["name"], tested_agents)

Validate locally:

python scripts/validate_registry.py
python scripts/validate_discovery_tasks.py

✅ Inclusion criteria

A discovery task belongs in the primary registry when it has a concrete optimization question, an agent-mutable output artifact, an executable or inspectable evaluator, a named metric and direction, and enough environment information to attempt reproduction. A method name or paper-level claim without an extractable task remains provenance or a discovery lead.

A benchmark belongs here when the evaluated system must do at least two of the following:

• act over multiple steps rather than answer once;
• use tools, a terminal, files, APIs, a browser, or an interactive environment;
• create or modify an executable artifact;
• run experiments and respond to environment feedback;
• manage a meaningful time, token, or compute budget;
• improve another model, algorithm, scientific result, or AI-development workflow.

Static QA and ordinary single-turn code-generation benchmarks are out of scope unless they are embedded in an agentic tool-use or long-horizon workflow.

Model coverage requires evidence from an official paper, repository, leaderboard, model card, or a reproducible third-party run with released artifacts. Search snippets and marketing tables without methodology remain discovery leads.

🤝 Contributing

Contributions are welcome. Good first contributions include:

• adding an executable discovery task with its question, seed/input, evaluator, environment, metric, and reference result;
• adding a missing benchmark to the discovery queue;
• attaching primary-source evidence for a model-family result;
• upgrading a tracked benchmark into a detailed registry entry;
• correcting an environment, score, model role, or integrity caveat;
• contributing reproducible evaluation artifacts.

Before opening a PR:

1. Read CONTRIBUTING.md.
2. Prefer primary sources.
3. Record the exact model, scaffold, benchmark version, budget, and date.
4. Distinguish agent, target, and judge models.
5. Run python scripts/validate_registry.py.
6. Run python scripts/validate_discovery_tasks.py when changing the task registry.

🔗 Related repositories

• Awesome RSI — research map of recursive self-improvement, harness evolution, AI R&D, and safety.
• Awesome AI for Research — papers, systems, and benchmarks across the research lifecycle.
• Awesome Agent Evals — annotated resources for building and evaluating agents.
• Awesome AI Agent Benchmarks — broad benchmark index across agent dimensions.
• Awesome AI Scientist Benchmarks — AI-scientist and proposal-to-code benchmark list.

📄 License

This repository is released under the MIT License. Individual benchmark papers, datasets, environments, and evaluation artifacts remain subject to their respective licenses.

---

If this registry helps your research, consider contributing a benchmark, an evaluation artifact, or a model-family result.