A curated list of evaluation resources for AI agents — platforms, open-source frameworks, benchmarks, and methodology.
Strong evals are what separate a viral demo from a production agent. The compounding nature of tool calls makes agent evaluation fundamentally harder than single-turn LLM eval, and the ecosystem is fragmenting fast. This list aims to be a tightly curated — not exhaustive — entry point.
- Evaluation Platforms
- Open-Source Eval Libraries
- Agent Benchmarks
- Reading & Methodology
- Related Lists
- Contributing
- License
Hosted platforms (some with open-source cores) that explicitly target agent trajectories, not just LLM calls.
- AgentOps — Agent-specific tracing + replay + cost/quality eval.
- LangSmith — Datasets + LLM-as-judge eval over agent traces; tight LangGraph integration.
- Langfuse — Open-source counterpart with agent-trace observability (MIT).
- Braintrust — Eval-first dev loop with strong agent-trajectory dataset UX.
Code-first frameworks built for agent evaluation, drop into a repo and run in CI.
- Harbor — Containerized agent-eval framework; runs trials at scale across cloud providers (Daytona, Modal). Hosts the Terminal-Bench 2.0 registry.
- Inspect AI — UK AISI's eval framework; the de-facto choice for frontier-model agent evals (tool use, multi-step tasks).
- agentevals — Ready-made trajectory evaluators for agents.
- AWS Agent Evaluation — Generative-AI-powered framework for testing virtual agents.
- Strands Evals — Multi-agent interaction analysis + automated experiment generation.
- DeepEval — Pytest-style; agent-specific metrics (task completion, tool correctness).
- RagCiteCheck — Python CLI for evidence-stability checks across RAG/agentic retrieval runs.
- SWE-bench / SWE-bench Verified — Resolve real GitHub issues end-to-end. The standard.
- Terminal-Bench — Agents operating a real shell across multi-step tasks; v2 ships via Harbor.
- LoCoBench-Agent — Long-context software engineering in an interactive setting.
- WebArena — Realistic self-hosted web environments (e-commerce, GitLab, Reddit clone).
- VisualWebArena — Multimodal extension of WebArena.
- BrowserGym — Gym-style toolkit for browser agents; common substrate.
- AgentLab — ServiceNow's dev+eval framework atop BrowserGym.
- Mind2Web — Generalist web agent benchmark.
- WebVoyager — Live-website multimodal agent eval.
- OSWorld — Multimodal benchmark for desktop computer-use agents.
- WindowsAgentArena — Windows-focused computer-use eval.
- Android in the Wild (AitW) — Mobile UI control.
- AndroidWorld — Reproducible Android benchmark.
- ToolBench — 16k+ real-world API tasks for multi-step tool-using agents.
- MCPEval — MCP-based automated task generation + deep eval for tool-using agents.
- τ-bench / τ²-bench — Sierra's go-to benchmark for conversational tool-use agents; policy-bound, dual-control.
- GAIA — General-assistant benchmark; multi-step research with tools.
- BrowseComp — Hard browsing/research questions for autonomous browsing agents.
- AgentBench — 8 environments covering OS, DB, web, code, games.
- AgentBoard — Fine-grained progress-rate metric across domains.
- AgentQuest — Modular framework for measuring agent progress.
- Anthropic — Demystifying evals for AI agents.
- Anthropic — Building effective agents.
- VoltAgent/awesome-ai-agent-papers — recent agent papers, with an eval section.
This repo focuses tightly on agent evaluation. For broader scope, see:
- Vvkmnn/awesome-ai-eval — broader "AI reliability" eval list.
- onejune2018/Awesome-LLM-Eval — LLM eval (EN + 中文), widely referenced.
- zhangxjohn/LLM-Agent-Benchmark-List — benchmark-only list.
- CSHaitao/Awesome-LLMs-as-Judges — LLM-as-judge survey.
- tjunlp-lab/Awesome-LLMs-Evaluation-Papers — academic survey-driven.
- BenchGecko/awesome-llm-benchmarks — benchmarks + leaderboards.
PRs welcome. See CONTRIBUTING.md. Quality bar:
- Agent-focused. Pure LLM benchmarks belong elsewhere unless heavily used for agent eval.
- One line per entry. Link first, terse description after. No marketing copy.
- Stable. Prefer projects with active maintenance and real adoption over preprints with no code.
- Honest. If a tool overlaps heavily with another, note the difference rather than duplicating.
CC0 1.0 Universal — Public Domain.