Test your LangGraph agents the way users actually use them — across multi-turn conversations, with deterministic tool mocks and an LLM judge that scores every step.
A pytest-based evaluation framework for testing LangGraph chat agents end-to-end.
Unit tests and mocked LLM calls can verify that your agent's plumbing is wired correctly, but they don't tell you whether the agent actually behaves well. This library fills that gap: it runs real LLM calls against your compiled graph, mocks tool responses so tests are deterministic, and uses an LLM-as-judge to assess whether the agent completed its goal and how good the response was.
The result is a quantitative signal — tool recall, response quality, latency — that you can track over time in CI and use to catch regressions before they reach production.
- Multi-turn conversations — define scenarios with multiple steps; the evaluator simulates a user when the agent needs clarification
- Deterministic tool mocks — replace any tool's coroutine with a fixed return value or callable, without touching the agent code
- Arg-level assertions — verify not just that a tool was called, but that it was called with the right arguments
- LLM-as-judge — works with any LangChain-compatible model (OpenAI, Anthropic, Google, etc.)
- JSON output — results saved per run, easy to diff in CI or feed into dashboards
- Terminal viewer —
eval-viewrenders a color-coded report from any result file
pip install langgraph-chat-agent-evalThe package depends on langgraph and langchain-core. It does not depend on any specific LLM provider — bring your own.
# tests/evaluation/scenarios/my_scenario.py
from langgraph_chat_agent_eval import (
EvaluationScenario, ConversationStep, ExpectedToolCall, ToolMock
)
def build_my_scenario() -> EvaluationScenario:
return EvaluationScenario(
name="search_and_create",
description="User searches for items then creates a new one",
shared_mocks=[
ToolMock(
tool_name="search_items",
return_value=[{"id": "1", "name": "Widget"}],
),
],
steps=[
ConversationStep(
name="search",
user_message="Find all widgets",
expected_tools=[
ExpectedToolCall("search_items", with_args={"query": "widgets"}),
],
expected_answer="Agent lists the available widgets",
),
ConversationStep(
name="create",
user_message="Create a new widget called Gadget",
expected_tools=[
ExpectedToolCall("create_item", with_args={"name": "Gadget"}),
ExpectedToolCall("search_items", required=False), # optional re-list
],
expected_answer="Agent confirms the widget was created",
tool_mocks=[
ToolMock("create_item", return_value={"id": "2", "name": "Gadget"}),
],
),
],
)# tests/evaluation/conftest.py
import pytest
from langchain_openai import ChatOpenAI # or any other provider
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.tools import BaseTool
from langgraph_chat_agent_eval import Evaluator
from my_agent import build_graph, get_tools, AgentConfig
pytest_plugins = ["langgraph_chat_agent_eval.fixtures"]
@pytest.fixture(scope="session")
def eval_context() -> AgentConfig:
return AgentConfig()
@pytest.fixture(scope="session")
def eval_graph(eval_context: AgentConfig):
return build_graph(eval_context, checkpointer=MemorySaver())
@pytest.fixture(scope="session")
def all_tools(eval_context: AgentConfig) -> list[BaseTool]:
return get_tools(eval_context)
@pytest.fixture(scope="session")
def evaluator() -> Evaluator:
return Evaluator(llm=ChatOpenAI(model="gpt-4o", temperature=0))# tests/evaluation/test_my_scenario.py
import pytest
from langgraph_chat_agent_eval import ConversationRunner
from scenarios.my_scenario import build_my_scenario
@pytest.mark.asyncio
@pytest.mark.evaluation
async def test_my_scenario(eval_context, eval_graph, evaluator, all_tools):
runner = ConversationRunner(eval_graph, evaluator, all_tools, eval_context)
metrics = await runner.run_scenario(build_my_scenario())
assert metrics.all_steps_completed
assert metrics.avg_tool_recall >= 0.8
assert metrics.avg_response_quality >= 0.6If using plain pip:
pytest tests/evaluation/ -m evaluation -vIf using uv:
uv run pytest tests/evaluation/ -m evaluation -vResults are saved to tests/evaluation/output/YYYYMMDD_HHMMSS_<scenario>.json.
If using plain pip:
eval-view # shows latest run for each scenario
eval-view path/to/result.json # shows a specific fileIf using uv:
uv run eval-view
uv run eval-view path/to/result.jsonThe viewer prints a summary panel, a per-step table, and a full conversation replay with tool call details and color-coded scores.
The examples/ directory contains a working end-to-end example with a simple notes agent (two tools: search_notes and create_note). It is a good starting point to verify your setup and understand how the framework fits together.
git clone https://github.com/your-org/langgraph-chat-agent-eval
cd langgraph-chat-agent-eval
uv syncSet your OpenAI API key (the example uses ChatOpenAI — swap in any provider you prefer):
export OPENAI_API_KEY=sk-...Run the evaluation:
uv run pytest examples/ -vView the results:
uv run eval-viewSee examples/README.md for a walkthrough of what each file does.
A sequence of ConversationSteps representing a complete user workflow. Shared mocks apply to all steps and can be overridden per step.
One logical goal within the conversation. The runner drives the agent through up to max_turns (default 5), with the evaluator acting as the user when follow-ups are needed.
Declares a tool the agent should call. Use required=False for optional tools (calling them earns recall credit; skipping them doesn't penalize). Use with_args to verify specific arguments were passed — matching is a shallow subset check.
ExpectedToolCall("search", required=True, with_args={"query": "widgets", "limit": 10})Replaces a tool's async coroutine with a stub. Use return_value for a fixed response, or side_effect for a callable (useful for stateful sequences) or an exception (to test error handling).
ToolMock("search", return_value=[{"id": "1"}])
ToolMock("flaky_tool", side_effect=TimeoutError("upstream down"))
ToolMock("stateful", side_effect=iter([result_1, result_2]).__next__)Wraps any LangChain BaseChatModel and exposes two methods used internally by the runner:
check_step_completion— judges whether the agent's response satisfied the step goalevaluate_step— scores response quality 0.0–1.0 on correctness, tool usage, clarity, and goal achievement
from langchain_anthropic import ChatAnthropic
evaluator = Evaluator(llm=ChatAnthropic(model="claude-opus-4-6"))Drives a scenario against a compiled graph. Each instance gets a unique thread_id for state isolation.
runner = ConversationRunner(
graph=eval_graph,
evaluator=evaluator,
all_tools=all_tools,
context=eval_context,
scenario_timeout_seconds=120, # optional wall-clock cap
)
metrics = await runner.run_scenario(scenario)run_scenario returns a ConversationMetrics object:
| Field | Description |
|---|---|
all_steps_completed |
True only if every step passed |
avg_tool_recall |
Mean of per-step recall (matched required tools / total required) |
avg_response_quality |
Mean of LLM-judged quality scores (0.0–1.0) |
total_turns |
Total agent turns across all steps |
steps |
List of StepMetrics with per-step detail |
Import with pytest_plugins = ["langgraph_chat_agent_eval.fixtures"] and override in your conftest.py:
| Fixture | Scope | Notes |
|---|---|---|
eval_context |
session | Your agent's config object — must override |
eval_graph |
session | Compiled StateGraph with MemorySaver — must override |
all_tools |
session | Full tool list for mock lookup — must override |
evaluator |
session | Evaluator instance — must override |
_clear_tool_cache |
function | autouse no-op; override to reset tool state between tests |
The framework assumes your graph:
- Has a
messageskey in its state holdingBaseMessageobjects - Uses
interrupt()before tool confirmation steps (auto-approved withCommand(resume="y")) - Uses async
BaseToolinstances (mocking replacestool.coroutine) - Is compiled with a
MemorySavercheckpointer (state persists across turns within a test)