Agent Evals

Agent Evals is a production-oriented framework for testing prompts, workflows, and AI agents with repeatable quality gates. It combines deterministic checks, LLM-as-judge rubrics, score aggregation, reports, and regression baselines in a small, auditable Python package.

The project is designed for teams that need CI-friendly evaluation without locking the core engine to a single model provider.

Highlights

• Deterministic evaluators for text, regex, JSON, JSON schema subsets, length limits, and forbidden content.
• LLM-as-judge evaluator with rubric, criteria, score range, threshold, and weighted aggregation.
• Provider-neutral core with typed protocols for targets, evaluators, and judge clients.
• OpenAI Responses API judge provider using structured JSON output.
• Command-based target adapter for evaluating any prompt, script, service wrapper, or agent runtime.
• Markdown and JSON reports suitable for CI artifacts.
• Baseline comparison to detect quality regressions case by case.
• Strict project setup with src layout, type hints, tests, Ruff, and mypy configuration.

Installation

git clone https://github.com/kaycke1337/agent-evals.git
cd agent-evals

python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

You can also run the CLI without installing the package:

PYTHONPATH=src python -m agent_evals.cli --help

Quick Start

Run the example deterministic suite against the local stub target:

PYTHONPATH=src python -m agent_evals.cli run \
  --suite examples/support_bot.eval.json \
  --target-command "python examples/target_stub.py" \
  --report reports/support_bot.json \
  --markdown reports/support_bot.md \
  --update-baseline baselines/support_bot.json

Example output:

# Eval report: Support Bot Smoke Evals

- Status: PASS
- Score: 1.000 / min 0.900
- Cases: 2 / 2 passed

Target Contract

Targets are executed as commands. The framework sends one test case as JSON on stdin:

{
  "id": "case-id",
  "input": "User message or task",
  "expected": {},
  "metadata": {},
  "tags": []
}

The target can return plain text or structured JSON:

{
  "output": "The response generated by the prompt or agent.",
  "metadata": {
    "model": "my-model",
    "trace_id": "optional"
  }
}

This contract keeps Agent Evals runtime-agnostic. You can evaluate local scripts, service clients, prompt runners, agent CLIs, or wrappers around production systems.

Suite Format

Suites are JSON or TOML files. Evaluators can be defined once at suite level or overridden per case.

{
  "id": "support-bot-smoke",
  "name": "Support Bot Smoke Evals",
  "min_score": 0.9,
  "evaluators": [
    {
      "name": "must-answer-politely",
      "type": "contains",
      "params": { "text": ["Hello", "help"] },
      "weight": 1,
      "required": true
    },
    {
      "name": "no-sensitive-promise",
      "type": "not_contains",
      "params": { "text": ["guaranteed 100%", "full password"] },
      "weight": 1,
      "required": true
    }
  ],
  "cases": [
    {
      "id": "refund-policy",
      "input": "I want to understand the refund policy.",
      "tags": ["support", "policy"]
    }
  ]
}

LLM-as-Judge

Set your environment variables:

export OPENAI_API_KEY="..."
export OPENAI_EVALS_JUDGE_MODEL="gpt-5.4-mini"

Run the quality gate example:

PYTHONPATH=src python -m agent_evals.cli run \
  --suite examples/quality_llm_judge.eval.json \
  --target-command "python examples/target_stub.py"

An LLM judge evaluator defines a rubric, criteria, score range, and threshold:

{
  "name": "quality-rubric",
  "type": "llm_judge",
  "params": {
    "score_min": 0,
    "score_max": 5,
    "threshold": 4,
    "rubric": "The response must be correct, useful, safe, and professional.",
    "criteria": [
      "Answers the user's request directly.",
      "Avoids unsafe or unsupported claims.",
      "Maintains a professional tone."
    ]
  },
  "weight": 1,
  "required": true
}

Regression Baselines

Create a known-good baseline:

PYTHONPATH=src python -m agent_evals.cli run \
  --suite examples/support_bot.eval.json \
  --target-command "python examples/target_stub.py" \
  --update-baseline baselines/support_bot.json

Compare future runs against it:

PYTHONPATH=src python -m agent_evals.cli run \
  --suite examples/support_bot.eval.json \
  --target-command "python examples/target_stub.py" \
  --baseline baselines/support_bot.json \
  --regression-tolerance 0.02

The run fails if a case score drops beyond the configured tolerance.

Evaluators

Type	Purpose
contains	Requires one or more text fragments in the output.
not_contains	Blocks forbidden terms or phrases.
regex	Validates format with a regular expression.
exact_match	Compares the output against an expected string.
json_valid	Requires valid JSON output.
json_schema	Validates a practical subset of JSON Schema without extra dependencies.
max_length	Enforces a maximum character count.
llm_judge	Scores quality with a model-based rubric.

CI Usage

Agent Evals exits with:

• 0 when the suite passes.
• 1 when the suite runs successfully but fails quality gates.
• 2 when configuration, execution, baseline, or provider setup fails.

Example CI command:

PYTHONPATH=src python -m agent_evals.cli run \
  --suite examples/support_bot.eval.json \
  --target-command "python examples/target_stub.py" \
  --report reports/support_bot.json \
  --markdown reports/support_bot.md \
  --baseline baselines/support_bot.json

Development

python -m pip install -e ".[dev]"
pytest
ruff check .
mypy

The codebase uses explicit dataclasses, typed protocols, focused modules, stable CLI behavior, and dependency-light standard-library implementations where practical.

Security

Do not commit API keys, production prompts containing secrets, user data, or private traces. Use environment variables for provider credentials. See SECURITY.md for reporting guidance.

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for the development workflow and project standards.

License

MIT. See LICENSE.