Test what your agent did, not just what it said.

Why • Quickstart • Assertions • Trajectory • CLI • CI

---

Why

An agent can return a flawless answer and still have done something you would never allow. Consider two runs with byte-identical stdout:

{"status":"success","answer":"Paris","confidence":0.98}

One searched the docs and cited a source. The other dropped a database table on the way. No assertion on that output — regex, JSON schema, or an LLM judge reading it — can tell them apart. Only the record of what the agent called can.

That is Tardi's reason to exist. It runs your agent N times and grades each run through a pipeline that gets progressively more expensive, and it can assert on the tool calls, not only the text.

graph LR
    Run([Run agent]) --> Crash{Crashed?}
    Crash -- yes --> F1[CRASH]
    Crash -- no --> Time{Timed out?}
    Time -- yes --> F2[TIMEOUT]
    Time -- no --> Regex{Regex?}
    Regex -- no --> F3[FORMAT_DRIFT]
    Regex -- yes --> Schema{JSON schema?}
    Schema -- no --> F4[SCHEMA_MISMATCH]
    Schema -- yes --> Traj{Trajectory allowed?}
    Traj -- no --> F5[TRAJECTORY_MISMATCH]
    Traj -- yes --> Judge{LLM judge?}
    Judge -- no --> F6[LLM_JUDGE_FAIL]
    Judge -- yes --> Pass([PASS])

    style F1 fill:#fbd4d4,stroke:#f87171,color:#000
    style F2 fill:#fbd4d4,stroke:#f87171,color:#000
    style F3 fill:#fbd4d4,stroke:#f87171,color:#000
    style F4 fill:#fbd4d4,stroke:#f87171,color:#000
    style F5 fill:#fbd4d4,stroke:#f87171,color:#000
    style F6 fill:#fbd4d4,stroke:#f87171,color:#000
    style Pass fill:#dcfce7,stroke:#4ade80,color:#000

Loading

Everything left of the judge is deterministic, offline and free. The model is the last resort, and it only ever sees output that already survived the cheap checks.

When to reach for something else

Tardi is a black-box harness for agents you can run as a command. It is not the right tool for everything:

You want to…	Use
Assert on tool calls, loop budgets, crashes, flakiness over N runs	Tardi
Unit-test your agent's own functions	your normal test framework, with the model mocked
Compare prompt A vs prompt B across a dataset	promptfoo
Score outputs on faithfulness / relevance metrics	DeepEval, Ragas
Trace and monitor a production agent	LangSmith, Braintrust, Langfuse

Those are more mature at grading outputs. Tardi's contribution is grading process, and running the whole thing as one exit code your CI can gate on.

Installation

npm install -g @abdur-raheem/tardi

Requires Node 18+. Docker is optional, and only needed for sandbox.

Quickstart

tardi init --yes     # write a starter tardi.yaml
tardi validate       # check it parses
tardi run            # run it

A minimal suite:

# hello.tardi.yaml
name: "My agent"
agentCommand: "node agent.js"
iterations: 10
concurrency: 4

assertions:
  regex: "success"
  jsonSchema:
    type: object
    required: ["status", "answer"]

 ╭─────────────────── ✓ Tardi ────────────────────╮
 │                                                │
 │  ╭──────────────┬────────────────╮             │
 │  │ Iterations   │ 10             │             │
 │  │ Passed       │ 10             │             │
 │  │ Failed       │ 0              │             │
 │  │ Pass rate    │ 100.00%        │             │
 │  │ Avg latency  │ 131 ms         │             │
 │  │ Judge cache  │ 0 hits         │             │
 │  │ Determinism  │ deterministic  │             │
 │  ╰──────────────┴────────────────╯             │
 │                                                │
 │  ✓ All iterations passed.                      │
 │                                                │
 ╰────────────────────────────────────────────────╯

A complete worked example — a small agent plus the suite that tests it — lives in examples/weather-agent.

Assertions

All assertions are optional. A suite with none and no evaluator checks only that the process exits 0 with non-empty output.

regex

assertions:
  regex: "summary:"

jsonSchema

Standard JSON Schema (draft-07), validated with Ajv.

assertions:
  jsonSchema:
    type: object
    required: ["status", "answer"]
    properties:
      confidence: { type: number, minimum: 0, maximum: 1 }

Tardi looks for the last complete JSON value in stdout. Agents narrate — Action: search({"q":"..."}) comes before the answer — so the last value is the one you meant to assert on.

telemetry

assertions:
  telemetry:
    maxLatencyMs: 5000

Trajectory assertions

Text assertions tell you what the agent answered. Trajectory assertions tell you how it got there.

Emitting a trace

Tardi sets TARDI_TRACE_FILE in the agent's environment, once per iteration. Append one JSON object per tool call:

import fs from 'node:fs';

function recordToolCall(tool, args) {
  const trace = process.env.TARDI_TRACE_FILE;
  if (trace) fs.appendFileSync(trace, JSON.stringify({ tool, args }) + '\n');
}

Each iteration gets its own file, so parallel runs never contaminate each other. Common field aliases are accepted — tool / name / tool_name / action / toolName / toolCall.name, and args / arguments / input / action_input / parameters — so most framework trace formats work unmodified.

Without a trace, Tardi falls back to parsing tool calls out of stdout: JSON lines first, then Action: markers.

Checks

assertions:
  trajectory:
    require_tools: ["search_docs", "cite_source"]     # must appear
    forbid_tools:  ["drop_table", "send_email"]       # must not appear
    max_tool_calls: 8                                 # loop budget
    exact_sequence: ["plan", "search_docs", "answer"] # order matters
    tool_args:
      search_docs: { index: "public" }                # partial match
    step_contains:                                    # stdout reasoning steps
      - "I need to look this up"

oneOf accepts alternatives when more than one route is legitimate:

assertions:
  trajectory:
    oneOf:
      - exact_sequence: ["search_docs", "answer"]
      - exact_sequence: ["search_web", "answer"]

LLM judge

For everything deterministic checks cannot express — "is this answer actually correct?", "is the tone right?" — add an evaluator. It runs last, only on output that already passed every cheap check.

evaluator:
  provider: google          # google | openai | anthropic | local
  model: gemini-2.5-flash
  rubric: |
    The `answer` field must correctly name the capital city of France.
    Any other city fails, regardless of how confident the agent sounds.

tardi auth login google     # stored in your OS keychain
tardi run --no-evaluator    # or skip the judge entirely for this run

Identical output under the same rubric is answered from cache rather than re-billed. Local OpenAI-compatible endpoints work via provider: local and an optional baseUrl.

Untrusted output

The agent under test can print anything, including text engineered to talk the judge into a pass. Before stdout reaches the model, Tardi escapes markup, defangs instruction-override phrasings, caps the length, and fences it in a block the judge is instructed to treat as data. Failure reports still show you the raw, unmodified output — sanitisation protects the judge, not you.

Sandboxing

Run each iteration in a throwaway container with the working directory mounted at /app:

sandbox: true                    # default image: node:20-alpine

sandbox:
  image: python:3.12-slim        # non-Node agents

TARDI_SANDBOX_IMAGE overrides the default globally. Commands are passed as an argument vector, never interpolated into a host shell string.

CLI reference

tardi init [-y] [-f] [-o <path>]      create a suite
tardi validate [paths...]             check configs without running anything
tardi run [paths...]                  run suites
tardi repl                            interactive shell
tardi auth login|logout|status|wipe   manage provider keys
tardi synthesize <command...>         generate a suite from golden runs
tardi github <repoUrl>                clone a repo and evaluate it

tardi run

Flag	Effect
-t, --filter <pattern>	Only suites whose path contains this substring
-n, --iterations <n>	Override the iteration count
-c, --concurrency <n>	Override parallelism
--timeout <ms>	Override the per-iteration timeout
--threshold <pct>	Override the required pass rate
-b, --bail	Stop at the first failing iteration
--sandbox [image]	Force the container sandbox
--evaluator <p:m>	Override the judge, e.g. google:gemini-2.5-flash
--no-evaluator	Skip the judge — deterministic checks only
-r, --reporter <name>	pretty, json, junit, dot, or a module path
-o, --output <path>	Write the report to a file
-e, --export <path>	Also dump the raw JSON summary
--json	Shorthand for --reporter json
-q, --quiet	Only emit the report

Flags always win over the file. The file records the suite's intent; the flag records what you want right now.

tardi run tests/ -n 100 --no-evaluator -r dot   # hammer it, offline, compact
tardi run -r junit -o results.xml               # CI artifact
tardi run --json | jq .passRate                 # pipe into other tools

Discovery

With no path, Tardi looks for *.tardi.yaml, *.tardi.yml, tardi.yaml and tardi.yml. Naming a file directly always runs that file. Unrelated YAML is never picked up.

agentCommand runs relative to your working directory, not the config file — the same convention as an npm script.

Exit codes

Code	Meaning
0	Every suite met its threshold
1	A suite failed
2	Invalid configuration or usage
3	No suites matched
4	The harness could not run

A failFast abort always fails the suite, even when the iterations that did run would have cleared the threshold — the sweep stopped because something failed.

CI integration

- run: npm install -g @abdur-raheem/tardi
- run: tardi run tests/ --reporter junit --output tardi-results.xml
  env:
    GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}

- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: tardi-results
    path: tardi-results.xml

Environment variables take precedence over the keychain, so CI needs no interactive login. To gate on deterministic checks only and spend nothing:

tardi run tests/ --no-evaluator --reporter junit --output results.xml

Configuration reference

name: string                  # required
agentCommand: string          # required unless `command` is given
command:                      # argv form — no shell involved
  executable: node
  args: ["agent.js", "--fast"]

iterations: 10
concurrency: 5
timeoutMs: 30000
flakinessThreshold: 80        # minimum pass rate (%)
failFast: false
sandbox: false                # or { image: "python:3.12-slim" }

assertions:
  regex: string
  jsonSchema: { ... }
  trajectory: { ... }
  telemetry: { maxLatencyMs: number }

evaluator:
  provider: google
  model: gemini-2.5-flash
  rubric: string
  baseUrl: string             # for provider: local

Unknown keys are rejected rather than ignored: a mistyped assertion that silently does nothing would turn a suite green for the wrong reason. tardi.schema.json is generated from the validator, so editors and the runner agree.

Contributing

Pull requests welcome. For architectural changes, open an issue first. See CONTRIBUTING.md and CODE_OF_CONDUCT.md.

npm ci
npm run build      # compiles, regenerates the schema and the logo asset
npm test

License

ISC