Agent Function-Calling Evaluation Harness

This repository is a complete, reproducible system for evaluating how reliably an LLM agent selects tools, fills parameters, decides when to call, and completes multi-step tool chains.

It was built as preparation for the PiPlan.ai agent layer. Because that production runtime and its real tools do not yet exist, the project uses a deterministic retail-support domain to exercise the same core mechanics:

user instruction
  -> model proposes tool calls
  -> agent loop executes them through a sandbox
  -> raw trace is recorded
  -> deterministic judge compares trace with golden behavior
  -> report explains the score and failure modes
  -> calibrated report is locked as a reproducible baseline

The result is not just a benchmark dataset or a collection of scripts. It is a reusable evaluation pipeline whose domain layer can later be replaced with PiPlan.ai tools while preserving the runner, trace, judging, reporting, and calibration architecture.

Purpose and Evaluation Boundary

The harness evaluates function-calling mechanism correctness:

• D1 — Tool Selection: did the agent choose the right function?
• D2 — Parameter Fill: were the arguments complete and correct?
• D3 — Invocation: did the agent call only when a call was appropriate?
• D4 — Chain: did a multi-step request/confirm or request/abort flow remain complete, ordered, and connected?

It does not judge whether a business decision is good. That distinction is intentional:

Layer 1: mechanism correctness  <- this project
Layer 2: decision quality       <- outside this project

System Components

Component	Responsibility
Case suite	Defines 72 deterministic CaseV1 instructions and golden tool behavior across D1-D4
Agent loop	Sends messages and tool schemas to the model, executes declarations, returns tool results, and continues multi-step reasoning
Retail domain	Provides 13 frozen tool contracts, users, orders, products, and return policy
Sandbox	Executes reads against fixtures and returns mutation stubs without changing the fixture
Runner	Resets state per case, invokes the agent, captures complete RawRunV1 traces, and survives case-level errors
Judge	Compares traces with golden calls using deterministic parameter and whitelist rules
Report	Produces overall, dimension, taxonomy, and worst-tool summaries in ReportV1
Calibration	Human-reviews failures, checks repeated full runs, and locks a model/config/input-bound BaselineV1
CLI	Exposes the complete system through make eval-fc and make eval-fc CASE=<id>

Application-level sandbox

The sandbox is a dry-run application executor, not a Docker or VM sandbox. Read tools query frozen JSON fixtures. Mutation tools create pending action stubs, and confirm_action or abort_action resolves those stubs without changing the source fixture. This makes every case repeatable and allows multi-step chains to be tested with zero external side effects.

Frozen retail domain

The OpenAI-compatible toolset contains exactly 13 tools:

• Read: get_user, get_order, list_orders, get_product, check_stock, get_return_policy
• Request: request_address_update, request_cancel, request_return, request_exchange, request_refund
• Confirmation: confirm_action, abort_action

All business mutations follow a two-stage request/confirm flow. That structure is intentionally analogous to PiPlan.ai's future propose/apply mechanism.

The fixtures contain:

• 3 users: U101-U103
• 8 orders: W441-W448
• 10 products: P201-P210
• pending, shipped, delivered, and cancelled order states
• return, exchange, refund, stock, negative, and ambiguity scenarios

Current Result

The canonical baseline is:

Field	Value
Model	qwen/qwen3.6-27b
Provider	Morph, pinned with fallback disabled
Cases	72
Canonical score	65/72 PASS, 90.3%
Judge misjudgment rate	0.0%
Golden errors	0
Baseline	eval/reports/baseline.json
Baseline SHA-256	14d4abafc88f03469a8a11b5f738eceb727fe6adae1326df7a55f367d0f2846a

An independent clean-environment run produced 66/72 PASS, 6 FAIL, 0 SKIP, and 0 AGENT_ERROR at 91.7%. It uses the same model, provider, configuration, and seven input hashes, so it is comparable to the baseline, but it does not replace it.

Quick Start

1. Create the environment

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
cp .env.example .env

Set the local values without committing or printing the API key:

OPENROUTER_API_KEY=sk-or-v1-your-real-key
EVAL_MODE=1

2. Run the full evaluation

make eval-fc

3. Run one formal case

make eval-fc CASE=fc-d1-001

The Makefile uses the active environment's python. Override it when needed:

make eval-fc PYTHON=/path/to/python

Both commands run the agent, validate RawRunV1, judge the trace offline, write and validate ReportV1, and print:

raw_run=<path>
report_path=<path>
eval_fc=passed

A case-level FAIL or AGENT_ERROR is a valid evaluation outcome and remains in the report. It does not make orchestration fail. Missing configuration, an unknown case, runner failure, or invalid raw/report artifact returns nonzero.

How a Run Works

For every selected case, the runner:

1. loads the frozen model, provider, prompt, tools, cases, and fixtures;
2. validates the formal case suite before any model request;
3. creates a fresh Sandbox;
4. sends only the case's instruction to the model;
5. executes model-declared tool calls and appends tool results;
6. records each call, parameter object, error, timestamp, and final text;
7. atomically writes RawRunV1;
8. validates the raw artifact;
9. judges it offline against the golden sequence;
10. writes and validates ReportV1.

The model never executes Python functions. It produces structured call declarations; the local agent loop owns parsing and execution.

Understanding Evaluation Results

The terminal output gives the headline result. The JSON report preserves the full evidence:

• summary.verdict_counts contains PASS, FAIL, SKIP, and AGENT_ERROR.
• summary.scored_count is PASS plus FAIL.
• summary.accuracy is PASS divided by scored_count.
• SKIP and AGENT_ERROR remain visible but are excluded from the accuracy denominator.
• summary.dimensions contains D1-D4 case counts, verdicts, denominators, and accuracies.
• summary.taxonomy_counts and summary.top_failures identify dominant deterministic failure classes.
• summary.worst_tools and summary.top_worst_tools identify registered tools implicated in FAIL cases.
• Every result preserves its verdict, taxonomy, golden calls, actual trace, warnings, and deterministic diff.

AGENT_ERROR means the harness could not obtain a normally scored agent result, for example because of an API failure, timeout, or MAX_STEPS. It is not silently counted as a model FAIL.

BaselineV1 binds a canonical report to its model slug, provider request, configuration, date, and seven input hashes. Results with a different model slug or case hash are not directly comparable as the same frozen evaluation.

Adding and Maintaining Evaluation Cases

eval/fixtures/cases/function_calling.jsonl is UTF-8 JSONL. Each nonblank line is one object, there are no blank lines, and the file ends with a newline.

Every CaseV1 has exactly these nine fields:

id
dimension
category
instruction
context
golden
negative
skip
skip_reason

Identity and context

• id is unique and follows fc-d[1-4]-NNN.
• dimension is D1-D4 and matches the dimension encoded in id.
• category and instruction are nonempty.
• Instructions are unique after trimming and case-folding.
• context contains exactly db_fixture and notes.
• context.db_fixture is "standard" and notes is nonempty.
• Cases reference the standard fixtures rather than copying their data.

Golden calls

golden contains:

• calls: a list;
• order_sensitive: a boolean;
• optional allowed_extra: a unique list of read-tool names.

Each golden call contains tool and params, may contain param_rules, and contains no other fields.

• tool must exist in the frozen toolset.
• params must satisfy the tool's JSON Schema, including required fields, enums, nested objects, and additionalProperties.
• param_rules keys must also exist in params and use one of the five rules below.
• user_id, order_id, product_id, item_id, and replacement_item_id must exist in the standard fixtures.
• action_id is not fixture-backed but must be nonempty; dynamic action IDs normally use any.

negative and skip are booleans. skip=true requires a nonempty skip_reason; otherwise skip_reason is null. The current suite has no SKIP cases. negative=true means no non-whitelisted call may remain after filtering.

A strict zero-call case uses:

{
  "golden": {
    "calls": [],
    "order_sensitive": false,
    "allowed_extra": []
  },
  "negative": true,
  "skip": false,
  "skip_reason": null
}

A case that may answer directly or optionally read data uses empty calls, negative=false, and an explicit read-only allowed_extra.

Three-variant contract

Each logical scenario has canonical Chinese, colloquial Chinese, and mixed Chinese/English variants. Only id and instruction differ. These fields are identical:

dimension
category
context
golden
negative
skip
skip_reason

Parameter Matching

Rule	Behavior
exact	Default recursive, type-aware equality. Numeric 1 equals 1.0; booleans are not numbers.
normalized_str	Unicode-preserving trim plus lowercase. Punctuation and internal whitespace remain significant.
normalized_date	ISO8601 date or datetime only. Date compares only with date; datetime only with datetime. Aware datetimes normalize to UTC. Naive and aware datetimes cannot be mixed and produce PARAM_WRONG_FORMAT. Natural-language dates are not parsed.
contains	The golden string must be a substring of the actual string.
any	The field must exist and be non-null and nonempty. A string action ID must have positive length.

Lists use order-independent, duplicate-insensitive set semantics. Objects are compared recursively.

• Missing golden field: PARAM_MISSING
• Schema-invalid value: PARAM_WRONG_FORMAT
• Unknown fixture-backed ID: PARAM_HALLUCINATED_ID
• Other unequal value: PARAM_WRONG_VALUE

Read-Only Whitelist

The global default allowed-extra tools are:

get_user
list_orders
get_return_policy

A case-level golden.allowed_extra completely replaces the default; it does not merge with it. An empty list is strict mode.

Only unmatched calls that are successful, schema-valid, fixture-valid, read-only, and named in the effective whitelist may be ignored. Mutations, confirmations, unknown tools, schema-invalid calls, and calls that reference unknown fixture IDs can never be whitelisted. The judge preserves golden-required calls before filtering additional reads.

Adding a Formal Scenario

The suite currently has 24 logical scenarios and three variants per scenario. Adding one logical scenario therefore adds three physical CaseV1 records.

1. Choose a dimension and a new category.
2. Write canonical Chinese, colloquial Chinese, and mixed-language variants.
3. Use the dimension's next consecutive IDs; the current next group is 019..021.
4. Append the three objects to function_calling.jsonl.
5. Add the category's dimension, start ID, tool sequence, ordering, negative, and allowed-extra contract to CATEGORY_SPECS in scripts/phase_2/validate_cases.py.
6. Change the total from 72 to 75 and the selected dimension from 18 to 21. A new negative scenario changes the negative count from 12 to 15; otherwise it remains 12.
7. Update scripts/phase_2/coverage_matrix_v1.md.
8. Update the full-run expected_case_count and result_count contract in scripts/phase_3/validate_raw_run.py, together with tests tied directly to the suite size, especially the full-runner assertions in tests/test_runner.py.
9. Search the code and tests for any other contracts bound to the case count, dimension count, or cases hash, and update the dependencies that apply to the expanded suite.
10. Submit the dimension/category, instructions, golden calls, whitelist behavior, and coverage-matrix changes for project review.

Temporary maintenance rehearsal

Use a disposable clean copy and run only checks valid before a new baseline:

python -m scripts.phase_2.validate_cases
python -m scripts.phase_0_5.validate_domain
python -m pytest -q tests/test_runner.py
make eval-fc CASE=<new-case-id>

Do not run old Phase 5 baseline validation, baseline-specific adjudication, baseline-bound tests, or complete pytest against the temporary 75-case suite. Its case hash has changed, so rejecting old raw files is correct. Delete the copy afterward; do not alter the canonical 72-case suite or baseline.

Permanent expansion

If the scenario becomes permanent:

1. complete project review and land the new suite, validator, coverage, and direct tests;
2. validate the suite/domain and run the new case;
3. archive the old baseline byte-for-byte under eval/reports/baseline_history/72_cases_<old-baseline-id>.json, verify its SHA-256, label it as 72-case-only, and only then vacate the canonical baseline path;
4. under the new case hash, perform normal Phase 5 calibration: full run, human FAIL review, real judge/golden fixes, three full runs, standard consistency, canonical report review, and new baseline lock;
5. when normal consistency passes, lock and validate without adjudication:

python -m scripts.phase_5.baseline lock \
  --review <new-review> \
  --consistency <new-consistency> \
  --output eval/reports/baseline.json
python -m scripts.phase_5.validate_phase_5 \
  --review <new-review> \
  --consistency <new-consistency> \
  --baseline eval/reports/baseline.json

6. update baseline-bound tests and run the complete regression suite.

The adjudication validator used by the canonical checks is tied only to the current 72-case baseline evidence. Do not reuse or generalize it. If a future three-run result requires independent adjudication, stop baseline locking and create an evidence-specific plan.

Validation and Reproducibility

Run the canonical offline checks:

python -m scripts.phase_2.validate_cases
python -m scripts.phase_0_5.validate_domain
python -m scripts.phase_5.validate_v6_adjudication \
  --adjudication scripts/phase_5/v6_v1_4_adjudication_v1.json
python -m scripts.phase_5.validate_phase_5 \
  --review scripts/phase_5/calibration_review_v1.json \
  --consistency scripts/phase_5/three_run_consistency_v6.json \
  --adjudication scripts/phase_5/v6_v1_4_adjudication_v1.json \
  --baseline eval/reports/baseline.json
python -m pytest -q
python -m compileall -q eval scripts tests
git diff --check

Current expected result:

case_validation=passed
domain_validation=passed
v6_v1_4_adjudication_validation=passed
phase_5_validation=passed
139 passed

The adjudication command above protects the baseline's supporting evidence; it is not a generic future-suite workflow.

Troubleshooting

• Missing API key: set OPENROUTER_API_KEY in the local environment or .env; never print or commit it.
• Wrong mode: set EVAL_MODE=1.
• Unknown case: use an ID present in function_calling.jsonl.
• 429/5xx: the runner applies the frozen retry policy and records a structured API error if exhausted.
• TIMEOUT/MAX_STEPS: the case becomes AGENT_ERROR while the batch continues.
• Validator failure: treat it as an artifact or suite-authoring defect; do not bypass hash, count, schema, or input-snapshot checks.

Repository Layout

Makefile                    one-command full/single entrypoint
eval/
  agent/                    model client, prompt, tool loader, agent loop
  fc/                       Sandbox, runner, judge, report, frozen config
  fixtures/                 CaseV1 suite, tool schemas, database, policy
  reports/                  raw runs, reports, locked baseline
scripts/
  phase_0/                  SDK and agent-loop learning demonstrations
  phase_0_5/                frozen-domain validator
  phase_1/                  deterministic evaluation design
  phase_2/                  case validator and coverage matrix
  phase_3/                  RawRunV1 validation and runner evidence
  phase_4/                  ReportV1 validation and judge evidence
  phase_5/                  review, consistency, diagnostics, baseline
  phase_6/                  run/validate/judge/report orchestration
tests/                      139 automated tests
docs/
  spec/                     authoritative FC Eval Harness v1.4 specification
  phases/                   implementation plans and acceptance history

Known Limitations

• The score comes from a virtual retail domain. It measures function-calling mechanics for the model, prompt, and loop, not production decision quality.
• PiPlan.ai must replace the domain and build a new baseline once its runtime and real tools exist.
• v1 does not use an LLM judge.
• v1 does not evaluate multi-turn user conversations, concurrent tool calls, deep semantic equivalence, cross-model ranking, or system performance.

Specification and Project History

The authoritative specification is docs/spec/FC_Eval_Harness_SPEC_v1.4.md.

The phase documents preserve detailed implementation notes and design rationale without turning this README into a chronological log:

• Phase 0 and 0.5: docs/phases/phase_0_plan.md, docs/phases/phase_0_5_plan.md
• Phase 1: docs/phases/phase_1_plan.md, scripts/phase_1/fc_eval_design_v1.md
• Phase 2: docs/phases/phase_2_plan.md, scripts/phase_2/coverage_matrix_v1.md
• Phase 3: docs/phases/phase_3_plan.md, scripts/phase_3/phase_3_verification.md
• Phase 4: docs/phases/phase_4_plan.md, scripts/phase_4/phase_4_verification.md
• Phase 5: docs/phases/phase_5_plan.md, scripts/phase_5/phase_5_verification.md
• Phase 6: docs/phases/phase_6_plan.md

The README describes the project as a whole; the phase documents retain the detailed record of how the system was built.