feat(evaluation): add option to save eval results to CSV - #6182
feat(evaluation): add option to save eval results to CSV#6182vaibhav-patel wants to merge 12 commits into
Conversation
Add an optional `output_file` parameter to `AgentEvaluator.evaluate` and `AgentEvaluator.evaluate_eval_set`. When set, per-invocation evaluation results for every metric (both passing and failing) are flattened and written to the given path as a CSV file, making it easy to persist and inspect results from pytest-based eval runs. The option is disabled by default, so existing behavior is unchanged. The parent directory is created if needed, and rows are appended so results from a directory of test files accumulate in a single file. CSV writing reuses the existing text/tool-call formatting helpers and relies on pandas, which is already part of the `eval` optional dependencies. Fixes google#2652.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
@googlebot I signed it! |
|
Hi @vaibhav-patel , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Your PR has been received by the team and is currently under review. We will provide feedback as soon as we have an update to share. |
|
Hi @ankursharmas , can you please review this. |
|
Thank you @vaibhav-patel for your contribution! 🎉 Your changes have been successfully imported and merged via Copybara in commit 3bbc8ed. Closing this PR as the changes are now in the main branch. |
Merge google#6182 Fixes google#2652 PiperOrigin-RevId: 959813686
Summary
Adds an optional, opt-in way to persist
AgentEvaluatorresults to a CSV file, as requested in #2652. A newoutput_fileparameter is added to bothAgentEvaluator.evaluateandAgentEvaluator.evaluate_eval_set. When provided, per-invocation results for every metric (both passing and failing) are written to the given path; when omitted (the default), behavior is unchanged.Motivation
Running agent evals from pytest currently only prints detailed tables (and only for failing metrics). Users want to save results to disk for later inspection/tracking. This implements the CSV suggestion from #2652, adapted to the current evaluation architecture (the codebase has changed substantially since the issue was filed).
What changed
output_file: Optional[str] = NoneonAgentEvaluator.evaluateandevaluate_eval_set.eval_set_id, eval_id, metric_name, threshold, score, eval_status, prompt, expected_response, actual_response, expected_tool_calls, actual_tool_calls..test.jsonfiles accumulates results in a single file.pandas, already declared in theevaloptional dependencies.Backward compatibility
Fully backward-compatible. The feature is disabled unless
output_fileis set.Testing
Added
tests/unittests/evaluation/test_agent_evaluator.pycovering row flattening, the missing-expected-invocation case, single-file writing with directory creation, and append-without-duplicate-header. All pass; the fulltests/unittests/evaluation/suite still collects cleanly.Fixes #2652.