Global Statistics

[mattpocock]

is there a way to provide global statistics at the end? For example let's say I have a total tokens or time spent generating for each task as metrics. I would like to have an avarage time spent generating and an avarage tokens used

Originally posted by @marcoripa96 in #341

[amannn]

+1 on this — we'd find aggregate run-level stats (tokens, cost, time) really useful too.

For context, we're using evalite to score AI translation quality for a small library of golden cases. Most runs replay from a committed cache (wrapAISDKModel(model, {caching: true})), but when we iterate on the prompt we re-run live and want to know what we just spent — and also what a cold re-run would cost as a budgeting signal.

What we ended up with as a workaround: a small wrapper script that snapshots the cache file's key set, spawns evalite run, and after it exits diffs the cache and prints a two-line summary:

Eval cost
  This run (live calls):        $0.0042  (5 calls)
  Full cache (cold re-run):     $0.1262  (92 entries)

This works because every live call writes one cache entry, so the cache file is the live-call log — no instrumentation needed. The trade-off is that we lose this if our caching layer ever stops being 1:1 with model calls.

We tried doing it natively via process.on('exit') from a middleware on the wrapped model first, but that turned out to be unreliable inside the vitest worker context (worked on Linux, not on macOS).

If evalite grew a first-class hook for this — say, an onFinished reporter callback that gets the aggregated trace usage from the run — we'd happily rip out the wrapper. Per-row token info via traces in custom columns (after #284) is already great for spotting expensive cases inline; an aggregate at the bottom of the table would close the loop.

For what it's worth, the per-run summary doesn't need to be opinionated about cost (which requires per-model pricing) — even just totals for inputTokens, outputTokens, totalTime per model would be enough for users to compute the rest. We're currently hardcoding our two model IDs and their gateway prices in our wrapper.

— written by Claude Code on behalf of @amannn