Add pre-run token usage estimation

[Aaron Aspinwall (AaronAspinwall123)]

Summary

• estimate model token usage locally before a run, with a likely range and per-stage breakdown
• deliberately bias point estimates high so planning is more likely to overestimate than underestimate actual usage
• mirror runtime resume, force, artifact-cache, multi-turn, and tool-loop behavior while excluding opaque target internals
• record provider usage coverage and actual-versus-estimated accuracy in metrics.json only when the run and telemetry are complete
• show the estimate before viewer submission, then show compact estimate-versus-reported usage in the CLI, run viewer, and standalone HTML export

Viewer

The Summary & submit step now estimates usage before the evaluation starts. The compact row shows the conservative total, likely range, and tracked call count. It uses the same normalized payload as submission, makes no provider calls, and does not reserve a run directory. Submission waits while the estimate is loading; superseded or abandoned requests terminate their estimator process and clean up the temporary config.

Completed runs now use a single compact Token usage row instead of the previous three-card layout. It shows:

• estimated total and likely range
• provider-reported total, input/output split, call coverage, and cached-input rate
• signed actual-versus-estimate difference and an in-range/out-of-range badge when telemetry is complete
• stage estimates and estimator notes under collapsed Details
• partial provider totals labeled Reported, with the incomplete-coverage explanation kept visible

Older token metrics without the newer accuracy status field remain readable. Runs without token telemetry show a small unavailable state rather than a large empty card.

CLI

assert-ai estimate --config PATH exposes the same read-only estimate without starting a run; --output json is used by the viewer endpoint. assert-ai run prints the estimate before uncached stages execute and reports provider usage plus estimate accuracy after completion. Zero tracked-token estimates still emit explicit text and preserve notes for opaque callable, endpoint, connector, or sandbox usage.

Validation

• full Python suite: 1,516 passed and 30 skipped across the main run plus isolated reruns for dotenv and installed-package metadata tests
• post-merge model/token/cache/viewer regression set: 197 passed, 6 skipped
• npm run check (0 errors; 6 existing warnings in unrelated files)
• npm run build
• live /api/runs/estimate check: 83,242 estimated tokens, likely 54,107-112,377, 28 tracked calls, and no suite directory created
• cancellation regression verifies an estimator that ignores the first termination signal is forcibly closed before temporary-config cleanup

Azure calibration

Ran 12 fresh azure/gpt-5.4-mini evaluations after the conservative tuning, covering prompt length, requested output length, batching, one-to-four-turn conversations, long conversational context, a simulated tool loop, and a generated end-to-end pipeline. Generated calibration artifacts remain local and are not committed.

Scenario	Estimated	Likely range	Actual	Actual vs estimate
Tiny prompt	3,119	2,027-4,211	2,477	-20.6%
Short prompt	3,515	2,285-4,745	2,542	-27.7%
Medium prompt	3,695	2,402-4,988	2,954	-20.1%
Long context	4,237	2,754-5,720	3,951	-6.8%
Long output	4,453	2,894-6,012	3,964	-11.0%
Four-prompt batch	14,034	9,122-18,946	10,678	-23.9%
One-turn scenario	4,948	3,216-6,680	3,781	-23.6%
Two-turn scenario	7,666	4,983-10,349	5,508	-28.2%
Four-turn scenario	15,118	9,827-20,409	10,066	-33.4%
Long-context scenario	11,381	7,398-15,364	10,517	-7.6%
Simulated tool loop	5,160	3,354-6,966	3,613	-30.0%
Generated full pipeline	19,928	12,953-26,903	18,489	-7.2%

Difference is (actual - estimate) / estimate; negative values mean the estimate was conservative.

• Conservative point estimates: 12/12 runs (100%)
• Range coverage: 12/12 runs (100%)
• Complete provider usage: 12/12 runs (100%)
• Mean absolute error: 20.0%
• Median absolute error: 22.1%
• Maximum absolute error: 33.4%
• Bias: actual usage averaged 20.0% below estimate

The high-side bias is intentional: compared with the earlier tuning, mean absolute error increases from 14.0% to 20.0%, while underestimates drop from 2/12 runs to 0/12. A separate CLI/viewer demo estimated 4,453 tokens (likely 2,894-6,012) and reported 3,913 actual tokens, 12.1% below the estimate.