docs/costs.md describes what drives the cost of a run and says honestly that the numbers are not measured. That is the right thing to write and the wrong thing to leave standing: for a team deciding whether to adopt this, the difference between "interesting" and "too expensive" is a number.
What to build
The data already exists per run: the transcript prints tokens and, when the endpoint publishes prices, the cost. It has never been collected.
- Aggregate a few dozen real runs per agent: input tokens, output tokens, wall time, and the size of the diff they were given.
- Publish medians and a rough upper bound in
docs/costs.md, with the model and the provider named, and dated. A number without those three is worse than none.
- Separately for the acceptance test, where the number of turns dominates and the variance is much larger.
Perhaps also
A cost ceiling. Nothing in the package stops a run from being expensive; the timeouts are the only backstop, and they measure time rather than money.
docs/costs.mddescribes what drives the cost of a run and says honestly that the numbers are not measured. That is the right thing to write and the wrong thing to leave standing: for a team deciding whether to adopt this, the difference between "interesting" and "too expensive" is a number.What to build
The data already exists per run: the transcript prints tokens and, when the endpoint publishes prices, the cost. It has never been collected.
docs/costs.md, with the model and the provider named, and dated. A number without those three is worse than none.Perhaps also
A cost ceiling. Nothing in the package stops a run from being expensive; the timeouts are the only backstop, and they measure time rather than money.