Symptom
When using Gemini 2.5 Flash via OpenRouter, the eval and chat UI report a cost
~10× lower than the actual OpenRouter bill. Concrete evidence from a recent
10-scenario eval sweep:
- Reported cost: ~$0.03
- Actual OpenRouter bill: ~$0.32
Root cause
The chat router (`apps/backend/src/api/routers/chat.py:226`) captures
`usage_metadata.output_tokens` from each agent chunk and passes it to
`calc_cost`. For models that use reasoning / thinking tokens
(Gemini 2.5 Pro/Flash with thinking mode, OpenAI o1-family, etc.), the
provider bills:
```
output_billed = visible_completion_tokens + reasoning_tokens
```
LangChain's standard `usage_metadata.output_tokens` only reports
visible completion tokens. The reasoning tokens are exposed via
`usage_metadata.output_token_details.reasoning` (LangChain >=0.3) or
the equivalent on a per-provider basis — but we don't read it. Result:
the reasoning tokens are billed by the provider but not counted by us.
Evidence from a real DB row:
| model |
input_tokens |
output_tokens |
cost_usd |
| openrouter/google/gemini-2.5-flash |
10790 |
0 |
$0.00324 |
| openrouter/google/gemini-2.5-flash |
14082 |
6 |
$0.00424 |
| openrouter/google/gemini-2.5-flash |
19395 |
0 |
$0.00582 |
A turn that emitted a full `submit_investigation` tool call (hundreds of
output tokens minimum) is being recorded as 0–6 output tokens. The
"missing" tokens are reasoning tokens.
LiteLLM's pricing table for `openrouter/google/gemini-2.5-flash` IS
correct (input $0.30/M, output $2.50/M — thinking-mode rates), so the
discrepancy is purely on the token-count side.
Affected models
Anything with reasoning tokens — confirmed Gemini 2.5 Pro/Flash; likely
also affects OpenAI o1-mini / o1 / o3, Claude with extended thinking,
DeepSeek R1 (`ollama/deepseek-r1`), Grok thinking variants.
Models without reasoning tokens (Sonnet, GPT-4o, Gemma) are unaffected
— their bills match our calc.
Proposed fix
In `chat.py` where we capture usage:
```python
um = getattr(chunk, "usage_metadata", None)
if um:
usage_meta = um
```
Extend to also pull `reasoning_tokens` from
`usage_metadata.output_token_details` and add it to the captured
`output_tokens` before calling `calc_cost`. Persist both numbers
separately on `usage_events` (new column or in the existing metadata
JSONB) so we can show "visible + reasoning" in the UI.
Schema-wise: optionally add a `reasoning_tokens INTEGER` column to
`usage_events` (migration 015 if we do it). Or stash in
`usage_events.metadata` as JSONB without a migration.
Workaround until fixed
Document in eval and chat UI that reported cost is a lower bound for
reasoning-mode models. Real bill: provider dashboard.
Acceptance
- A turn that emits 500+ output tokens is recorded with output_tokens >= 500
(visible + reasoning) for Gemini 2.5 Flash.
- cost_usd on usage_events matches the provider's billed amount within ~5%.
- Tested manually with Gemini 2.5 Flash and at least one OpenAI o1-family
model.
Filed from chat 2026-06-01 after the eval cost discrepancy surfaced.
Symptom
When using Gemini 2.5 Flash via OpenRouter, the eval and chat UI report a cost
~10× lower than the actual OpenRouter bill. Concrete evidence from a recent
10-scenario eval sweep:
Root cause
The chat router (`apps/backend/src/api/routers/chat.py:226`) captures
`usage_metadata.output_tokens` from each agent chunk and passes it to
`calc_cost`. For models that use reasoning / thinking tokens
(Gemini 2.5 Pro/Flash with thinking mode, OpenAI o1-family, etc.), the
provider bills:
```
output_billed = visible_completion_tokens + reasoning_tokens
```
LangChain's standard `usage_metadata.output_tokens` only reports
visible completion tokens. The reasoning tokens are exposed via
`usage_metadata.output_token_details.reasoning` (LangChain >=0.3) or
the equivalent on a per-provider basis — but we don't read it. Result:
the reasoning tokens are billed by the provider but not counted by us.
Evidence from a real DB row:
A turn that emitted a full `submit_investigation` tool call (hundreds of
output tokens minimum) is being recorded as 0–6 output tokens. The
"missing" tokens are reasoning tokens.
LiteLLM's pricing table for `openrouter/google/gemini-2.5-flash` IS
correct (input $0.30/M, output $2.50/M — thinking-mode rates), so the
discrepancy is purely on the token-count side.
Affected models
Anything with reasoning tokens — confirmed Gemini 2.5 Pro/Flash; likely
also affects OpenAI o1-mini / o1 / o3, Claude with extended thinking,
DeepSeek R1 (`ollama/deepseek-r1`), Grok thinking variants.
Models without reasoning tokens (Sonnet, GPT-4o, Gemma) are unaffected
— their bills match our calc.
Proposed fix
In `chat.py` where we capture usage:
```python
um = getattr(chunk, "usage_metadata", None)
if um:
usage_meta = um
```
Extend to also pull `reasoning_tokens` from
`usage_metadata.output_token_details` and add it to the captured
`output_tokens` before calling `calc_cost`. Persist both numbers
separately on `usage_events` (new column or in the existing metadata
JSONB) so we can show "visible + reasoning" in the UI.
Schema-wise: optionally add a `reasoning_tokens INTEGER` column to
`usage_events` (migration 015 if we do it). Or stash in
`usage_events.metadata` as JSONB without a migration.
Workaround until fixed
Document in eval and chat UI that reported cost is a lower bound for
reasoning-mode models. Real bill: provider dashboard.
Acceptance
(visible + reasoning) for Gemini 2.5 Flash.
model.
Filed from chat 2026-06-01 after the eval cost discrepancy surfaced.