feat(cost): real usage across all providers, /cost subcommands, budget gates and session persistence - #1385
Merged
Merged
Conversation
…t gates and session persistence Cost tracking overhaul so /cost reports what the session actually spent: - Bedrock now parses real usage from all four surfaces: InvokeModel Anthropic body, Converse typed TokenUsage, OpenAI-compatible family and the Mantle endpoint. Previously it was a chars/4 estimate dressed up with real prices. - ClaudeAI captures usage on the plain SendPrompt path too, buffered and OAuth stream, with per-instance state replacing the stale package global; tool path mirrors into the instance. - ZAI, MiniMax and Moonshot tool paths now mark usage as real API data and surface cached and reasoning token details via the shared parser. - Gemini captures cachedContentTokenCount and thoughtsTokenCount. - OpenRouter requests accounting and records the actually billed usage.cost, which overrides local table math. - The fallback chain forwards usage and stop reason from the entry that served the request and exposes the served provider and model. - Workers and subagents, MoA plain and XML rounds, and scheduler runs are now accounted for, attributed to the model that served them. Pricing engine: - Cache pricing per family with correct semantics: Anthropic cache tokens are additive to input tokens, OpenAI and Gemini cached tokens are a subset of the prompt count and are billed once at the discounted rate. Cache read rates: Claude 10 percent, OpenAI 50 percent, Gemini and DeepSeek 25 percent. - Unknown models are no longer silent zero cost: the record carries a pricing-known flag and /cost lists unpriced models explicitly. - deepseek-reasoner resolves to the R1 tier instead of the generic one. Command surface: - /cost reset closes and persists the period and starts a fresh one; /cost last shows the previous snapshot; /cost sessions lists recent ones; /cost export writes the snapshot as JSON. Completer and palette wired; lenient arg parsing. - Summary adds reasoning tokens, per-model source tags, provider-billed totals, real cache savings in dollars and the unpriced-model notice. Budget: - Proactive one-shot notice the turn a session crosses the warning or exceeded level, in chat, agent and coder. - CHATCLI_BUDGET_HARD_STOP refuses new turns once the budget is exhausted, in chat, agent loop and RPC surfaces. Off by default. - Budget envs are reloadable via /reload and exposed in /config. Persistence: - Write-through snapshots to the costs store under the user config dir, atomic writes, 90-day retention, saved on shutdown and reset.
diillson
added
the
large-pr-approved
Contributor
Quality GateResult: ✅ all floors passed
Config: .github/quality-gate.yml. Workflow: |
added 2 commits
August 24, 2026 19:42
…ing, chat-ask double count, cache math and races Ten confirmed findings from the verification pass: - Workers record usage PER CALL (not one merged tally): a single provider-billed call no longer marks a whole worker run's tokens as billed, and the budget hard stop now gates every worker LLM call so an in-flight dispatch wave stops mid-run when the budget exhausts. - Chat-ask/knowledge turns are no longer double-counted: the tool-round records flag the turn and handleChatTurnResult skips its buffered re-record. - Cache carve-out only applies when a discount rate exists — families without a published cache rate bill cached tokens at the plain input price instead of free. - Cache additive-vs-subset semantics follow the reporting schema, not the model name: Claude served via OpenRouter uses subset accounting. - OpenAI native-tools path mirrors usage into the client state (agent sessions on OpenAI were recording estimates or stale counts). - ZAI, MiniMax, Moonshot and OpenAI tool paths clear per-call usage so a usage-less response falls back to estimation, never a stale re-count. Claude LastUsage is instance-only (global sink write-only). - estimateTurnCostUSD delegates to the record formula — a single source of truth so footer and /cost can never drift. - Cost snapshots write through unique temp files (concurrent saves no longer race) and stale .tmp files are pruned. - OpenRouter family-substring matches propagate the real pricing-known flag so unpriced models surface instead of reading as free. - The scheduler bridge uses a dedicated client instance, ending the shared-client usage-state clobber with interactive turns; scheduled runs are budget-gated.
diillson
force-pushed
the
feat/cost-tracking-overhaul
branch
from
August 24, 2026 22:52
eabb468 to
e9ab03c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full overhaul of cost tracking so
/costreports what the session actually spent — across every provider, mode and surface — plus subcommands, proactive budget notices, an optional hard stop, and cross-session persistence.Real usage plumbing (was estimated or lost)
TokenUsageincl. cache read/write), OpenAI-compatible family and the Mantle endpoint. Previously chars/4 estimates dressed up with real prices.SendPromptpath (buffered + OAuth stream via SSEmessage_start/message_delta), with per-instance state — the package-global is now write-only legacy, so parallel clients (workers, MoA) can never cross-attribute or re-count tokens. The OAuth title request never clobbers the turn's numbers.IsRealvia the shared parser (were labeled "character estimate"), and mirror into the client usage state.cachedContentTokenCountandthoughtsTokenCount.OPENROUTER_API_URLgateways keep a strictly OpenAI-compatible payload) and records the actually billedusage.cost. A key mixing billed and unbilled calls prices the uncovered remainder from the tables and adds both parts.LastUsage/LastStopReasonfrom — and attributes to — the entry that actually served the request.Pricing engine
PricingKnownflag + explicit/costnotice listing unpriced models.deepseek-reasonerresolves to the R1 tier./costcan no longer disagree.Command surface
/cost reset(closes + persists the period),/cost last(--last/-l),/cost sessions,/cost export [path]. Lenient parsing; completer + palette wired (bare/coststill runs the summary directly).(API)/(estimate), provider-billed lines, real cache savings in dollars, and stable largest-spend-first ordering.Budget
CHATCLI_BUDGET_HARD_STOPrefuses new turns once exhausted — chat, agent loop, RPC/gateway, scheduler and MoA (the unattended/parallel surfaces where runaway spend hurts most)./reload, inreloadableEnvVars, and surfaced in/config.Persistence
~/.chatcli/costs/, saved on shutdown and reset, 90-day retention, labeled with the bound session name.Verification
go build ./...clean; full test suite green (103 packages), including new coverage for pricing/cache semantics, mixed billed+table keys, budget transitions and hard stop, reset/persistence round-trip,/costsubcommand rendering, Bedrock/Claude-stream/chain/tally usage capture, and completer/palette wiring.large-pr-approved.Adversarial review round 2 (applied in b62943c)
A second full verification pass confirmed 10 findings, all fixed: per-call worker usage recording (billed-token accounting stays correct and the budget hard stop now gates every worker LLM call mid-wave); chat-ask turns no longer double-counted; cache carve-out only with a real discount rate (zero-rate families bill cached tokens at input price); cache additive/subset semantics follow the reporting schema (Claude via OpenRouter = subset); OpenAI native-tools usage mirrored (agent sessions on OpenAI were estimating); per-call usage resets on ZAI/MiniMax/Moonshot/OpenAI tool paths;
estimateTurnCostUSDdelegates to the single record formula; unique-temp atomic snapshot writes + stale.tmppruning; OpenRouter family matches propagate the pricing-known flag; scheduler bridge on a dedicated client (no shared usage-state clobber) and budget-gated.