Skip to content

Show cache hit rate in the summary, turn table, and in-flight line - #2

Closed
shensmart wants to merge 2 commits into
stv1024:mainfrom
shensmart:feat/cache-hit-rate
Closed

shensmart wants to merge 2 commits into
stv1024:mainfrom
shensmart:feat/cache-hit-rate

Conversation

@shensmart

@shensmart shensmart commented Oct 14, 2026 •

Copy link
Copy Markdown

Why

cacheRatio (cache reads over the whole prompt) has been in shared/pricing.ts since v0.6.0, but it only reached the UI inside the CACHE cell's bottom line when the turn also had a known price. For unpriced models — including anything not in the built-in table and without an override — that line fell back to –, so the ratio disappeared even though the token counts were right there. The session and overview summaries never showed it at all, which is where a cache-hit number is most useful (it is a session-level property, not a per-turn one).

What

Render the existing ratio in three places, without touching the data layer:

  • Session and overview summary rows — a new cache hit stat next to the raw cache token count, aggregated over turns.
  • Turn history CACHE cell — shown independent of pricing. When a price is known the cell keeps its current shape ($0.01 · 69%); when it is not, it now shows just 69% instead of –.
  • In-progress turn line — appended to the existing token line: in 125.9k · cache 2.58M (95%) · out 20.8k · 33 calls. This replaces a near-duplicate (95% cached) suffix on the same line.

Because input is already provider-normalized to the fresh-token form by semantics.ts, the ratio is comparable between exclusive-input (Claude) and inclusive-input (Codex) adapters without further changes. Derived at render time, so existing ledger records are reinterpreted correctly — no migration, no new persisted field.

Verification

  • tsc --noEmit clean; node --test shared/*.test.ts server/*.test.ts → 65/65 pass (the existing cacheRatio tests already cover the arithmetic, including the cache-write denominator).
  • Rendered against real data through the 0.11.2 desktop web client: overview shows cache hit 74%, the panel's per-turn table shows 69% / 83%, and the in-progress line shows (95%). Console clean.
  • paseo plugin ls → running, no error.

Notes

  • This is UI-only; pricing, cost residual handling, and the ledger file format are untouched.
  • Happy to split the summary stat out if you would rather keep the turn table and summary changes separate.

Update: average output throughput, and a pill that fits

Two follow-ups on top of the cache-hit work above.

Can a plugin see token speed? Not the model's rate. Paseo's usage API carries only token counts, cost and context-window fill — no per-token or first-token timing — so the only signal is the span between a turn's start and end. Summing that per summary gives an average output throughput (output tokens ÷ wall clock). Wall clock covers thinking, tool calls, and waiting on the provider, so it is a floor on generation speed, not the token rate; the UI labels it avg out and the README says so explicitly rather than implying a measurement the protocol cannot make.

Summary.durationMs is added as a non-negative field defaulting to 0, so a client reading summaries from a server that predates it still parses. It is summed next to the token counts in both summaryFromRows and the overview accumulator, and rows without a timing contribute nothing rather than dropping the turn.

The pill now leads with tokens, hit rate, and throughput. Context-window fill is dropped from the pill: the chat surface already shows it next to the composer, so it was duplicating the host at the expense of a number that cannot be seen there.

before after
unpriced model 6.50M tok · ctx 12% 26.77M·95%·74 t/s
priced model ≈$0.43 · ctx 47% ≈$0.43·90%·74 t/s

The host caps the pill's button at 160px, and the icon, gaps and padding take 50px of it, so the label has about 110px before the host ellipsizes. Measured advance widths per glyph (12px system-ui, tabular numerals) put % at 11.1px against 1 at 5.6px, so the label is assembled against a measured width table instead of a character count, and drops the throughput (then the hit rate) rather than letting the host clip it. The count itself is never dropped. pillLabel is a pure function with tests covering the ordering and the budget.

Formatting helpers move to shared/format.ts so the pill, panel, and overview render from one implementation.

Verification

  • tsc --noEmit clean; node --test shared/*.test.ts server/*.test.ts → 75/75 pass.
  • Rendered against live data through the 0.11.2 desktop web client at 1440px and 980px: pill reads 26.77M·95%·74 t/s at 157px, unclipped; panel summary shows cache hit 95% and avg out 74 tok/s; per-turn CACHE column shows 69% / 83% / 96% / 97%. Console clean.
  • Noted for review: the throughput figure is a wall-clock average, so it will read low on turns dominated by tool calls or long thinking. If you would rather it not ship at all than ship under that caveat, say so and I will drop that part and keep the pill on tokens + hit rate.

`cacheRatio` (cache reads over the whole prompt) has existed in
shared/pricing.ts since v0.6.0, but the only place it reached the UI was the
CACHE cell's bottom line, and only when the turn also had a known price — so
turns from unpriced models hid the ratio entirely and the session/overview
summary never showed it.

Render it in three places instead:

- the session and overview summary rows, aggregated over turns, as a
  "cache hit" stat next to the raw cache token count;
- each finished turn's CACHE cell, independent of whether a price is known
  (the cost keeps precedence when there is one);
- the in-progress turn line, which already carried the token counts.

Input is the provider-normalized fresh-token count, so the ratio stays
comparable between exclusive-input (Claude) and inclusive-input (Codex)
adapters. No data-layer change: the ratio is derived at render time, so
existing ledger records are reinterpreted correctly.
… and hit rate

Paseo exposes no per-token or first-token timing, so the only speed signal
available to a plugin is the span between a turn's start and end. Summing that
span per summary (and per overview range) gives an average output throughput:
output tokens over wall clock. Wall clock includes thinking, tool calls, and
waiting on the provider, so this is a floor on generation speed rather than the
model's token rate — the README and the code comments say so, and the label is
"avg out" rather than "speed".

Summary.durationMs is added as a non-negative field that defaults to 0, so a
client reading summaries from a server that predates it still parses. It is
summed alongside the token counts in both summaryFromRows and the overview
accumulator; rows without a timing contribute nothing instead of dropping the
turn.

The composer pill then gains the throughput, and gains the cache hit rate that
was previously panel-only. The host caps the pill's button at 160px, and the
icon, gaps and padding take 50px, so the label has about 110px before the host
ellipsizes it. Measured widths per glyph (12px system-ui, tabular numerals)
show `%` at 11.1px against `1` at 5.6px, so the label is assembled against a
measured advance-width table instead of a character count, and drops the
throughput (then the hit rate) rather than letting the host clip it.

Context-window fill is dropped from the pill: the chat surface already shows it
next to the composer, so spending pill width on it duplicated the host at the
expense of the number that cannot be seen there. Formatting helpers move to
shared/format.ts so the pill, panel and overview render from one implementation,
and the pill label becomes a pure function with its own tests.
@shensmart

Copy link
Copy Markdown
Author

Superseded by a fresh PR: the branch was rebuilt so the pill design (compact 33.99M(96%) 70t/s label, context fill removed) and the per-turn throughput landed as one reviewed change instead of follow-up commits. No upstream activity on this one. Closing to keep the queue clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant