Skip to content

Show cache hit rate and average throughput; rebuild the composer pill around them - #3

Closed
shensmart wants to merge 4 commits into
stv1024:mainfrom
shensmart:feat/usage-hit-and-throughput
Closed

shensmart wants to merge 4 commits into
stv1024:mainfrom
shensmart:feat/usage-hit-and-throughput

Conversation

@shensmart

Copy link
Copy Markdown

Why

cacheRatio (cache reads over the whole prompt) has been in shared/pricing.ts since v0.6.0, but it only reached the UI inside the CACHE cell's bottom line when the turn also had a known price. For unpriced models that line fell back to –, so the ratio disappeared even though the token counts were right there. The session and overview summaries never showed it at all, which is where a cache-hit number is most useful.

Separately, nothing in the UI showed how fast a session was running, and the composer pill was spending its width on the context-window percentage — a figure the chat surface already displays next to the pill.

What

Cache hit rate — rendered in three places, derived at render time so existing ledger records are reinterpreted correctly (no migration, no new persisted field):

  • session and overview summary rows, as a cache hit stat aggregated over turns;
  • each finished turn's CACHE cell, now independent of pricing ($0.01 · 69% when priced, otherwise just 69%);
  • the in-progress turn line.

Because input is already provider-normalized to fresh-token form by semantics.ts, the ratio is comparable across exclusive-input (Claude) and inclusive-input (Codex) adapters.

Average output throughput — Paseo reports no per-token or first-token timing, so the only signal is the span between a turn's start and end. Summary.durationMs (non-negative, defaulting to 0 so an older server still parses) is summed next to the token counts in summaryFromRows and in the overview accumulator, and output ÷ duration is shown in the session/overview summary, per turn, and in the pill. Wall clock includes thinking, tool calls, and provider wait, so this is a floor on generation speed, not the model's token rate; the README says so explicitly rather than implying a measurement the protocol cannot make.

Per-turn values share the OUT column's second line, which unpriced turns left as –. A priced turn keeps its output cost there, since that is a reported figure and the rate is one press away in the summary.

Composer pill — now 33.99M(96%) 70t/s, or $0.43(90%) 70t/s when the model has a price.

Why context fill left the pill

It was already on screen. The chat surface renders the context-window bar and its context 267.6k / 950.0k label immediately beside the composer, so the pill was repeating a number the user could read one element to the left — while omitting the throughput, which appears nowhere else in the host. With the pill capped at 160px there is room for tokens, hit rate, or context, not all three; the panel still shows the full context bar (and keeps it when idle).

The cap is 160px, of which the icon, gaps and padding take 50px, leaving ~110px of label before the host ellipsizes. Measured advance widths per glyph (12px system-ui, tabular numerals) put % at 11.1px against 1 and ( at ~5.6px, so the label is assembled against a measured width table rather than a character count. If the exact count is too wide it is re-rendered more coarsely (33.99M → 34M), then the throughput is dropped, and last the hit rate — the count itself is never dropped.

Two details found by measuring rather than estimating:

  • the clip boundary sits between a 110.9px label (renders) and a 113px one (ellipsized), and per-glyph summing reads ~1% high, so the budget is calibrated against real labels;
  • 33.99M(96%) 70t/s needs a thin space, not a plain one: 109px versus 111px, and 111px is the width that clips.

Formatting helpers move to shared/format.ts so the pill, panel and overview render from one implementation, and pillLabel becomes a pure function with its own tests.

Verification

  • tsc --noEmit clean; node --test shared/*.test.ts server/*.test.ts → 77/77 pass (new tests cover the hit-rate ratio, the throughput derivation, the duration aggregation, and the pill's fit-and-degrade order).
  • Rendered against live data through the 0.11.2 desktop web client at several viewport widths: pill reads 33.99M(96%) 70t/s at 109px inside a 159px button, unclipped; session summary shows cache hit 97% and avg out 68 tok/s; per-turn CACHE column shows 69% / 83% / 96% / 97%; per-turn OUT column shows 56t/s / 48t/s / 86t/s / 108t/s. Console clean.
  • Priced-model path checked by construction (the cost keeps the second line); no priced session was available on this machine to render, so that branch is covered by the formatting tests rather than a screenshot.

Notes for review

  • This supersedes Show cache hit rate in the summary, turn table, and in-flight line #2, which was closed unactioned; the branch was rebuilt so this lands as one change rather than follow-up commits.
  • Happy to drop the throughput entirely if you would rather not ship a derived speed figure at all, or to keep the pill at tokens + hit rate only. Both are isolated to shared/format.ts, shared/pill-label.ts, and the two view files.

`cacheRatio` (cache reads over the whole prompt) has existed in
shared/pricing.ts since v0.6.0, but the only place it reached the UI was the
CACHE cell's bottom line, and only when the turn also had a known price — so
turns from unpriced models hid the ratio entirely and the session/overview
summary never showed it.

Render it in three places instead:

- the session and overview summary rows, aggregated over turns, as a
  "cache hit" stat next to the raw cache token count;
- each finished turn's CACHE cell, independent of whether a price is known
  (the cost keeps precedence when there is one);
- the in-progress turn line, which already carried the token counts.

Input is the provider-normalized fresh-token count, so the ratio stays
comparable between exclusive-input (Claude) and inclusive-input (Codex)
adapters. No data-layer change: the ratio is derived at render time, so
existing ledger records are reinterpreted correctly.
… and hit rate

Paseo exposes no per-token or first-token timing, so the only speed signal
available to a plugin is the span between a turn's start and end. Summing that
span per summary (and per overview range) gives an average output throughput:
output tokens over wall clock. Wall clock includes thinking, tool calls, and
waiting on the provider, so this is a floor on generation speed rather than the
model's token rate — the README and the code comments say so, and the label is
"avg out" rather than "speed".

Summary.durationMs is added as a non-negative field that defaults to 0, so a
client reading summaries from a server that predates it still parses. It is
summed alongside the token counts in both summaryFromRows and the overview
accumulator; rows without a timing contribute nothing instead of dropping the
turn.

The composer pill then gains the throughput, and gains the cache hit rate that
was previously panel-only. The host caps the pill's button at 160px, and the
icon, gaps and padding take 50px, so the label has about 110px before the host
ellipsizes it. Measured widths per glyph (12px system-ui, tabular numerals)
show `%` at 11.1px against `1` at 5.6px, so the label is assembled against a
measured advance-width table instead of a character count, and drops the
throughput (then the hit rate) rather than letting the host clip it.

Context-window fill is dropped from the pill: the chat surface already shows it
next to the composer, so spending pill width on it duplicated the host at the
expense of the number that cannot be seen there. Formatting helpers move to
shared/format.ts so the pill, panel and overview render from one implementation,
and the pill label becomes a pure function with its own tests.
Both cells now show three lines: the token count, the reported cost, then the
derived number that describes the count — the hit rate under CACHE, the
throughput under OUT.

The previous arrangement put the throughput on OUT's second line, which only
worked because the second line was empty for unpriced models. As soon as a
price is configured the cost takes that line back and the throughput vanishes,
which is exactly what a reader wants to see least: configuring pricing should
not cost them the speed figure. A third line keeps the reported cost in place
and gives each derived value a stable home.

The hit rate moves off CACHE's cost line for the same reason: it was sharing
`$0.01 · 98%` with the cost, so a long cost could crowd it.
@shensmart

Copy link
Copy Markdown
Author

Withdrawn for now — no upstream activity, and the design is still being iterated locally (the pill and the per-turn rows changed again since this was opened). I'll reopen or repost once it settles. The branch is kept.

@shensmart shensmart closed this Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant