feat(viewer): explain prompt cache misses in the turn detail - #437
feat(viewer): explain prompt cache misses in the turn detail#437YoungCan-Wang wants to merge 3 commits into
Conversation
A turn that misses the prompt cache shows Cache Read 0 with a large Cache Create, but the token row alone cannot say why: a cold start and a turn that idled past the 5-minute cache TTL look identical. Reviewers had to cross-check timestamps by hand to tell them apart. Add a diagnostic card above the detail sections that names the reason. Cold start reads "Initial prompt cache creation"; a gap longer than the TTL reads "Cache expired (idle longer than 5 min)". Turns that extend the cache normally render no card, so the card's presence is itself a signal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc5cf59e8f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (idx > 0 && typeof findPrevSameModel === 'function') { | ||
| const prevSame = findPrevSameModel(idx); | ||
| if (prevSame.idx >= 0) { | ||
| prevEntry = filtered[prevSame.idx]; |
There was a problem hiding this comment.
When a sidebar search, path filter, or tool filter hides the actual preceding turn, findPrevSameModel searches only filtered. A cold write that becomes the first visible result is therefore labeled “Initial prompt cache creation,” while later results may be compared with an older turn and falsely attributed to TTL or structural changes. Cache diagnostics should use the complete chronological entry chain so changing a UI filter cannot change the reported cause.
Useful? React with 👍 / 👎.
| if (diff.toolsChanged) { | ||
| return { reasonKey: 'cache_miss_tools', reasonText: t('cache_miss_tools'), lowConfidence: false }; |
There was a problem hiding this comment.
When an existing tool's description or input_schema changes without its name or the tool count changing, structuralDiff leaves toolsChanged false because it compares only names and counts. This branch consequently cannot emit cache_miss_tools for the schema-change scenario named by the diagnostic and instead falls through to an unknown or TTL explanation; compare the tool definitions as well as their names.
Useful? React with 👍 / 👎.
| const prevMsgs = getMessages(prevBody); | ||
| const curMsgs = getMessages(curBody); | ||
| if (diff.unchangedMsgs === 0 && (prevMsgs.length > 0 || curMsgs.length > 0)) { | ||
| return { reasonKey: 'cache_miss_history', reasonText: t('cache_miss_history'), lowConfidence: false }; |
There was a problem hiding this comment.
When histories share any leading message but a later message is modified or removed—for example, [A, B] becoming [A, C]—structuralDiff records the changed tail while leaving unchangedMsgs nonzero. This condition therefore misses the common history-modification and truncation cases advertised by cache_miss_history and falls through to a TTL or unknown explanation; use the removed/modified portions rather than requiring the histories to have no common prefix.
Useful? React with 👍 / 👎.
| function cacheTtlMs(usage) { | ||
| const detail = usage && usage.cache_creation; | ||
| if (detail && typeof detail === 'object') { | ||
| if (detail.ephemeral_1h_input_tokens) return 3600000; | ||
| if (detail.ephemeral_5m_input_tokens) return 300000; | ||
| } | ||
| return 300000; |
There was a problem hiding this comment.
When the predecessor only read an existing cache, or is a dashboard stub, its usage lacks the cache_creation tier breakdown and this unconditional fallback treats it as a five-minute cache. A valid one-hour cache can consequently be reported as expired for any gap between five and sixty minutes, even though this code explicitly supports that tier when metadata happens to be present; an absent tier should produce the unknown diagnosis rather than a confident five-minute expiry.
Useful? React with 👍 / 👎.
Viewer CSS and JS coverage are measured by sweeping the entries produced by _contract_cases(), and none of those cases reported a cold cache write. The diagnostic card therefore never rendered during the sweep, so all five of its new selectors counted as unexercised and viewer_css_diff came out at 0%. Add a cache_diagnostic_card case with three cold-write entries covering the initial, TTL-expired, and unidentifiable-cause paths. The last one renders the low-confidence variant, so .cache-diag-card.low-confidence is exercised too. _cache_diag_record moves above _contract_cases() because the parametrize decorator calls it at import time. viewer_css_diff: 0% -> 100% (5/5). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0060d0a1a4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const prevSame = findPrevSameModel(idx); | ||
| if (prevSame.idx >= 0) { | ||
| prevEntry = filtered[prevSame.idx]; |
There was a problem hiding this comment.
When a conversation switches models while retaining the same message prefix, findPrevSameModel can return the earlier model through its response-link, thread, or prefix strategies because those strategies do not check the model. Treating that result as a trusted cache predecessor causes the new model’s first cold write to be attributed to a system/tool change, TTL expiry, or an unknown rebuild instead of initial cache creation; validate that the selected predecessor uses the current model before diagnosing it.
Useful? React with 👍 / 👎.
| if (!prevEntry) { | ||
| return { reasonKey: 'cache_miss_initial', reasonText: t('cache_miss_initial'), lowConfidence: false }; |
There was a problem hiding this comment.
When the preceding same-thread request has zero cache reads and zero cache creation—for example, because the conversation was still below the provider’s minimum cacheable size—the next request can be the first one that actually creates a cache. Since this branch checks only whether a predecessor entry exists, that first creation is reported as an unknown rebuild or even TTL expiry rather than initial cache creation; treat predecessors without any cache activity as not having established a cache.
Useful? React with 👍 / 👎.
Anthropic writes a small delta to the cache on nearly every turn, so cache_creation > 0 was not the evidence the diagnosis treated it as: only a write with no accompanying read means the prefix was genuinely cold. Follow the prompt hash chain when attributing the miss. The prompt is hashed as an ordered chain -- tools, then system, then messages -- so cache_read == 0 proves the first segment missed and an edit in a later segment cannot be the cause. Compare only the region a breakpoint actually covers, so appending a message beyond the last breakpoint stays the normal path instead of being reported as an invalidating edit, and name the earliest changed segment rather than a later one that would send the reader looking in the wrong place. Read the TTL tier from the request's cache_control, falling back to the response's cache_creation tier, and return 0 when neither names one. The previous response-only read made the 5-minute default the normal path and turned unknown lifetimes into confident expiry claims. Pick the predecessor from the unfiltered history: the sidebar filters are a viewing choice and must not change what caused a miss. Candidates have to share the model and conversation and have taken part in caching themselves, and a shared session confirms a predecessor whose earlier messages were rewritten -- which the message-prefix test alone rejects. Sweep renderCacheDiagnostic while the full corpus is loaded in the coverage harness; live mode narrows entries to two websocket records before the detail loop, so no cache-bearing entry was ever rendered.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eef33bdacb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (prevScopes.tools && curScopes.tools) { | ||
| out.toolsChanged = JSON.stringify(normalizeCacheable(getRequestTools(prevBody))) | ||
| !== JSON.stringify(normalizeCacheable(getRequestTools(curBody))); |
There was a problem hiding this comment.
When cache_control is attached to an early tool, later tools are outside that breakpoint's cached prefix, but this comparison serializes the entire tool list. If the cache subsequently expires or is evicted while an uncached later tool also changes, the cold write is incorrectly attributed to modified tool schemas rather than TTL or an unknown cause. Preserve the tool breakpoint index and compare only through the last shared cached tool.
Useful? React with 👍 / 👎.
| if (a === undefined || b === undefined) { out.historyChanged = true; break; } | ||
| if (!msgContentEqual(a, b)) { out.historyChanged = true; break; } |
There was a problem hiding this comment.
When a cached message contains tool blocks, msgContentEqual ignores cache-relevant fields such as a tool_use block's id and a tool_result block's tool_use_id or is_error. Changing only one of those fields therefore leaves historyChanged false even though the request prefix changed, causing the diagnostic to fall through to TTL or unknown instead of reporting the history modification. Compare normalized full blocks while stripping only cache_control.
Useful? React with 👍 / 👎.
| // Only enforced when both sides are labelled; captures without session | ||
| // headers still get a best-effort predecessor. | ||
| if (session && cacheSessionKey(cand) && cacheSessionKey(cand) !== session) continue; |
There was a problem hiding this comment.
When the current turn has a session key but the nearest earlier cache-bearing turn has no session metadata, this condition accepts that unlabeled turn and immediately returns it, even if an older turn has the current session's exact key. In traces that interleave labeled and unlabeled conversations, the card consequently compares the wrong request body and timestamp and can confidently report an unrelated structural change or expiry; continue searching for an exact session match before using an unlabeled fallback.
Useful? React with 👍 / 👎.
| // No qualifying predecessor: nothing earlier shared this model and left a | ||
| // cache behind, so this turn is the one that created it. | ||
| if (!prevEntry) { | ||
| return { reasonKey: 'cache_miss_initial', reasonText: t('cache_miss_initial'), lowConfidence: false }; |
There was a problem hiding this comment.
When a trace begins with a resumed conversation or contains only a suffix of a session, a missing captured predecessor does not prove that this request created the cache for the first time. For example, the first request recorded after resuming an idle Claude session can be a cold write because its earlier cache expired, but this branch confidently labels it “Initial prompt cache creation.” Treat the capture boundary as unknown unless the trace contains evidence that the conversation itself starts here.
Useful? React with 👍 / 👎.
| const prev = findCachePredecessor(entry); | ||
| const diag = diagnoseCacheInvalidation(entry, prev.entry, prev.exact); |
There was a problem hiding this comment.
When the viewer uses TRACE_RECORDS_API without embedded raw lines, only the selected entry has been fetched; predecessor entries in entries remain stubs. This synchronous call passes that stub into the diagnosis, so structureAvailable is false and the stub lacks both request-side cache TTL controls and response-side tier details. Consequently dashboard users get the unknown diagnosis for tool, system, history, and TTL misses even though the predecessor payload is available from the records API; fetch or preload the predecessor before rendering the card.
Useful? React with 👍 / 👎.
Address review feedback on #436. Cache-miss attribution is removed from this change. It reads the prompt hash chain rather than token counts, which is a separate concern with its own failure modes, so it ships on its own in #437. Cost estimation: - Derive Anthropic cache rates from each model's input rate via the published multipliers instead of transcribing four numbers per model, so a repricing touches one value. - Return null for models absent from the pricing table rather than falling back to Sonnet rates. The silent fallback is what got the earlier cost feature removed in v0.1.5: a wrong number reads exactly like a right one. - Carry PRICING_AS_OF into the UI so a stale figure is visibly stale, and disclose when a session mixes priced and unpriced models. - Subtract cache reads from input_tokens only for gateways that report them nested, so an OpenAI-shaped Claude response is not billed twice. Bloat detection: - Collapse the size test into toolResultBloatInfo(block), with detectEntryToolBloat built on it, so the sidebar badge and the detail banner cannot disagree about what counts as oversized. - Show the largest result when a turn carries several, with the count in the tooltip. Drop the typeof-function guards around same-bundle calls; they were masking load-order bugs rather than tolerating them. Evidence screenshots are regenerated from a real two-turn Claude Code session (opus-5, 37 turns) whose Read of renderers.js produces a genuine 50.8KB tool result, replacing a fixture screenshot that showed neither the cost stats nor a bloat badge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 participant
Summary
A turn that misses the prompt cache shows
Cache Read 0alongside a largeCache Create, but the token row alone cannot say why. A cold start and a turn that idled past the 5-minute cache TTL produce an identical row, so telling them apart meant cross-checking turn timestamps by hand.This adds a diagnostic card above the detail sections that names the reason:
Validation
pytest tests/test_viewer_contracts.py tests/test_viewer_js_units.py— 38 passedpytest tests/— 1088 passed, 2 pre-existing local failures unrelated to this change (test_kimi_code_client_reverse_proxy,test_codex_zstd_request_body; both fail identically on a clean tree from missingtomllib/backports.zstdin the local interpreter)ruff check .andruff format --check .— cleanscripts/check_coverage.py— all 6 checks pass (viewer_js_diff100.00%, 10/10 changed JS functions covered)scripts/check_screenshots.py .agents/evidence/pr/— PASS=248 WARN=7 FAIL=0scripts/check_legibility.py— passedEvidence
Captured with Playwright against a real trace viewer HTML under
.traces/cache-invalidation-diagnostics/trace_cache_diagnostics.html, recorded from a live 138-turn Claude Code session. The System Prompt, Messages, Response, and Full JSON sections are collapsed in these frames: the card and token row sit above them, so nothing diagnostic is hidden.Turn 1, cold start —
Cache Read 0/Cache Create 35,115:Turn 3, after a 12:09 AM to 7:45 AM gap — same zero-read shape, different reason:
Turn 2, healthy extension —
Cache Read 34,905/Cache Create 718, no card:The two miss frames are the useful pair: both show
Cache Read 0, and the card text is the only thing distinguishing a cold start from a TTL expiry.🤖 Generated with Claude Code