feat(always-on): measure the injected prefix, don't census the disk - #28
Conversation
The always-on breakdown answered "what did you put in your context" by walking files under ~/.claude. The context is not assembled from disk — Claude Code assembles it and records it, itemised, as typed `attachment` rows ahead of the first assistant turn, and none of those were read. Measured over 501 local sessions, `--json --since-days 30`: skill listing 967 tok -> 5,683 tok (skillCount 15 -> 78) hook output no field -> 948 tok auto-memory folded -> 2,990 tok (projectClaudeMd falls to match) fixedPrefix no field -> 49,623 tok (82% of the floor, not yours to cut) WHY THIS IS A CLASS, NOT FOUR BUGS. A disk census has no subscriber to a Claude Code release: built-in skills ship inside the binary and are never on disk, and the agent listing / deferred-tool delta move when the tool updates while nothing in ~/.claude does. So it cannot be wrong once and then fixed — it goes stale every release, silently. It was also NOT stale in one direction, which the plan assumed. It never read `settings.json > skillOverrides`, so six skills the user had switched off were sized and billed every turn; and `readdirSync().isDirectory()` is false for a symlinked directory, so four more were invisible. Net under, per-skill over. "Scale the floor up" would not have fixed it — the membership set was wrong, not just the magnitude. The listing's own `names[]` is now the authority, and `skillCarry[].loaded` says so. The disk census is retained and DEMOTED: it attributes a measured block to the user's files, it no longer decides how big the block is. Plugin and user-command listings appear INSIDE the injected skill listing, so they are slices of it rather than addends to it — the old formula added them twice the moment the listing became measured. Also: - `~/.claude/commands/**` is walked at all now. Six `opsx:*` commands load into every listing and nothing had ever counted them. - `mcpDeferred` is read from `deferred_tools_delta` in the sessions AS THEY RAN, not inferred from `ENABLE_TOOL_SEARCH` in the auditing process. - Unknown is `null` + a named reason, never 0. A session that recorded no attachments does not vote a 0 into any median, and does not vote on deferral either — that conflation was caught by a test mid-implementation. - A negative reconciliation remainder FAILS and is reported per session. Clamping is how this defect class hides: the number stays plausible while the model underneath is wrong. VERIFIED NOT TO MOVE, run against origin/main at the same instant: standingContextTokens (60,457 — already correct as of #27, and the 40,487 in the plan was a pre-#27 figure), pluginSkillTokens (a true zero: the walk runs, the enabled plugin ships no skills/ dir), spend.totalUsd, fluency.score. Fixtures are RECORDED from real transcripts by scripts/record-injected-fixture.mjs, never hand-authored — structure, field names, string lengths and usage numbers preserved, content replaced with same-length filler. A fixture written from the design doc would have passed against a broken parser, because the doc had the kind list and the per-kind content field wrong. All four deliberate mutations of this logic are caught by the tests written for them. Aggregate schemaVersion 9 -> 10. `skillDescriptionTokens` is REMOVED rather than kept beside `skillListingTokens`: a field that keeps its name while changing meaning lets a downstream comparison silently mix two quantities. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Greptile SummaryThe PR replaces disk-censused always-on estimates with measurements from first-turn transcript attachments and introduces aggregate schema version 10.
Confidence Score: 4/5The resumed-transcript turn mismatch should be fixed before merging because it can silently inflate the fixed-prefix and standing-context measurements. Attachment collection can stop at a replayed assistant row that is subsequently deduplicated, while aggregation reconciles those attachments against a later retained turn containing additional conversation history. Files Needing Attention: src/adapters/claudeCode.ts and src/alwaysOn.ts
|
| Filename | Overview |
|---|---|
| src/adapters/claudeCode.ts | Collects first-turn attachment and user-message measurements, but closes collection before resumed-message deduplication. |
| src/alwaysOn.ts | Reworks always-on accounting around measured components; resumed transcripts can pair those components with a later retained turn. |
| src/injectedPrefix.ts | Defines attachment-field sizing, component folding, residual handling, and negative-remainder reconciliation. |
| src/aggregate.ts | Introduces schema version 10 with nullable measured-prefix fields and removes the old censused skill field. |
| src/report.ts | Presents the expanded measured always-on breakdown and reconciliation information. |
| scripts/record-injected-fixture.mjs | Records structurally faithful, length-preserving redacted transcript fixtures. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
T[Transcript rows] --> P[Parse pre-turn attachments]
P --> C[Injected component counts]
T --> U[First retained main-chain usage]
C --> R[Per-session reconciliation]
U --> R
R --> M[Median always-on metrics]
M --> A[Schema v10 aggregate and report]
Comments Outside Diff (1)
-
src/alwaysOn.ts, line 428 (link)Resumed prefix uses wrong turn
When a resumed transcript's first assistant message is deduplicated and a later unseen turn is retained,
firstMainsupplies that later turn's history-inflated usage for attachments collected before the skipped turn, causing inflated fixed-prefix and standing-context costs while masking reconciliation mismatches.
Reviews (1): Last reviewed commit: "feat(always-on): measure the injected pr..." | Re-trigger Greptile
…upload v10 put the first sentence-shaped strings into the uploaded aggregate. Everything before it was a model id or a hash, 25 chars at the longest, so the solo-capture endpoint's 200-char per-string ceiling had no plausible way to bite. These sentences are 82 and 118 chars — inside the bound, with the margin now moving every time someone rewords one. What a breach costs is why this is a test and not a comment: the server screens every string leaf and 400s the WHOLE capture on a breach, and sendCapture() ignores the response by design so telemetry can never break a local run. The failure mode is therefore the entire record silently not arriving, for every user, from the release that lengthened a sentence. Nothing downstream can detect it; the corpus just thins. Extracts the three literals into UNMEASURED_REASONS so they are enumerable, and bounds them. The second test checks the call sites actually use the set — a bound over a constant nobody references is worth nothing. Mutation-checked both ways: lengthening a reason past the ceiling, and writing an inline literal at a call site. MAX_UPLOADED_STRING_CHARS restates the server's constant rather than importing it; cc-audit does not depend on the backend, and a copied number with a test behind it beats a coupling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…later turn (#30) Review findings from #28 and #29, both real. ### 1. Resumed prefix used the wrong turn (#28, Greptile) `injectedOpen` closes at the first main-chain assistant row — 34 lines BEFORE the cross-file `seen` dedup — while `computeAlwaysOn` measures the prefix from `firstMain`, the first RETAINED turn. On a resumed transcript those are different rows: the closing row is a replay owned by an earlier file and gets dropped, so `firstMain` lands on a later turn whose cacheRead has the whole replayed conversation folded in. Turn-1-sized `attributed` against a much larger `measured` INFLATES `fixedPrefixTokens`, and `reconcile()` cannot catch it because it only fires on a NEGATIVE remainder. Inflation is the silent direction. Closing before the dedup was deliberate and stays — it is what stops a whole resumed session's attachments folding into "turn 1". The bug was the other half. Fixed by recording which row closed the window and which row `firstMain` will pick, and declining to measure the session when they differ — counted, not silently dropped. `standingContextTokens` is untouched: a resumed turn really did carry that much. Only the BREAKDOWN is unanswerable, and `unmeasured` now says so with its own reason rather than borrowing the no-attachments one. **ZERO INSTANCES IN THE AUTHORING CORPUS: 0 of 483 sessions with attachments, of 926.** Verified by same-instant runs of origin/main and this branch — 0 of 30 fields moved. So this fixes no number today; it closes a path that would have been invisible when it did fire. That also means the case cannot be recorded, and a suite running only real local data would stay green with the guard deleted. It is built from RECORDED bytes anyway: a resumed replay is "rows this parse has seen, then rows it has not", so seeding `seen` from a prefix of a recorded transcript and re-parsing the whole of it reproduces the shape exactly, with no invented rows. Two recorder bugs surfaced doing that, both the fixture-encodes-an-impossible-shape failure the recorder exists to prevent: - `message.id` was filled with CONSTANT filler, so every assistant row shared an id and the parser merged them — a "two-turn" recording came back as one turn with one prefix. Identity keys are now a memoized bijection: distinctness and intentional sharing both survive, no real value is carried. The old comment claimed ids "key nothing across fixtures"; they key the dedup and the streamed-row merge. - the recorder counted assistant ROWS, not turns. Now counts distinct message ids. ### 2. Corrected rates had no offline coverage (#29, Greptile) `pricingDrift` degrades to pass when it cannot fetch — correct, and it means an offline run proves nothing about the rows it just corrected. `pricingPinned.test.ts` is the deterministic companion: drift asks whether the VENDOR moved, this asks whether OUR table moved underneath us. Only rows with a reason to be pinned are in it; pinning all 29 would train people to update the expectation to match the code. ### 3. A stale comment that would have caused the next mis-pricing Writing that test failed it, on `gpt-5.5-pro`. The header claimed "the `pro` tiers publish no cached-input rate" and are set to cachedInput == input. **False since gpt-5.4-pro/5.5-pro**: both publish 3/M, the ordinary 10%, confirmed against LiteLLM, and the table correctly carries it. A reader trusting that comment would have "restored" them to equal-input and over-billed every cached token they read. The real exception set is three rows and is now named with reasons rather than derived from the word "pro" — gpt-5-pro and gpt-5.2-pro (genuinely no published rate) and codex-mini-latest (genuinely 25%). 410/410 tests, typecheck and lint clean. Mutation-checked: reverting luna to the launch tier, "fixing" gpt-5.5-pro to equal-input, and dropping the prefix-turn gate each fail only their own test. The two-turn fixture asserts its own precondition (turn 2's prefix 61,014 -> 61,863) so a future re-recording cannot quietly leave the guard proving nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Greptile P2 on #30, valid — and the failure was one step worse than reported. `tool_result.tool_use_id` names a `tool_use.id`, which `claudeCode.ts:270` reads to fold is_error and the result timestamp back onto the issuing turn. `id` went through the identity bijection; `tool_use_id` fell through to length-preserving filler. Because it is LENGTH-preserving, the two real ids were the same length and both results came out as the same string — so the fixture did not merely lose the link, it pointed two distinct calls at one key. A tool_result naming a tool_use that does not exist is a transcript Claude Code cannot emit, which is the exact class of defect recording fixtures exists to prevent (and the second instance of it in this PR). Added to IDENTITY and the two-turn fixture re-recorded from the same transcript: 4 lines change, the recorded prefixes are byte-identical (61,014 -> 61,863), so the resumed-transcript assertions rest on the same numbers as before. Also asserted, across EVERY injected-prefix fixture rather than the one that regressed: every tool_use_id resolves to a tool_use in the same file, and distinct results name distinct calls. The second half is not redundant — containment alone passes the moment filler collides with a real id, which is how this hid. 414/414 (the fixture join check runs once per fixture file), typecheck and lint clean. Checked the guard bites: the pre-fix fixture fails it by name ("dangling tool_use_id redacted-fixture-content redac"). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four merged PRs since 0.7.0 (#27 #28 #29 #30), and two of them change numbers a user has already read, which is what makes this a release rather than a merge. MINOR, not patch: #28 changes what the always-on breakdown MEANS — measured off the transcript instead of censused off disk — and takes the aggregate schemaVersion 9 -> 10, removing `skillDescriptionTokens` rather than keeping a field whose name outlived its meaning. THE ONE TO SHIP FOR: 0.7.0 prices gpt-5.6-luna 5x over and terra 1.25x over on every Codex session it has audited since the vendor repriced (backend f4500a27, 2026-07-31, two days after 0.7.0 went out). That was fixed on main on 2026-08-14 and reaches nobody until this publishes — a merged pricing fix in an unreleased CLI is a fix that does not exist for users. Verified before tagging, not just built: - 414/414 tests, typecheck and lint clean - `npm run build:npm` -> 411.9kb single file, `--version` reports 0.8.0 (so the esbuild --define baked the bump, which is the thing a version bump can silently miss) - ran the BUNDLE against the real local corpus, not just `--help`: schemaVersion 10, standingContextTokens 60,963 — within 0.4% of the 60,738 measured independently by differencing `claude -p` runs — `skillDescriptionTokens` absent from the payload as #28 intended, `unmeasured` empty. Tag v0.8.0 after merge; .github/workflows/publish.yml guards tag == package.json and publishes + attaches the standalone binaries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What
The always-on breakdown answered "what did you put in your context" by walking files under
~/.claude. The context is not assembled from disk — Claude Code assembles it and records it, itemised, as typedattachmentrows ahead of the first assistant turn. cc-audit read none of them.Measured over 501 local sessions,
--json --since-days 30:skillCount: 15)skillCount: 78)projectClaudeMdTokensWhy it's a class, not four bugs
A disk census has no subscriber to a Claude Code release. Built-in skills ship inside the binary and are never on disk; the agent listing and deferred-tool delta move when the tool updates while nothing in
~/.claudedoes. It can't be wrong once and then fixed — it goes stale every release, silently.It was also not stale in one direction, which the plan assumed:
settings.json > skillOverrideswas never read, so six skills the user had switched off were sized and billed every turn.readdirSync().isDirectory()is false for a symlinked directory, so four more were invisible.Net under-reporting, per-skill over-reporting. "Scale the floor up" would not have fixed it — the membership set was wrong, not just the magnitude. The listing's own
names[]is the authority now, andskillCarry[].loadedsays so with a reason.The disk census is retained and demoted: it attributes a measured block to your files; it no longer decides how big the block is. Plugin and user-command listings appear inside the injected skill listing, so they're slices of it, not addends to it — the old formula would have added them twice the moment the listing became measured.
Also
~/.claude/commands/**is walked at all now. Sixopsx:*commands load into every listing; nothing had ever counted them.mcpDeferredis read fromdeferred_tools_deltain the sessions as they ran, not inferred fromENABLE_TOOL_SEARCHin the auditing process — a different process answering a different question.null+ a named reason, never0. A session that recorded no attachments doesn't vote a0into any median, and doesn't vote on deferral either. That second conflation was a bug I wrote and a test caught mid-implementation.What I verified did NOT move
Run against
origin/mainat the same instant (the first attempt showed spurious drift purely because the two runs were 15 minutes apart and the trailing 30-day edge moved):Zero drift across all 13 shared fields, plus
spend.totalUsdandfluency.scoreidentical to the last digit. NotablystandingContextTokens(60,457 — already correct as of #27; the 40,487 in the plan was a pre-#27 figure) andpluginSkillTokens(a true zero: the walk runs, and the one enabled plugin ships noskills/dir).Tests
Fixtures are recorded from real transcripts by
scripts/record-injected-fixture.mjs— structure, field names, string lengths and usage numbers preserved, content replaced with same-length filler. Never hand-authored: a fixture written from the design doc would have passed against a broken parser, because the doc had both the kind list and the per-kind content field wrong.Four deliberate mutations were each caught by the specific test written for them: clamping
fixedPrefix, sizing the JSON envelope, routing known-zero kinds to the residual, and letting an attachment-less session vote0into the medians.402/403pass. The one failure ispricingDrift(OpenAI rates vs LiteLLM), which fails identically onorigin/main.Breaking
Aggregate
schemaVersion9 → 10.skillDescriptionTokensis removed, not kept besideskillListingTokens— a field that keeps its name while changing meaning lets a downstream comparison silently mix two quantities, which is the defect class this PR closes./v1/public/solo/dataconsumers now and will land that before this merges.Spec:
~/repos/openspec/changes/standing-config-measured-not-censused/(seefindings-task-0.md— task 0 refuted four of this change's own premises before any code moved).🤖 Generated with Claude Code