Skip to content

feat(always-on): measure the injected prefix, don't census the disk - #28

Merged
pa-arth merged 2 commits into
mainfrom
feat/standing-config-measured
Aug 14, 2026
Merged

feat(always-on): measure the injected prefix, don't census the disk#28
pa-arth merged 2 commits into
mainfrom
feat/standing-config-measured

Conversation

@pa-arth

@pa-arth pa-arth commented Aug 13, 2026

Copy link
Copy Markdown
Owner

What

The always-on breakdown answered "what did you put in your context" by walking files under ~/.claude. The context is not assembled from disk — Claude Code assembles it and records it, itemised, as typed attachment rows ahead of the first assistant turn. cc-audit read none of them.

Measured over 501 local sessions, --json --since-days 30:

component before after
skill listing 967 tok (skillCount: 15) 5,683 tok (skillCount: 78)
hook output no field 948 tok
auto-memory folded into projectClaudeMdTokens 2,990 tok, its own field
fixed prefix no field 49,623 tok — 82% of the floor, and not yours to cut

Why it's a class, not four bugs

A disk census has no subscriber to a Claude Code release. Built-in skills ship inside the binary and are never on disk; the agent listing and deferred-tool delta move when the tool updates while nothing in ~/.claude does. It can't be wrong once and then fixed — it goes stale every release, silently.

It was also not stale in one direction, which the plan assumed:

  • settings.json > skillOverrides was never read, so six skills the user had switched off were sized and billed every turn.
  • readdirSync().isDirectory() is false for a symlinked directory, so four more were invisible.

Net under-reporting, per-skill over-reporting. "Scale the floor up" would not have fixed it — the membership set was wrong, not just the magnitude. The listing's own names[] is the authority now, and skillCarry[].loaded says so with a reason.

The disk census is retained and demoted: it attributes a measured block to your files; it no longer decides how big the block is. Plugin and user-command listings appear inside the injected skill listing, so they're slices of it, not addends to it — the old formula would have added them twice the moment the listing became measured.

Also

  • ~/.claude/commands/** is walked at all now. Six opsx:* commands load into every listing; nothing had ever counted them.
  • mcpDeferred is read from deferred_tools_delta in the sessions as they ran, not inferred from ENABLE_TOOL_SEARCH in the auditing process — a different process answering a different question.
  • Unknown is null + a named reason, never 0. A session that recorded no attachments doesn't vote a 0 into any median, and doesn't vote on deferral either. That second conflation was a bug I wrote and a test caught mid-implementation.
  • A negative reconciliation remainder fails and is reported per session. Clamping is how this defect class hides: the number stays plausible while the model underneath is wrong.

What I verified did NOT move

Run against origin/main at the same instant (the first attempt showed spurious drift purely because the two runs were 15 minutes apart and the trailing 30-day edge moved):

Zero drift across all 13 shared fields, plus spend.totalUsd and fluency.score identical to the last digit. Notably standingContextTokens (60,457 — already correct as of #27; the 40,487 in the plan was a pre-#27 figure) and pluginSkillTokens (a true zero: the walk runs, and the one enabled plugin ships no skills/ dir).

Tests

Fixtures are recorded from real transcripts by scripts/record-injected-fixture.mjs — structure, field names, string lengths and usage numbers preserved, content replaced with same-length filler. Never hand-authored: a fixture written from the design doc would have passed against a broken parser, because the doc had both the kind list and the per-kind content field wrong.

Four deliberate mutations were each caught by the specific test written for them: clamping fixedPrefix, sizing the JSON envelope, routing known-zero kinds to the residual, and letting an attachment-less session vote 0 into the medians.

402/403 pass. The one failure is pricingDrift (OpenAI rates vs LiteLLM), which fails identically on origin/main.

Breaking

Aggregate schemaVersion 9 → 10. skillDescriptionTokens is removed, not kept beside skillListingTokens — a field that keeps its name while changing meaning lets a downstream comparison silently mix two quantities, which is the defect class this PR closes.

⚠️ The backend reader is not in this PR yet. Per the spec, the contract and its reader ship together or neither ships — I'm tracking down the /v1/public/solo/data consumers now and will land that before this merges.

Spec: ~/repos/openspec/changes/standing-config-measured-not-censused/ (see findings-task-0.md — task 0 refuted four of this change's own premises before any code moved).

🤖 Generated with Claude Code

The always-on breakdown answered "what did you put in your context" by
walking files under ~/.claude. The context is not assembled from disk —
Claude Code assembles it and records it, itemised, as typed `attachment`
rows ahead of the first assistant turn, and none of those were read.

Measured over 501 local sessions, `--json --since-days 30`:

  skill listing   967 tok  ->  5,683 tok   (skillCount 15 -> 78)
  hook output     no field ->    948 tok
  auto-memory     folded   ->  2,990 tok   (projectClaudeMd falls to match)
  fixedPrefix     no field -> 49,623 tok   (82% of the floor, not yours to cut)

WHY THIS IS A CLASS, NOT FOUR BUGS. A disk census has no subscriber to a
Claude Code release: built-in skills ship inside the binary and are never
on disk, and the agent listing / deferred-tool delta move when the tool
updates while nothing in ~/.claude does. So it cannot be wrong once and
then fixed — it goes stale every release, silently.

It was also NOT stale in one direction, which the plan assumed. It never
read `settings.json > skillOverrides`, so six skills the user had switched
off were sized and billed every turn; and `readdirSync().isDirectory()` is
false for a symlinked directory, so four more were invisible. Net under,
per-skill over. "Scale the floor up" would not have fixed it — the
membership set was wrong, not just the magnitude. The listing's own
`names[]` is now the authority, and `skillCarry[].loaded` says so.

The disk census is retained and DEMOTED: it attributes a measured block to
the user's files, it no longer decides how big the block is. Plugin and
user-command listings appear INSIDE the injected skill listing, so they are
slices of it rather than addends to it — the old formula added them twice
the moment the listing became measured.

Also:
- `~/.claude/commands/**` is walked at all now. Six `opsx:*` commands load
  into every listing and nothing had ever counted them.
- `mcpDeferred` is read from `deferred_tools_delta` in the sessions AS THEY
  RAN, not inferred from `ENABLE_TOOL_SEARCH` in the auditing process.
- Unknown is `null` + a named reason, never 0. A session that recorded no
  attachments does not vote a 0 into any median, and does not vote on
  deferral either — that conflation was caught by a test mid-implementation.
- A negative reconciliation remainder FAILS and is reported per session.
  Clamping is how this defect class hides: the number stays plausible while
  the model underneath is wrong.

VERIFIED NOT TO MOVE, run against origin/main at the same instant:
standingContextTokens (60,457 — already correct as of #27, and the 40,487
in the plan was a pre-#27 figure), pluginSkillTokens (a true zero: the walk
runs, the enabled plugin ships no skills/ dir), spend.totalUsd, fluency.score.

Fixtures are RECORDED from real transcripts by
scripts/record-injected-fixture.mjs, never hand-authored — structure, field
names, string lengths and usage numbers preserved, content replaced with
same-length filler. A fixture written from the design doc would have passed
against a broken parser, because the doc had the kind list and the per-kind
content field wrong. All four deliberate mutations of this logic are caught
by the tests written for them.

Aggregate schemaVersion 9 -> 10. `skillDescriptionTokens` is REMOVED rather
than kept beside `skillListingTokens`: a field that keeps its name while
changing meaning lets a downstream comparison silently mix two quantities.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Aug 13, 2026

Copy link
Copy Markdown

Greptile Summary

The PR replaces disk-censused always-on estimates with measurements from first-turn transcript attachments and introduces aggregate schema version 10.

  • Parses and reconciles injected skill, hook, MCP, deferred-tool, agent, residual, and fixed-prefix components.
  • Retains disk inspection for attribution while using transcript listings as the loading authority.
  • Adds recorded transcript fixtures, nullable unknown-state reporting, and expanded local and aggregate reporting.

Confidence Score: 4/5

The resumed-transcript turn mismatch should be fixed before merging because it can silently inflate the fixed-prefix and standing-context measurements.

Attachment collection can stop at a replayed assistant row that is subsequently deduplicated, while aggregation reconciles those attachments against a later retained turn containing additional conversation history.

Files Needing Attention: src/adapters/claudeCode.ts and src/alwaysOn.ts

Important Files Changed

Filename Overview
src/adapters/claudeCode.ts Collects first-turn attachment and user-message measurements, but closes collection before resumed-message deduplication.
src/alwaysOn.ts Reworks always-on accounting around measured components; resumed transcripts can pair those components with a later retained turn.
src/injectedPrefix.ts Defines attachment-field sizing, component folding, residual handling, and negative-remainder reconciliation.
src/aggregate.ts Introduces schema version 10 with nullable measured-prefix fields and removes the old censused skill field.
src/report.ts Presents the expanded measured always-on breakdown and reconciliation information.
scripts/record-injected-fixture.mjs Records structurally faithful, length-preserving redacted transcript fixtures.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  T[Transcript rows] --> P[Parse pre-turn attachments]
  P --> C[Injected component counts]
  T --> U[First retained main-chain usage]
  C --> R[Per-session reconciliation]
  U --> R
  R --> M[Median always-on metrics]
  M --> A[Schema v10 aggregate and report]
Loading

Comments Outside Diff (1)

  1. src/alwaysOn.ts, line 428 (link)

    P1 Resumed prefix uses wrong turn

    When a resumed transcript's first assistant message is deduplicated and a later unseen turn is retained, firstMain supplies that later turn's history-inflated usage for attachments collected before the skipped turn, causing inflated fixed-prefix and standing-context costs while masking reconciliation mismatches.

    Fix in Claude Code Fix in Cursor

Fix all with Greploop

Fix All in Claude Code Fix All in Cursor

Reviews (1): Last reviewed commit: "feat(always-on): measure the injected pr..." | Re-trigger Greptile

…upload

v10 put the first sentence-shaped strings into the uploaded aggregate. Everything
before it was a model id or a hash, 25 chars at the longest, so the solo-capture
endpoint's 200-char per-string ceiling had no plausible way to bite. These
sentences are 82 and 118 chars — inside the bound, with the margin now moving every
time someone rewords one.

What a breach costs is why this is a test and not a comment: the server screens
every string leaf and 400s the WHOLE capture on a breach, and sendCapture() ignores
the response by design so telemetry can never break a local run. The failure mode
is therefore the entire record silently not arriving, for every user, from the
release that lengthened a sentence. Nothing downstream can detect it; the corpus
just thins.

Extracts the three literals into UNMEASURED_REASONS so they are enumerable, and
bounds them. The second test checks the call sites actually use the set — a bound
over a constant nobody references is worth nothing. Mutation-checked both ways:
lengthening a reason past the ceiling, and writing an inline literal at a call site.

MAX_UPLOADED_STRING_CHARS restates the server's constant rather than importing it;
cc-audit does not depend on the backend, and a copied number with a test behind it
beats a coupling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pa-arth
pa-arth merged commit adf1a5b into main Aug 14, 2026
0 of 2 checks passed
@pa-arth
pa-arth deleted the feat/standing-config-measured branch August 14, 2026 14:44
pa-arth added a commit that referenced this pull request Aug 15, 2026
…later turn (#30)

Review findings from #28 and #29, both real.

### 1. Resumed prefix used the wrong turn (#28, Greptile)

`injectedOpen` closes at the first main-chain assistant row — 34 lines BEFORE the
cross-file `seen` dedup — while `computeAlwaysOn` measures the prefix from
`firstMain`, the first RETAINED turn. On a resumed transcript those are different
rows: the closing row is a replay owned by an earlier file and gets dropped, so
`firstMain` lands on a later turn whose cacheRead has the whole replayed
conversation folded in. Turn-1-sized `attributed` against a much larger `measured`
INFLATES `fixedPrefixTokens`, and `reconcile()` cannot catch it because it only
fires on a NEGATIVE remainder. Inflation is the silent direction.

Closing before the dedup was deliberate and stays — it is what stops a whole
resumed session's attachments folding into "turn 1". The bug was the other half.

Fixed by recording which row closed the window and which row `firstMain` will
pick, and declining to measure the session when they differ — counted, not
silently dropped. `standingContextTokens` is untouched: a resumed turn really did
carry that much. Only the BREAKDOWN is unanswerable, and `unmeasured` now says so
with its own reason rather than borrowing the no-attachments one.

**ZERO INSTANCES IN THE AUTHORING CORPUS: 0 of 483 sessions with attachments, of
926.** Verified by same-instant runs of origin/main and this branch — 0 of 30
fields moved. So this fixes no number today; it closes a path that would have been
invisible when it did fire.

That also means the case cannot be recorded, and a suite running only real local
data would stay green with the guard deleted. It is built from RECORDED bytes
anyway: a resumed replay is "rows this parse has seen, then rows it has not", so
seeding `seen` from a prefix of a recorded transcript and re-parsing the whole of
it reproduces the shape exactly, with no invented rows.

Two recorder bugs surfaced doing that, both the fixture-encodes-an-impossible-shape
failure the recorder exists to prevent:
- `message.id` was filled with CONSTANT filler, so every assistant row shared an id
  and the parser merged them — a "two-turn" recording came back as one turn with one
  prefix. Identity keys are now a memoized bijection: distinctness and intentional
  sharing both survive, no real value is carried. The old comment claimed ids "key
  nothing across fixtures"; they key the dedup and the streamed-row merge.
- the recorder counted assistant ROWS, not turns. Now counts distinct message ids.

### 2. Corrected rates had no offline coverage (#29, Greptile)

`pricingDrift` degrades to pass when it cannot fetch — correct, and it means an
offline run proves nothing about the rows it just corrected. `pricingPinned.test.ts`
is the deterministic companion: drift asks whether the VENDOR moved, this asks
whether OUR table moved underneath us. Only rows with a reason to be pinned are in
it; pinning all 29 would train people to update the expectation to match the code.

### 3. A stale comment that would have caused the next mis-pricing

Writing that test failed it, on `gpt-5.5-pro`. The header claimed "the `pro` tiers
publish no cached-input rate" and are set to cachedInput == input. **False since
gpt-5.4-pro/5.5-pro**: both publish 3/M, the ordinary 10%, confirmed against
LiteLLM, and the table correctly carries it. A reader trusting that comment would
have "restored" them to equal-input and over-billed every cached token they read.
The real exception set is three rows and is now named with reasons rather than
derived from the word "pro" — gpt-5-pro and gpt-5.2-pro (genuinely no published
rate) and codex-mini-latest (genuinely 25%).

410/410 tests, typecheck and lint clean. Mutation-checked: reverting luna to the
launch tier, "fixing" gpt-5.5-pro to equal-input, and dropping the prefix-turn gate
each fail only their own test. The two-turn fixture asserts its own precondition
(turn 2's prefix 61,014 -> 61,863) so a future re-recording cannot quietly leave the
guard proving nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Greptile P2 on #30, valid — and the failure was one step worse than reported.
`tool_result.tool_use_id` names a `tool_use.id`, which `claudeCode.ts:270` reads to
fold is_error and the result timestamp back onto the issuing turn. `id` went
through the identity bijection; `tool_use_id` fell through to length-preserving
filler. Because it is LENGTH-preserving, the two real ids were the same length and
both results came out as the same string — so the fixture did not merely lose the
link, it pointed two distinct calls at one key. A tool_result naming a tool_use
that does not exist is a transcript Claude Code cannot emit, which is the exact
class of defect recording fixtures exists to prevent (and the second instance of it
in this PR).

Added to IDENTITY and the two-turn fixture re-recorded from the same transcript: 4
lines change, the recorded prefixes are byte-identical (61,014 -> 61,863), so the
resumed-transcript assertions rest on the same numbers as before.

Also asserted, across EVERY injected-prefix fixture rather than the one that
regressed: every tool_use_id resolves to a tool_use in the same file, and distinct
results name distinct calls. The second half is not redundant — containment alone
passes the moment filler collides with a real id, which is how this hid.

414/414 (the fixture join check runs once per fixture file), typecheck and lint clean. Checked the guard bites: the pre-fix fixture
fails it by name ("dangling tool_use_id redacted-fixture-content redac").

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pa-arth pa-arth mentioned this pull request Aug 15, 2026
pa-arth added a commit that referenced this pull request Aug 15, 2026
Four merged PRs since 0.7.0 (#27 #28 #29 #30), and two of them change numbers a
user has already read, which is what makes this a release rather than a merge.

MINOR, not patch: #28 changes what the always-on breakdown MEANS — measured off the
transcript instead of censused off disk — and takes the aggregate schemaVersion 9 ->
10, removing `skillDescriptionTokens` rather than keeping a field whose name outlived
its meaning.

THE ONE TO SHIP FOR: 0.7.0 prices gpt-5.6-luna 5x over and terra 1.25x over on every
Codex session it has audited since the vendor repriced (backend f4500a27, 2026-07-31,
two days after 0.7.0 went out). That was fixed on main on 2026-08-14 and reaches
nobody until this publishes — a merged pricing fix in an unreleased CLI is a fix that
does not exist for users.

Verified before tagging, not just built:
- 414/414 tests, typecheck and lint clean
- `npm run build:npm` -> 411.9kb single file, `--version` reports 0.8.0 (so the
  esbuild --define baked the bump, which is the thing a version bump can silently
  miss)
- ran the BUNDLE against the real local corpus, not just `--help`: schemaVersion 10,
  standingContextTokens 60,963 — within 0.4% of the 60,738 measured independently by
  differencing `claude -p` runs — `skillDescriptionTokens` absent from the payload as
  #28 intended, `unmeasured` empty.

Tag v0.8.0 after merge; .github/workflows/publish.yml guards tag == package.json and
publishes + attaches the standalone binaries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant