Skip to content

Fix: Combine the accountant fixes for #906-#919 on the re-land - #927

Draft
boazdavid wants to merge 18 commits into
feat/reland-accountantfrom
integ/batch1
Draft

boazdavid wants to merge 18 commits into
feat/reland-accountantfrom
integ/batch1

Conversation

@boazdavid

Copy link
Copy Markdown
Collaborator

This PR stacks on the draft re-land #923. It must not merge unless the repo owner says yes to the re-land. Savings figures change.

Summary

What the merge changed

  • dash/accountant_switch_test.go: both branches added a test at the same place. Both tests stay.
  • dash/schema.go, store.go, query.go, event.go, accountant_price.go: gates adds acc_priced, recording adds estimated_saved_usd. Both columns stay, in this order, everywhere.
  • Git merged the INSERT placeholder row without a conflict. It had 91 placeholders for 92 columns, so every insert failed at run time and 48 proxy tests failed. Fixed in the last merge commit. A reviewer should check the column, placeholder and argument counts again.

How it was tested

  • go test ./dash/... passes.
  • go test ./proxy/... passes except TestALateFirstByteOnTheRealTransportNeverReachesTheClient. That test also fails on the re-land base 189c169d. It expects an answer within 5 ms and this machine is loaded.
  • apply, keepalive, components, tooldefer pass.

Known limits

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

…ws (#911)

The B2 double-count fix dropped a whole request from DeclFilterSavings when the
accountant priced its toolfilter removal. That also dropped Requests, Reads and
Since, so ToolFilterDocFor lost its Realized block and the item list in
adaptToolsSaved could sum to more than the total.

Now the row stays in the scan. An EXISTS flag column marks it and
accumulateDeclFilterRow skips only the USD. carriedCost does the same for the
items, so items and total agree.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The default tail breakpoint (#795) adds a cache_control mark to the last
message after the pipeline ran. A string content also becomes one text
block. The message gets longer on the wire, and no operation reported
the extra bytes. The sum check failed on every request with a trailing
system message (section=messages ops=0 bodies=-63 bytes), so every
request fell back with check_sum and booked no saving.

apply now appends one rewrite operation for the last message, in wire
bytes, when markTail places a mark. Its ChangeID comes from the message
before the mark, so it is the same on every request. The operator
tailbreakpoint is an envelope operator in dash, so it gets its own row.

No accountant check or plausibility rule is weakened. The check now has
the operation it was missing.

Harness (FAKE_TOKDIV=8, tailx, 7 requests): net_saved_usd 0 -> 0.0817.
Truth (control cost - tail on cost): 0.0736. Fallbacks 7 -> 0.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…ted (#914)

A request the provider answered with a non-2xx status billed nothing: cost 0 and baseline 0.
Its saved_gross and saved_unique were still booked, so tokens saved disagreed with dollars
saved. In the run_m4 repro (turn 1 returns 200, turn 2 returns 400), /api/stats showed
saved_gross 28657 and saved_unique 19105 for 2 requests.

finish() now calls Event.DropUnbilledSavings. On a status that is not 2xx it sets tokens_after
to tokens_before, and zeroes saved_unique, attempted_tokens, and saved_gross, saved_unique and
acted on every component row. Status 0 (never reached a provider, such as /compact) and any
2xx (billed, even without usage) are untouched. attempted_tokens goes to 0 so Overview's
saved_gross_attempted / attempted_tokens ratio keeps one row population on both sides. No
separate attempted counter is added: attempted_tokens already holds what was eligible, and no
reader needs the tokens of a request that did not bill.

After: saved_gross 9553 and saved_unique 9553 for the same 2 requests (turn 1 only); the 400 row
reads tokens_before 19243, tokens_after 19243, saved_unique 0. Dollars do not change.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…y bound (#907)

PriceMode compared the tokens the library saved with e.TokensBefore.
TokensBefore counts message text only. Removed tool declarations are not
in it. A 121-token conversation that drops 2,188 tokens of declarations
looked like a request that removed more than it had, so the event fell
back with implausible_size and lost every other component's saving too.

countedSavedTokens now skips operations of the operators that act on the
tools section (toolfilter, toolschema, expand, adjudicate). The bound is
unchanged for every operation on messages. The accountant checks are not
touched. The bound cannot be made against the full request size here,
because the event has no count of tool and system tokens.

Harness (FAKE_TOKDIV=7, MSGMULT=30, tf2, 8 requests): fallbacks
8 implausible_size -> 0. net_saved_usd 0 -> 0.0554. Truth (control cost -
treatment cost): 0.0564.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…TTL by the billed tier (#912, #919)

kaSavedGate needed a request_components row with priced_by='accountant' to
accept a rescue the ping only wrote. A keep-alive-only setup stores no
component rows, so the rescue was zeroed again.

The request row now stores acc_priced (additive column, default 0), set when the
accountant override applied. The gate trusts keepalive_saved_usd of such a row
as the library's own decision. It is a plain column read placed after the
credited-row guard, so the probe order is unchanged. Rows with acc_priced = 0
keep the SQL gate.

The entry TTL given to the library was 1h when every breakpoint asked for it,
before the billed tier was known. accBilledTTL now lowers the TTL (and the
breakpoint TTLs) to 5m after the usage arrives, when the response billed a
5m write, before Compute runs.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…908)

Tool deferral runs after the pipeline reported its operations. It marks
declarations defer_loading, adds the search tool, and puts a names list
at messages[1]: appended to the system message there, or as a new
system message. The tools half is found by the envelope reporter. The
names list was reported by nobody, so the sum check failed on the
messages section (ops=0 bodies=-308 bytes) and conservation failed
with it, on every request of a deferred thread.

proxy now diffs the body before and after applyToolDefer and reports
the change as a tooldefer operation: a one-message splice for a new
message, a rewrite in place for an appended list. tooldefer is an
envelope operator in dash, so it gets its own row. No accountant check
is weakened.

Harness (FAKE_TOKDIV=4, MSGMULT=1500, td, 8 requests): check_sum
fallbacks 8 -> 0, failed checks 8+7 -> 0. The Overview still does not
credit the deferred declarations: the library sees bytes, and a
declaration marked defer_loading keeps its bytes. net_saved_usd is
-0.0006 (the names list and search tool as cost), truth is +0.0185.
/api/tool-defer still reads 0.0412. Making the two agree needs a
product decision, so it is not part of this change.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The first request on a thread the proxy has no history for (a restart with a missing or stale
state file, or a session begun before the proxy) is priced by the library as if the provider
would have written every removed token. If the cache was in fact warm, that is about 2x too high.
It was flagged estimated, but still went into baseline_cost_usd, net_saved_usd and total_saved_usd.
In run_m5, request 1 booked baseline 0.0424785 (saving 0.042015); the hand figure is 0.021569.

Choice: option (a), not (b). The proxy cannot price the unseen prefix at the read tier because the
size of that prefix is the very thing the missing history held. The usage only shows how far the
compacted body matched, not how much the client had cached. A hand figure needs that size, and the
proxy does not have it. So the proxy books no saving on an estimated request, and keeps the figure.

Changes:
- PriceMode: on an estimated request baseline_cost_usd equals cost_usd and every component
  saved_usd, saved_first_usd and saved_replay_usd is 0. The held figure goes to the new
  requests.estimated_saved_usd (Event.EstimatedSavedUSD, additive column, 0 on old rows). The
  keep-alive rescue and the split saving do not depend on a break and stay booked.
- Overview and the lean path sum it as estimated_saved_usd, beside estimated_requests. It is in
  no total, so net saved = total saved = sum of components = keep-alive tab, with no filter.
- UI: the Preset card says how many requests are not counted and shows the held figure. The
  request detail shows it on its own line marked with an approximate sign.
- docs/dashboard.md describes the stored field and the rule.

Numbers (run_m5, same binary and harness): net_saved_usd 0.0911627 before, 0.0491477 after.
Request 1 baseline 0.0424785 before, 0.0004635 (equal to cost) after, with 0.042015 held in
estimated_saved_usd. Requests 2 and 3 are unchanged (0.0243054, 0.0257421).

Golden files: stats.json, stats_lean.json and stats_windowed.json gain estimated_saved_usd: 0.
No golden fixture holds an estimated row, so no other number moves.

Test changes: TestSavingsRecordEndsWithTheInjectedForwardedBody and TestDashboardBooksTheAccountantSaving
sent the first request of a thread and read its booked saving. They now assert the estimate is held
out; the second one sends a second request and asserts the booked saving there.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The only conflict is in dash/accountant_switch_test.go. Both branches
add a test at the same place. Keep both tests and close each function.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…ches

Both branches add a column next to price_estimated: acc_priced (gates)
and estimated_saved_usd (recording). Keep both, in this order, in the
schema, the migration list, the INSERT columns, the INSERT values and
the SELECT and Scan lists.

Git merged the placeholder row of the INSERT without a conflict, so it
had 91 placeholders for 92 columns. Every insert failed at run time and
48 proxy tests failed. Add the missing placeholder.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The key rounded since and until down to 30 s, so a narrower window got
the wider window's cached totals. The key now holds the exact values.
The UI rounds its own now to 30 s so relative ranges still share a key.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…past nested cache marks (#917)

Wrap the snapshot in a proxy version and refuse another one. Sync before
the rename, skip an unchanged write and remove stale temp files at start.
Strip cache_control at any depth before hashing a body element.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The keeper's ping row and PingSavings now recover and log, so a library
panic cannot end the process. accBegin recovers and drops its record, so
a panic on a hostile body no longer reaches the pre-forward recover.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…e map (#920)

A realistic first turn books its cost when the price is known. The
reported rows had no price: the price map loads in the background and
the first requests arrived before it finished. No code change.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>

@boazdavid boazdavid left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer Kind Started by Context Review Round Head Verdict
Reviewer 1 (glm, user of @boazdavid) GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 offload-review skill, @boazdavid's session fresh 1 of 2 1 of 5 850893277db4 blocking

Verdict: blocking

Findings

  1. dash/toolsuggest.go:199 — the filter saving of an estimated request appears in no dollar total

    • What fails: a request the accountant priced as an estimate (the first request of every thread) books no saving anywhere (#915: BaselineCostUSD == CostUSD, every component SavedUSD 0), but its toolfilter row still carries priced_by = 'accountant' (dash/accountant_price.go:331 sets it before the estimated branch zeroes the dollars). declFilterInBaseline then matches that row, so accumulateDeclFilterRow returns at dash/toolsuggest.go:253 and the measured credit adds no dollars. The comment and docs/dashboard.md:125 justify this with "the net saving has them" — for an estimated row the net saving has nothing.
    • Proof: trace an estimated request with filtered_decl_tokens > 0. Accountant side: dash/accountant_price.go:335-349 holds fig.DeltaUSD in EstimatedSavedUSD, outside every total. Measured side: dash/toolsuggest.go:199-200 selects the row as in-baseline, dash/toolsuggest.go:253 skips the USD. Result: the Realized block and /api/adapt/tools count the request and its reads but show $0 for it. The same drop applies to the carriedCost path (dash/adapttools.go:473-479). Every thread's first request is estimated, so this is systematic, not an edge. The new test TestFilterCreditKeepsFactsAndSkipsDollarsForAccountantPricedRows covers only non-estimated rows.
    • Fix: make declFilterInBaseline false for a row whose request is estimated (join r.price_estimated), so the measured credit keeps the measured dollars for exactly those rows the books hold out. Or hold the filter part out of EstimatedSavedUSD and say so in the docs. Pick one owner for the figure; today it has none.
  2. dash/ui/app.js:1828 — the Overview says estimated requests are "not counted above", but rows written before this change are counted

    • What fails: price_estimated changed meaning in this PR. Before, an estimated row booked its figure (BaselineCostUSD = CostUSD + DeltaUSD, base commit 189c169, dash/accountant_price.go). After, an estimated row books nothing and the UI text reads "N of M requests are not counted above ... Estimated saving, not in any total: $X". No migration unbooks the old rows, so a database that has both shows a false claim: the old rows are in NetSavedUSD and TotalSavedUSD, and the card says they are not. The request detail page (dash/ui/app.js:3710-3714) shows the same contradiction: "Not counted ... no saving is booked" beside a non-zero net saving that is in the totals, and "Estimated saving (not counted): ≈ $0.00" for a row whose estimate was booked.
    • Proof: git show 189c169:dash/accountant_price.go — the base accOverride ends with e.BaselineCostUSD = e.CostUSD + fig.DeltaUSD; e.PriceEstimated = estimated. Overview.EstimatedRequests is SUM(r.price_estimated) (dash/overview.go:681), so those rows are counted, and NetSavedUSD sums their booked baseline_cost_usd. The docs answer this with "A row written before the column existed reads as exact" (docs/dashboard.md:161) — false: price_estimated is a pre-existing column and those rows read as estimated. Exposure is bounded: the accountant line is not in any tag (newest tag v0.3.4, 2026-10-01, predates #768), so only databases written by the unreleased branch are affected. That is still the class of row this repo elsewhere handles on purpose (acc_priced defaults 0, priced_by empty, "rows written before the switch keep their legacy figures").
    • Fix: distinguish the two meanings. Count only rows where the hold-out applied (for example price_estimated = 1 AND estimated_saved_usd > 0, or a new marker), or migrate the old rows. Then fix the docs sentence at docs/dashboard.md:161.
  3. dash/event.go:809 — a rejected request that still reported usage books dollars beside zeroed tokens

    • What fails: DropUnbilledSavings zeroes the token saving of every non-2xx row, on the ground that "the provider billed none of it, so tokens and dollars agree" (docs/dashboard.md:166-170). But finish runs the accountant and PriceMode whenever usageOK is true, with no status test (proxy/dashcapture.go:733-745), and the response path parses usage from the body whatever the status (proxy/proxy.go:2308). A gateway that returns a 4xx/429 body with a usage object then produces a row with tokens_after == tokens_before, zero SavedUnique, but a non-zero BaselineCostUSD - CostUSD and component SavedUSD from the accountant figure. Tokens and dollars disagree — the exact defect #914 set out to remove.
    • Proof: DropUnbilledSavings (dash/event.go:809-818) tests only the status; c.accReq != nil && usageOK && !c.accDone (proxy/dashcapture.go:733) tests none; responseUsageWhy at proxy/proxy.go:2308 runs on respBody of any status. The new test TestFinishBooksNoSavedTokensOnARejectedRequest covers only rejected rows with usageOK false, so the gap is untested.
    • Fix: skip the accountant compute (or force the fallback) when e.Status is not 0 and not 2xx, or extend DropUnbilledSavings to zero the dollar figures too.

Non-blocking

  • dash/adapttools.go:473-479 — the inBaseline branch never increments out.pricedRequests, so st.Priced = scan.pricedRequests > 0 (dash/toolapi.go:675) reports "not priced" when every in-scope filtered row is accountant-priced. A deliberate $0 then reads as "no measurement". Consider counting the row as priced with a known rate.
  • dash/accountant_price.go:335-350 — the estimated hold-out also zeroes the headttl row (premium and rescue). The comment names only the keep-alive rescue and the split as staying. The first request of a thread is where a 1h head is written, so the premium cost shows as $0 exactly there. If this is intended, say so in the comment and docs; if not, keep the ttl legs booked as their own column.
  • dash/event.go:866 — Price resets AccountantPriced but not EstimatedSavedUSD. No production caller mixes PriceMode and Price today (only costOnlyCopy in tests), so this is latent. Reset both for symmetry.
  • proxy/accountant_envelope.go:291 — messageOps returns nil for any change it cannot explain, and the sum check then sends the event to the no-saving fallback. Fail-open, documented. Acceptable.

What I checked

  • The full diff 189c169..8508932 (33 files) and the code around every hunk.
  • The tail-breakpoint operation: markTail runs in a defer in apply.BodyOpts (apply/apply.go:409-422) after wireSizes chained the pipeline ops, so tailOp's SizeBefore meets the chain's end on the same message. The OpsDropped == 0 guard, the stable ChangeID, and the append-after-pipeline ordering are consistent with the library's sum check.
  • messageOps and the tooldefer wiring: the full-slice append in proxy/proxy.go:1596 avoids aliasing; the insertion and append forms match the wireSizes splice conventions; ops for messages, tools and system cover the client-to-forwarded diff in order.
  • PriceMode/accOverride: the estimated hold-out, AccountantPriced, the plausibility bound with toolSectionOperators (operator strings match envelopeOwner's output), and the storage round-trip (insertBatch columns, scanRequest, additiveColumns migration, golden files).
  • The keep-alive gate: acc_priced = 1 sits after the col > 0 guard and before the probes, so the credited-row short-circuit order holds; every kaSaved/kaSavedSince caller aliases requests; manual pings book manual_saved_usd, not keepalive_saved_usd, so the gate does not touch them.
  • accBilledTTL: the CacheWrite1h < CacheWrite test matches Usage's documented semantics (usage.go:31) and keepalive.ResolveTier's rule; the breakpoint copy protects the slice the sibling store keeps; it runs before computeSafe on the only compute path (rounds <= 1).
  • DropUnbilledSavings ordering in finish (status is set in the event literal before it runs), and the status-0 and 2xx exclusions.
  • DeclFilterSavings, DeclFilterSavingsByTenant, carriedCost: the inBaseline SELECT column, the skipUSD return after the tier-token facts, and byID initialization for every key.
  • Release containment: the accountant line (#768 and its fixes) is in no tag; newest tag v0.3.4 predates it. This bounds finding 2 to databases written by the unreleased branch.
  • Commit list (9 fix commits, 3 merges) read in full.

@boazdavid boazdavid left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer Kind Started by Context Review Round Head Verdict
Reviewer 2 (glm, user of @boazdavid) GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 offload-review skill, @boazdavid's session fresh 2 of 2 1 of 5 850893277db4 blocking

Verdict: blocking

Findings

  1. dash/event.go:809 — a rejected request that still parsed usage books dollar savings, so tokens and dollars disagree.
    • What fails: DropUnbilledSavings zeroes the token side only (TokensAfter, SavedUnique, per-row SavedGross). It does not stop the accountant path. In finish (proxy/dashcapture.go:733-750) the order is: DropUnbilledSavings, then computeSafe (it runs, because it is gated on usageOK, not on status), then PriceMode. If a provider or gateway returns a non-2xx status whose body still parses usage (responseUsageWhy at proxy/proxy.go:2307 parses the body with no status check, and usageOK is sticky), then: tokens book 0, but accOverride applies, sets each component SavedUSD from the library figure and BaselineCostUSD = CostUSD + DeltaUSD. The row then shows $saved_usd > 0 beside saved_gross = 0. This is the exact disagreement the PR says cannot happen: docs/dashboard.md says "The provider billed none of it, so tokens and dollars agree", and the code comment at dash/event.go:804 says "cost and baseline are 0", which is false when usage parsed. The ping path already guards this (proxy/accountant_ping.go:106 refuses a non-2xx ping); the request path does not.
    • Proof: trace above; no line between DropUnbilledSavings (proxy/dashcapture.go:529) and PriceMode (proxy/dashcapture.go:750) reads c.status. accOverride (dash/accountant_price.go:250) reads no status either, and DropUnbilledSavings setting Acted = false removes the FallbackActedWithoutFigure gate, which makes the override easier, not harder.
    • Fix: gate the accountant compute (or the override) on a 2xx status, as PingSavings does. Or zero the dollar side in DropUnbilledSavings too. Reachability is low (an error body that carries usage), so the team may accept it — but then the docs and the code comment must say so.

Non-blocking

  • dash/accountant_price.go:335 — the estimate hold is all-or-nothing. A first request whose net figure is a cost (the tooldefer names list, an expand or adjudicate injection) now books 0 instead of that cost, so net saved is overstated by it. The docs and the issue speak only of savings that are too high. The held figure can be negative and renders as "Estimated saving (not counted): ≈ $-x". The cost side is measured, not estimate-dependent. Small figures; deliberate and test-covered (proxy/accountant_inject_test.go). Name it in the docs, or hold out only the removal half.
  • proxy/accountant_envelope.go:291 — the comment on messageOps says "Any other change returns nil". The equal-length case returns one op per changed message, so a two-message change is explained, not refused. That is safe (it explains more), but the comment overstates the refusal. Fix the comment.
  • dash/event.go:866 — Price resets PriceEstimated and AccountantPriced but not EstimatedSavedUSD. No production path calls Price after PriceMode today (pings and the import path build fresh events), but a future caller could store a held figure on a row with price_estimated = 0. Reset it there too, for symmetry with PriceMode.

What I checked

  • The full diff 189c169d..HEAD (9 fix commits, 3 merges), every changed file, and the code around each change: apply/apply.go, apply/ops.go, proxy/proxy.go, proxy/accountant_wire.go, proxy/accountant_envelope.go, proxy/dashcapture.go, dash/accountant_price.go, dash/event.go, dash/keepalive.go, dash/adapttools.go, dash/toolsuggest.go, dash/overview.go, dash/overview_lean.go, dash/query.go, dash/schema.go, dash/store.go, dash/ui/app.js, both docs files, all new and changed tests.
  • The accountant library at the pinned version (v0.1.1-0.20261104203351-5fa19864db4c, read from the module cache): how MessageSplice positions are undone (compute.go undo/apply), when penalty_unknown is set (PhaseUnknown, whole result), what isChange keeps, and the ended-change rules. The tail op and the tooldefer op chain correctly with pipeline ops, and the ChangeID schemes (hash of before-bytes; position for an insert) match the library's replace-an-active-change semantics.
  • The insert SQL: 92 columns, 92 placeholders, 92 arguments; the new columns sit at the same place in all three lists.
  • tooldefer/rewrite.go placeList: it produces exactly the two shapes messageOps explains (append to a system message at index 1, or insert one at index 1, including the one-message append). Anything else makes messageOps return nil, the sum check fails, and the event falls back — the safe direction.
  • accBilledTTL against entryFacts and the library's TTL use: it only lowers 1h to 5m, only with an exact lifetime, only when a write was billed below the 1h tier, and it copies the breakpoints slice before mutating. SharedWarm is documented as judged with the asked tier.
  • The keep-alive gate change: every kaSaved/kaSavedSince caller aliases requests as r., the acc_priced column is in both the CREATE TABLE and additiveColumns, and the credited-row guard stays first. When the override applies, keepalive_saved_usd is always the library's figure (accOverride sets it), so trusting acc_priced = 1 is sound. Ping events use Price, never PriceMode, so they never carry the marker.
  • The plausibility-bound exclusion (toolSectionOperators): context-guru.expand and context-guru.adjudicate operators are emitted only by the envelope reporter for tools-section declarations; pipeline message components use their own names, so no message operation escapes the bound.
  • #911 read-side changes: DeclFilterSavings and the carriedCost filter scan skip dollars for accountant-priced rows and keep requests, tokens, dates and Since (set before the skip). The filter scan's pricedRequests is not consumed by any reader that could misread a $0.
  • Fail-open: no change touches the forwarded body; all changes are reporting and pricing. A wrong or missing operation falls back to no saving, never to a wrong forward.
  • Test coverage: each of the six issues (#906, #907, #908, #911, #912/#919, #914, #915) has a unit test and, where it needs the wire, an end-to-end test. I did not run them (live tests not approved); I read them and their helpers.

A 4xx or 5xx body can carry a usage object. The accountant then priced a
saving beside the zeroed tokens. Skip the compute for a status that is not
0 and not 2xx, as PingSavings does. The row falls back with no_result.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…old rows as held

- The measured filter credit owns the filter dollars of a request whose
  figure was held out (declFilterInBaseline is false when estimate_held).
- New column requests.estimate_held marks the hold-out. Rows from before
  keep their legacy meaning and are not counted as not counted.
- Count a baseline-priced filtered row as priced in adapttools.
- Price resets EstimatedSavedUSD and EstimateHeld.
- Name what the hold-out covers (ttl legs, negative figure) in code, UI, docs.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
@boazdavid

Copy link
Copy Markdown
Collaborator Author

Round 1 fixes are pushed to integ/batch1: dcd1b31, 359a770, 0a397f9.

Review 1, finding 1 (estimated filter saving in no total): fixed in 0a397f9. Owner: the measured filter credit. declFilterInBaseline is now false when estimate_held = 1, so the credit keeps its dollars for exactly the rows the books hold out, and EstimatedSavedUSD stays out of every total. Reason: the credit is measured from the rate card and the tier and does not depend on the unknown cache state. Test: TestFilterCreditKeepsDollarsForAHeldEstimatedRequest (Realized, /api/adapt/tools, Overview baseline delta 0). The non-blocking pricedRequests point is fixed: a baseline-priced filtered row now counts as priced. docs/dashboard.md updated.

Review 1, finding 2 (price_estimated changed meaning): fixed in 0a397f9. New column requests.estimate_held (default 0), set only when the hold-out applied. The Overview count (full and lean), the Overview text and the request detail read it. A row written before #915 has estimate_held = 0, keeps its booked figure and is not "not counted". The docs sentence is replaced. Test: TestOverviewCountsEstimatedRequests now holds an old-meaning row and new-meaning rows.

Review 1 finding 3 and Review 2 finding 1 (rejected request with usage books dollars): fixed in dcd1b31. The accountant compute is skipped for a status that is not 0 and not 2xx (acc_fallback reason rejected_status). Test TestFinishBooksNoDollarsOnARejectedRequestThatReportedUsage sends a 429 and a 400 with a usage object: tokens and dollars are both 0. A 200 control books a saving, and the test fails without the fix.

Non-blocking: the hold-out covers the head-TTL legs; the comment and the docs now say so, and say that the keep-alive rescue and split stay booked. A negative held figure renders as "a cost of $x" and the docs name it. Price now resets EstimatedSavedUSD and EstimateHeld. The messageOps comment is corrected in 359a770.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

@boazdavid

Copy link
Copy Markdown
Collaborator Author

Round 1 replies. All findings are fixed in 0a397f9d and later. Details are in the earlier summary comment. Head is now 0206b444: it also merges #928 (#916, #917, #918, #920).

Reviewer 1, finding 1 (filter saving of an estimated request in no total). Fixed in 0a397f9d. The measured filter credit owns the figure. declFilterInBaseline is now (r.estimate_held = 0 AND EXISTS ...). A held request adds measured dollars to Realized and /api/adapt/tools. The totals still hold it out, so nothing is counted twice. Test: TestFilterCreditKeepsDollarsForAHeldEstimatedRequest. The non-blocking pricedRequests item is fixed in the same commit.

Reviewer 1, finding 2 (price_estimated changed meaning). Fixed in 0a397f9d. New column requests.estimate_held, default 0, set only when the hold-out applied. The Overview and the request detail read it. Old rows keep their legacy meaning. The docs sentence "reads as exact" is replaced. Test: TestOverviewCountsEstimatedRequests has an old-meaning row beside new-meaning rows. The INSERT has 93 columns and 93 placeholders, counted by script after the merge with #928.

Reviewers 1 and 2, rejected request with usage. Fixed in dcd1b31f. The accountant compute is skipped when the status is not 0 and not 2xx. The row falls back with reason rejected_status and books tokens 0, baseline equal to cost, and every component saving 0. Test: TestFinishBooksNoDollarsOnARejectedRequestThatReportedUsage (a 429 and a 400 with a usage object; a 200 control books a saving). It fails when the fix is off.

Non-blocking, all done.

  • The TTL legs are held out with the rest of the estimate. The code comment and the docs say so. The keep-alive rescue and the split saving stay booked.
  • A negative held figure is a cost. The UI says "a cost of $x" and the docs say so.
  • Price and PriceMode reset EstimatedSavedUSD and EstimateHeld.
  • The messageOps comment is fixed (359a770d).

Known limits, stated plainly.

  • The first request of a thread books 0 where the true saving is about 0.0216 (context-guru#915). The held figure is not corrected.
  • With the default tail breakpoint the Overview reports 0.0817 against a true 0.0736 (+11%). The cause is not found.
  • Tool defer (Accountant integration: tool defer fails check_sum and conservation on every request #908) stops the fallbacks but does not credit the deferral.
  • TestALateFirstByteOnTheRealTransportNeverReachesTheClient is a 5 ms timing test. It fails on the untouched base 189c169d on a loaded machine.

@boazdavid boazdavid left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer Kind Started by Context Review Round Head Verdict
Reviewer 2 (glm, user of @boazdavid) GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 offload-review skill, @boazdavid's session resumed 2 of 2 2 of 5 0206b444ccbc approve

Verdict: approve

Findings

None. Every round-1 finding is fixed and verified in the code. The new commits (#916, #917, #918, #920 and the round-1 fixes) contain no defect I could confirm.

Earlier findings, re-checked

  • My finding 1 (rejected request with usage books dollars) — fixed in dcd1b31f. capture.rejected() (proxy/dashcapture.go:880) returns c.status != 0 && (c.status < 200 || c.status > 299). This matches DropUnbilledSavings exactly. finish now skips computeSafe and PriceMode for such a row and sets accNoResult = "rejected_status" (proxy/dashcapture.go:733-745). Tokens and dollars are both 0. The test TestFinishBooksNoDollarsOnARejectedRequestThatReportedUsage covers a 429 and a 400 with a usage object, with a 200 control that still books.
  • Reviewer 1, finding 1 (estimated filter saving in no total) — fixed in 0a397f9d. declFilterInBaseline now starts with r.estimate_held = 0 AND (dash/toolsuggest.go:199). The measured filter credit keeps its dollars for exactly the rows the books hold out. EstimatedSavedUSD stays outside every total. The test TestFilterCreditKeepsDollarsForAHeldEstimatedRequest covers Realized, /api/adapt/tools and the Overview.
  • Reviewer 1, finding 2 (price_estimated changed meaning) — fixed in 0a397f9d. New column requests.estimate_held, default 0, set only when the hold-out applied. The INSERT has 93 columns, 93 placeholders and 93 arguments; I counted them by script. Price, PriceMode, scanRequest and requestCols all carry the field. The Overview (full and lean) and the request detail read estimate_held. TestOverviewCountsEstimatedRequests holds an old-meaning row that must not count.

New commits, reviewed as a first review

  • 2da3cb04 (#916, stats cache key). cacheKey now uses the exact query (dash/api.go). I traced the UI side: rangeMs maps to === 'now' to 0, writeRange omits until when it is 0, and resolveTime rounds now-X ranges down to 30 s. So the common view still hits the cache, and two windows that differ by one second no longer share a key. Sound.
  • 6138171b (#917, state file). accStateVersion = 2 with a {Version, Snapshot} wrap; restore refuses other versions and bare snapshots. save syncs the temp file before the rename, skips the write when the bytes are unchanged (with an os.Stat re-check so a removed file is written again), and cleans stale temps. withoutCacheMarks now finds cache marks nested in objects and arrays, up to 64 deletions, with gjson path escaping for . * ? | # @. A failed delete breaks the loop and the mark stays in the hash, so the hash only becomes more conservative. Fail-safe. TestSavingsStateSkipsAnUnchangedWrite is consistent: the first save from the fresh tracker writes (its snapshot differs), the second skips, a removed file is written again.
  • a1b89cf1 (#918, panic recovery). KeepAlivePing, PingSavings and accBegin each recover a panic and log it. I checked the accBegin drop path: accEnd guards c.accReq == nil (proxy/accountant_wire.go:45) and savingsTracker.drop guards req != nil. A hostile body can no longer reach the pre-forward recover and discard the pipeline rewrite. Sound.
  • ab085e8e (#920). Pins that a first request books a cost when the price map is loaded. The old $0 row was an unloaded price map, not a pricing defect. No production change.
  • 359a770d. Comment fix only, on messageOps. Correct now.
  • 0206b444. Merge of the four above into integ/batch1. No new content beyond the commits read above.

Non-blocking

  • dash/ui/app.js:3711-3714 — the comment says the net saving of a price_estimated row is "Marked '≈'", but no code marks it now. The base marked it: (e.price_estimated ? '≈ ' : '') plus the caveat "the net may be too high". The fix for reviewer 1's finding 2 switched the detail page to estimate_held and dropped the old marking. A row written before #915 (price_estimated = 1, estimate_held = 0) keeps its booked figure and now shows an unmarked net. At head the two flags are always set together, so such rows exist only in databases written by the unreleased branch. The docs make no false claim about them. Suggest: mark the net with "≈" when price_estimated && !estimate_held, or fix the comment.

What I checked

  • Last review: round 1, 850893277db4cfd5736e06a1bfc55de881693ab9, submitted 2026-11-05T05:55:34Z.
  • New commits read: 8 — 2da3cb04 (#916 cache key), 6138171b (#917 state file), a1b89cf1 (#918 panic recovery), ab085e8e (#920 first-request test), dcd1b31f (rejected-status fix), 359a770d (messageOps comment), 0a397f9d (estimate_held), 0206b444 (merge).
  • Read in full: 2 reviews, 2 comments, 0 inline comments.
  • New items since my last review, and what I did about each:
    • The three round-1 fix commits and the merge: verified each fix in the code, as listed under "Earlier findings".
    • Reviewer 1's two findings and both reviewers' non-blocking notes: checked the fixes (r.estimate_held = 0 AND, the estimate_held column and its readers, pricedRequests++, the head-TTL legs named in the comment and docs, estUsd rendering a negative figure as a cost, Price resetting EstimatedSavedUSD and EstimateHeld, the messageOps comment).
    • The author's comment with the fix details: read in full; each claim matched the code.
    • The author's "Known limits" list: read. The +11 % Overview gap with the default tail breakpoint is stated plainly and the cause is not found. I did not re-raise it; it is an accuracy question of the accountant line, not a defect in a changed line, and it is disclosed.
    • The merge of #928: reviewed as new commits above.
  • The full diff 8508932..0206b444 (1240 lines), every changed file, and the code around each change: proxy/dashcapture.go, proxy/accountant_state.go, proxy/accountant_savings_body.go, proxy/accountant_ping.go, proxy/accountant_wire.go, proxy/accountant_state_test.go, proxy/rejected_savings_test.go, cmd/context-guru-proxy/main.go, dash/event.go, dash/schema.go, dash/store.go, dash/query.go, dash/accountant_price.go, dash/toolsuggest.go, dash/adapttools.go, dash/overview.go, dash/overview_lean.go, dash/api.go, dash/ui/app.js, all new and changed tests, both docs files.
  • The insert SQL after the merge: 93 columns, 93 placeholders, 93 arguments, counted by script.
  • One design point I judged correct rather than a defect: finish does not drop the ping facts for a rejected request (pings.drop stays in the non-rejected branch). The facts were never consumed by a compute that booked nothing, so the next request may still credit a rescue from them without double counting.
  • I did not run the tests (live tests not approved). I read them and their helpers.

@boazdavid boazdavid left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer Kind Started by Context Review Round Head Verdict
Reviewer 1 (glm, user of @boazdavid) GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 offload-review skill, @boazdavid's session resumed 1 of 2 2 of 5 0206b444ccbc approve

Verdict: approve

Earlier findings

  1. My finding 1 (filter dollars of a held estimated request in no total) — fixed in 0a397f9d.

    • declFilterInBaseline is now (r.estimate_held = 0 AND EXISTS (...)) (dash/toolsuggest.go:204). Both users of the const share it: declFilterSavingsQuery (dash/toolsuggest.go:214), DeclFilterSavingsByTenant (dash/toolsuggest.go:274) and carriedCost through inBaselineCol (dash/adapttools.go:341). So every credit path keeps the dollars of a held row.
    • The author's reason is sound: the measured credit is computed from the rate card and the billed tier. It does not depend on the unknown cache state, so it is the right owner.
    • accOverride sets EstimateHeld = true only in the estimated branch (dash/accountant_price.go:354), and PriceMode resets it (dash/accountant_price.go:218). The stored figure stays out of every total. No path counts a held row twice.
    • TestFilterCreditKeepsDollarsForAHeldEstimatedRequest proves the fix: 10 rows, 4 held, all carrying a toolfilter row with priced_by = accountant. It fails without the estimate_held term, because the EXISTS clause alone matches held rows too.
  2. My finding 2 (price_estimated changed meaning, UI claim false for old rows) — fixed in 0a397f9d.

    • New column requests.estimate_held (dash/schema.go:118, additiveColumns dash/schema.go:781, default 0). The INSERT has 93 columns, 93 placeholders and 93 arguments, with boolInt(e.EstimateHeld) at the same position as the column (dash/store.go:596, dash/store.go:676). scanRequest reads it back (dash/query.go:196, dash/query.go:226).
    • Both Overview paths count it: full (dash/overview.go:682) and lean (dash/overview_lean.go:119). The UI text and the request detail key off estimate_held (dash/ui/app.js:1834, dash/ui/app.js:3715-3718). A pre-#915 row has estimate_held = 0, keeps its booked figure, and is no longer called "not counted". The docs sentence is replaced (docs/dashboard.md:162-166).
    • TestOverviewCountsEstimatedRequests now holds one old-meaning row beside new-meaning rows and wants 2, not 3.
  3. My finding 3 and reviewer 2's finding 1 (a rejected request that reported usage booked dollars beside zeroed tokens) — fixed in dcd1b31f.

    • I checked this independently, not only from the author's comment. finish now tests c.rejected() before the compute (proxy/dashcapture.go:741): a status that is not 0 and not 2xx sets accNoResult = "rejected_status", skips computeSafe, and PriceMode falls back with no_result. noResultWhy reports accNoResult before the rounds reason (proxy/dashcapture.go:884-887). Status 0 stays excluded, as before.
    • The ping facts are not dropped on a rejected request. That is correct: the request booked nothing, so the next request can consume them once. No double credit.
    • TestFinishBooksNoDollarsOnARejectedRequestThatReportedUsage sends a 429 and a 400 with a usage object, and a 200 control that books a saving. It pins the fix.
    • I agree with reviewer 2's finding 1 after this check. The fixes match.
  4. Non-blocking items — all fixed. priceCarriedSession counts an in-baseline row as priced (dash/adapttools.go:480). The TTL legs are named as held out in the comment (dash/accountant_price.go:345-352) and the docs. Price resets EstimatedSavedUSD and EstimateHeld (dash/event.go:873). A negative held figure renders as "a cost of $x" (estUsd, dash/ui/app.js:105). The messageOps comment matches the code (359a770d).

Findings

None in the new commits.

Non-blocking

  • Rows written by the round-1 head read as not held. A database written by 8508932 holds rows with price_estimated = 1, estimated_saved_usd > 0 and baseline_cost_usd = cost_usd, but no estimate_held value. The new column defaults to 0, so such a row reads as "counted", and declFilterInBaseline matches it: its filter dollars appear in no total, the defect of my round-1 finding 1. The author's comment covers only pre-#915 rows, which did book their figure; rows from the round-1 branch did not. Exposure is small: the branch is unreleased and the window is hours. A one-time UPDATE requests SET estimate_held = 1 WHERE price_estimated = 1 AND estimated_saved_usd != 0 in the migration would catch them, minus rows whose figure was exactly 0.
  • The +11% Overview figure, cause not found. The author states: with the default tail breakpoint the Overview reports 0.0817 against a true 0.0736. The direction is flattering, not conservative, and it is the headline number of the product. I could not prove the mechanism by reading the code, so I do not make it a finding. Track it before the release: find the cause, or change the default.
  • docs/dashboard.md:169 says acc_fallback reason rejected_status. The stored reason is no_result (dash/accountant_price.go:131). rejected_status appears only in the acc_fallback log line, as the why attribute (proxy/accountant_shadow.go:20-23). Say "log reason" in the docs, so a user inspecting the database finds what the docs promise.

What I checked

  • Last review: round 1, 850893277db4cfd5736e06a1bfc55de881693ab9, posted 2026-11-05T05:53:59Z.
  • New commits read: 8 (2da3cb04 #916, 6138171b #917, a1b89cf1 #918, ab085e8e #920, dcd1b31f, 359a770d, 0a397f9d, merge 0206b444). Full diff 8508932..0206b444 read, and each changed file around the changes.
  • Read in full: 2 reviews (mine and reviewer 2's, round 1), 2 issue comments, 0 inline comments. New items since my last review: the 8 commits above and the author's two comments.
  • The estimate_held wiring end to end: PriceMode/accOverride (set and reset), Price, the column DDL and additiveColumns, insertBatch (93/93/93, counted by script), scanRequest, Overview full and lean, the UI text and the request detail, docs/dashboard.md, docs/data-dependence.md.
  • declFilterInBaseline and its three users, accumulateDeclFilterRow's skipUSD, priceCarriedSession's in-baseline branch, and TestFilterCreditKeepsDollarsForAHeldEstimatedRequest.
  • dcd1b31f: the rejected() gate, its place before computeSafe, the ping-facts drop skip, noResultWhy's new order, and the test's 200 control.
  • 2da3cb04 (#916): cacheKey keeps since/until exact. The UI quantizes now to 30 s in resolveTime (dash/ui/app.js:281) and omits until while the window is live (dash/ui/app.js:300-302), so a repaint burst still shares a key. dashCacheTTL is 30 s, so the key changes no faster than the body expires.
  • 6138171b (#917): the version wrap (accStateVersion = 2, an unwrapped or other-version file is refused and the proxy starts empty), Sync before the rename, the skip-when-unchanged write (a stale mtime is then discarded by the one-hour age rule, which is consistent: an unchanged state describes entries no cache keeps), removeStaleTemps scoped to path + ".tmp*", and findCacheMark/withoutCacheMarks: the walk enters objects and arrays only, strings stay data, keys are escaped for the gjson path, and the 64-mark cap is far above the wire's four. The hash change is why the version bumped.
  • a1b89cf1 (#918): the three recovers — KeepAlivePing (cmd/context-guru-proxy/main.go:976), PingSavings with named returns and injectable library calls, and accBegin, whose recover calls accDrop before the panic can reach the pre-forward recover and cost the rewrite. savings.drop is nil-safe (proxy/accountant_savings.go:34).
  • ab085e8e (#920): test only; pins first-request pricing and the unloaded-price-map cause of the $0 row.
  • I did not run any build or test. Live tests are not approved; I read the code and the tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New/ToDo

Development

Successfully merging this pull request may close these issues.

1 participant