Repository navigation
Conversation
…ws (#911) The B2 double-count fix dropped a whole request from DeclFilterSavings when the accountant priced its toolfilter removal. That also dropped Requests, Reads and Since, so ToolFilterDocFor lost its Realized block and the item list in adaptToolsSaved could sum to more than the total. Now the row stays in the scan. An EXISTS flag column marks it and accumulateDeclFilterRow skips only the USD. carriedCost does the same for the items, so items and total agree. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The default tail breakpoint (#795) adds a cache_control mark to the last message after the pipeline ran. A string content also becomes one text block. The message gets longer on the wire, and no operation reported the extra bytes. The sum check failed on every request with a trailing system message (section=messages ops=0 bodies=-63 bytes), so every request fell back with check_sum and booked no saving. apply now appends one rewrite operation for the last message, in wire bytes, when markTail places a mark. Its ChangeID comes from the message before the mark, so it is the same on every request. The operator tailbreakpoint is an envelope operator in dash, so it gets its own row. No accountant check or plausibility rule is weakened. The check now has the operation it was missing. Harness (FAKE_TOKDIV=8, tailx, 7 requests): net_saved_usd 0 -> 0.0817. Truth (control cost - tail on cost): 0.0736. Fallbacks 7 -> 0. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…ted (#914) A request the provider answered with a non-2xx status billed nothing: cost 0 and baseline 0. Its saved_gross and saved_unique were still booked, so tokens saved disagreed with dollars saved. In the run_m4 repro (turn 1 returns 200, turn 2 returns 400), /api/stats showed saved_gross 28657 and saved_unique 19105 for 2 requests. finish() now calls Event.DropUnbilledSavings. On a status that is not 2xx it sets tokens_after to tokens_before, and zeroes saved_unique, attempted_tokens, and saved_gross, saved_unique and acted on every component row. Status 0 (never reached a provider, such as /compact) and any 2xx (billed, even without usage) are untouched. attempted_tokens goes to 0 so Overview's saved_gross_attempted / attempted_tokens ratio keeps one row population on both sides. No separate attempted counter is added: attempted_tokens already holds what was eligible, and no reader needs the tokens of a request that did not bill. After: saved_gross 9553 and saved_unique 9553 for the same 2 requests (turn 1 only); the 400 row reads tokens_before 19243, tokens_after 19243, saved_unique 0. Dollars do not change. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…y bound (#907) PriceMode compared the tokens the library saved with e.TokensBefore. TokensBefore counts message text only. Removed tool declarations are not in it. A 121-token conversation that drops 2,188 tokens of declarations looked like a request that removed more than it had, so the event fell back with implausible_size and lost every other component's saving too. countedSavedTokens now skips operations of the operators that act on the tools section (toolfilter, toolschema, expand, adjudicate). The bound is unchanged for every operation on messages. The accountant checks are not touched. The bound cannot be made against the full request size here, because the event has no count of tool and system tokens. Harness (FAKE_TOKDIV=7, MSGMULT=30, tf2, 8 requests): fallbacks 8 implausible_size -> 0. net_saved_usd 0 -> 0.0554. Truth (control cost - treatment cost): 0.0564. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…TTL by the billed tier (#912, #919) kaSavedGate needed a request_components row with priced_by='accountant' to accept a rescue the ping only wrote. A keep-alive-only setup stores no component rows, so the rescue was zeroed again. The request row now stores acc_priced (additive column, default 0), set when the accountant override applied. The gate trusts keepalive_saved_usd of such a row as the library's own decision. It is a plain column read placed after the credited-row guard, so the probe order is unchanged. Rows with acc_priced = 0 keep the SQL gate. The entry TTL given to the library was 1h when every breakpoint asked for it, before the billed tier was known. accBilledTTL now lowers the TTL (and the breakpoint TTLs) to 5m after the usage arrives, when the response billed a 5m write, before Compute runs. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…908) Tool deferral runs after the pipeline reported its operations. It marks declarations defer_loading, adds the search tool, and puts a names list at messages[1]: appended to the system message there, or as a new system message. The tools half is found by the envelope reporter. The names list was reported by nobody, so the sum check failed on the messages section (ops=0 bodies=-308 bytes) and conservation failed with it, on every request of a deferred thread. proxy now diffs the body before and after applyToolDefer and reports the change as a tooldefer operation: a one-message splice for a new message, a rewrite in place for an appended list. tooldefer is an envelope operator in dash, so it gets its own row. No accountant check is weakened. Harness (FAKE_TOKDIV=4, MSGMULT=1500, td, 8 requests): check_sum fallbacks 8 -> 0, failed checks 8+7 -> 0. The Overview still does not credit the deferred declarations: the library sees bytes, and a declaration marked defer_loading keeps its bytes. net_saved_usd is -0.0006 (the names list and search tool as cost), truth is +0.0185. /api/tool-defer still reads 0.0412. Making the two agree needs a product decision, so it is not part of this change. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The first request on a thread the proxy has no history for (a restart with a missing or stale state file, or a session begun before the proxy) is priced by the library as if the provider would have written every removed token. If the cache was in fact warm, that is about 2x too high. It was flagged estimated, but still went into baseline_cost_usd, net_saved_usd and total_saved_usd. In run_m5, request 1 booked baseline 0.0424785 (saving 0.042015); the hand figure is 0.021569. Choice: option (a), not (b). The proxy cannot price the unseen prefix at the read tier because the size of that prefix is the very thing the missing history held. The usage only shows how far the compacted body matched, not how much the client had cached. A hand figure needs that size, and the proxy does not have it. So the proxy books no saving on an estimated request, and keeps the figure. Changes: - PriceMode: on an estimated request baseline_cost_usd equals cost_usd and every component saved_usd, saved_first_usd and saved_replay_usd is 0. The held figure goes to the new requests.estimated_saved_usd (Event.EstimatedSavedUSD, additive column, 0 on old rows). The keep-alive rescue and the split saving do not depend on a break and stay booked. - Overview and the lean path sum it as estimated_saved_usd, beside estimated_requests. It is in no total, so net saved = total saved = sum of components = keep-alive tab, with no filter. - UI: the Preset card says how many requests are not counted and shows the held figure. The request detail shows it on its own line marked with an approximate sign. - docs/dashboard.md describes the stored field and the rule. Numbers (run_m5, same binary and harness): net_saved_usd 0.0911627 before, 0.0491477 after. Request 1 baseline 0.0424785 before, 0.0004635 (equal to cost) after, with 0.042015 held in estimated_saved_usd. Requests 2 and 3 are unchanged (0.0243054, 0.0257421). Golden files: stats.json, stats_lean.json and stats_windowed.json gain estimated_saved_usd: 0. No golden fixture holds an estimated row, so no other number moves. Test changes: TestSavingsRecordEndsWithTheInjectedForwardedBody and TestDashboardBooksTheAccountantSaving sent the first request of a thread and read its booked saving. They now assert the estimate is held out; the second one sends a second request and asserts the booked saving there. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The only conflict is in dash/accountant_switch_test.go. Both branches add a test at the same place. Keep both tests and close each function. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…ches Both branches add a column next to price_estimated: acc_priced (gates) and estimated_saved_usd (recording). Keep both, in this order, in the schema, the migration list, the INSERT columns, the INSERT values and the SELECT and Scan lists. Git merged the placeholder row of the INSERT without a conflict, so it had 91 placeholders for 92 columns. Every insert failed at run time and 48 proxy tests failed. Add the missing placeholder. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The key rounded since and until down to 30 s, so a narrower window got the wider window's cached totals. The key now holds the exact values. The UI rounds its own now to 30 s so relative ranges still share a key. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…past nested cache marks (#917) Wrap the snapshot in a proxy version and refuse another one. Sync before the rename, skip an unchanged write and remove stale temp files at start. Strip cache_control at any depth before hashing a body element. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
The keeper's ping row and PingSavings now recover and log, so a library panic cannot end the process. accBegin recovers and drops its record, so a panic on a hostile body no longer reaches the pre-forward recover. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…e map (#920) A realistic first turn books its cost when the price is known. The reported rows had no price: the price map loads in the background and the first requests arrived before it finished. No code change. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
boazdavid
left a comment
There was a problem hiding this comment.
| Reviewer | Kind | Started by | Context | Review | Round | Head | Verdict |
|---|---|---|---|---|---|---|---|
| Reviewer 1 (glm, user of @boazdavid) | GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 | offload-review skill, @boazdavid's session | fresh | 1 of 2 | 1 of 5 | 850893277db4 |
blocking |
Verdict: blocking
Findings
-
dash/toolsuggest.go:199 — the filter saving of an estimated request appears in no dollar total
- What fails: a request the accountant priced as an estimate (the first request of every thread) books no saving anywhere (#915:
BaselineCostUSD == CostUSD, every componentSavedUSD0), but itstoolfilterrow still carriespriced_by = 'accountant'(dash/accountant_price.go:331 sets it before the estimated branch zeroes the dollars).declFilterInBaselinethen matches that row, soaccumulateDeclFilterRowreturns at dash/toolsuggest.go:253 and the measured credit adds no dollars. The comment and docs/dashboard.md:125 justify this with "the net saving has them" — for an estimated row the net saving has nothing. - Proof: trace an estimated request with
filtered_decl_tokens > 0. Accountant side: dash/accountant_price.go:335-349 holdsfig.DeltaUSDinEstimatedSavedUSD, outside every total. Measured side: dash/toolsuggest.go:199-200 selects the row as in-baseline, dash/toolsuggest.go:253 skips the USD. Result: the Realized block and/api/adapt/toolscount the request and its reads but show $0 for it. The same drop applies to thecarriedCostpath (dash/adapttools.go:473-479). Every thread's first request is estimated, so this is systematic, not an edge. The new testTestFilterCreditKeepsFactsAndSkipsDollarsForAccountantPricedRowscovers only non-estimated rows. - Fix: make
declFilterInBaselinefalse for a row whose request is estimated (joinr.price_estimated), so the measured credit keeps the measured dollars for exactly those rows the books hold out. Or hold the filter part out ofEstimatedSavedUSDand say so in the docs. Pick one owner for the figure; today it has none.
- What fails: a request the accountant priced as an estimate (the first request of every thread) books no saving anywhere (#915:
-
dash/ui/app.js:1828 — the Overview says estimated requests are "not counted above", but rows written before this change are counted
- What fails:
price_estimatedchanged meaning in this PR. Before, an estimated row booked its figure (BaselineCostUSD = CostUSD + DeltaUSD, base commit189c169, dash/accountant_price.go). After, an estimated row books nothing and the UI text reads "N of M requests are not counted above ... Estimated saving, not in any total: $X". No migration unbooks the old rows, so a database that has both shows a false claim: the old rows are inNetSavedUSDandTotalSavedUSD, and the card says they are not. The request detail page (dash/ui/app.js:3710-3714) shows the same contradiction: "Not counted ... no saving is booked" beside a non-zero net saving that is in the totals, and "Estimated saving (not counted): ≈ $0.00" for a row whose estimate was booked. - Proof:
git show 189c169:dash/accountant_price.go— the baseaccOverrideends withe.BaselineCostUSD = e.CostUSD + fig.DeltaUSD; e.PriceEstimated = estimated.Overview.EstimatedRequestsisSUM(r.price_estimated)(dash/overview.go:681), so those rows are counted, andNetSavedUSDsums their bookedbaseline_cost_usd. The docs answer this with "A row written before the column existed reads as exact" (docs/dashboard.md:161) — false:price_estimatedis a pre-existing column and those rows read as estimated. Exposure is bounded: the accountant line is not in any tag (newest tag v0.3.4, 2026-10-01, predates #768), so only databases written by the unreleased branch are affected. That is still the class of row this repo elsewhere handles on purpose (acc_priceddefaults 0,priced_byempty, "rows written before the switch keep their legacy figures"). - Fix: distinguish the two meanings. Count only rows where the hold-out applied (for example
price_estimated = 1 AND estimated_saved_usd > 0, or a new marker), or migrate the old rows. Then fix the docs sentence at docs/dashboard.md:161.
- What fails:
-
dash/event.go:809 — a rejected request that still reported usage books dollars beside zeroed tokens
- What fails:
DropUnbilledSavingszeroes the token saving of every non-2xx row, on the ground that "the provider billed none of it, so tokens and dollars agree" (docs/dashboard.md:166-170). Butfinishruns the accountant andPriceModewheneverusageOKis true, with no status test (proxy/dashcapture.go:733-745), and the response path parses usage from the body whatever the status (proxy/proxy.go:2308). A gateway that returns a 4xx/429 body with ausageobject then produces a row withtokens_after == tokens_before, zeroSavedUnique, but a non-zeroBaselineCostUSD - CostUSDand componentSavedUSDfrom the accountant figure. Tokens and dollars disagree — the exact defect #914 set out to remove. - Proof:
DropUnbilledSavings(dash/event.go:809-818) tests only the status;c.accReq != nil && usageOK && !c.accDone(proxy/dashcapture.go:733) tests none;responseUsageWhyat proxy/proxy.go:2308 runs onrespBodyof any status. The new testTestFinishBooksNoSavedTokensOnARejectedRequestcovers only rejected rows withusageOKfalse, so the gap is untested. - Fix: skip the accountant compute (or force the fallback) when
e.Statusis not 0 and not 2xx, or extendDropUnbilledSavingsto zero the dollar figures too.
- What fails:
Non-blocking
- dash/adapttools.go:473-479 — the
inBaselinebranch never incrementsout.pricedRequests, sost.Priced = scan.pricedRequests > 0(dash/toolapi.go:675) reports "not priced" when every in-scope filtered row is accountant-priced. A deliberate $0 then reads as "no measurement". Consider counting the row as priced with a known rate. - dash/accountant_price.go:335-350 — the estimated hold-out also zeroes the
headttlrow (premium and rescue). The comment names only the keep-alive rescue and the split as staying. The first request of a thread is where a 1h head is written, so the premium cost shows as $0 exactly there. If this is intended, say so in the comment and docs; if not, keep the ttl legs booked as their own column. - dash/event.go:866 —
PriceresetsAccountantPricedbut notEstimatedSavedUSD. No production caller mixesPriceModeandPricetoday (onlycostOnlyCopyin tests), so this is latent. Reset both for symmetry. - proxy/accountant_envelope.go:291 —
messageOpsreturns nil for any change it cannot explain, and the sum check then sends the event to the no-saving fallback. Fail-open, documented. Acceptable.
What I checked
- The full diff
189c169..8508932(33 files) and the code around every hunk. - The tail-breakpoint operation:
markTailruns in a defer inapply.BodyOpts(apply/apply.go:409-422) afterwireSizeschained the pipeline ops, sotailOp'sSizeBeforemeets the chain's end on the same message. TheOpsDropped == 0guard, the stableChangeID, and the append-after-pipeline ordering are consistent with the library's sum check. messageOpsand the tooldefer wiring: the full-slice append in proxy/proxy.go:1596 avoids aliasing; the insertion and append forms match thewireSizessplice conventions; ops for messages, tools and system cover the client-to-forwarded diff in order.PriceMode/accOverride: the estimated hold-out,AccountantPriced, the plausibility bound withtoolSectionOperators(operator strings matchenvelopeOwner's output), and the storage round-trip (insertBatchcolumns,scanRequest,additiveColumnsmigration, golden files).- The keep-alive gate:
acc_priced = 1sits after thecol > 0guard and before the probes, so the credited-row short-circuit order holds; everykaSaved/kaSavedSincecaller aliasesrequests; manual pings bookmanual_saved_usd, notkeepalive_saved_usd, so the gate does not touch them. accBilledTTL: theCacheWrite1h < CacheWritetest matchesUsage's documented semantics (usage.go:31) andkeepalive.ResolveTier's rule; the breakpoint copy protects the slice the sibling store keeps; it runs beforecomputeSafeon the only compute path (rounds <= 1).DropUnbilledSavingsordering infinish(status is set in the event literal before it runs), and the status-0 and 2xx exclusions.DeclFilterSavings,DeclFilterSavingsByTenant,carriedCost: theinBaselineSELECT column, theskipUSDreturn after the tier-token facts, andbyIDinitialization for every key.- Release containment: the accountant line (#768 and its fixes) is in no tag; newest tag v0.3.4 predates it. This bounds finding 2 to databases written by the unreleased branch.
- Commit list (9 fix commits, 3 merges) read in full.
boazdavid
left a comment
There was a problem hiding this comment.
| Reviewer | Kind | Started by | Context | Review | Round | Head | Verdict |
|---|---|---|---|---|---|---|---|
| Reviewer 2 (glm, user of @boazdavid) | GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 | offload-review skill, @boazdavid's session | fresh | 2 of 2 | 1 of 5 | 850893277db4 |
blocking |
Verdict: blocking
Findings
- dash/event.go:809 — a rejected request that still parsed usage books dollar savings, so tokens and dollars disagree.
- What fails:
DropUnbilledSavingszeroes the token side only (TokensAfter,SavedUnique, per-rowSavedGross). It does not stop the accountant path. Infinish(proxy/dashcapture.go:733-750) the order is:DropUnbilledSavings, thencomputeSafe(it runs, because it is gated onusageOK, not on status), thenPriceMode. If a provider or gateway returns a non-2xx status whose body still parses usage (responseUsageWhyat proxy/proxy.go:2307 parses the body with no status check, andusageOKis sticky), then: tokens book 0, butaccOverrideapplies, sets each componentSavedUSDfrom the library figure andBaselineCostUSD = CostUSD + DeltaUSD. The row then shows$saved_usd > 0besidesaved_gross = 0. This is the exact disagreement the PR says cannot happen: docs/dashboard.md says "The provider billed none of it, so tokens and dollars agree", and the code comment at dash/event.go:804 says "cost and baseline are 0", which is false when usage parsed. The ping path already guards this (proxy/accountant_ping.go:106 refuses a non-2xx ping); the request path does not. - Proof: trace above; no line between
DropUnbilledSavings(proxy/dashcapture.go:529) andPriceMode(proxy/dashcapture.go:750) readsc.status.accOverride(dash/accountant_price.go:250) reads no status either, andDropUnbilledSavingssettingActed = falseremoves theFallbackActedWithoutFiguregate, which makes the override easier, not harder. - Fix: gate the accountant compute (or the override) on a 2xx status, as
PingSavingsdoes. Or zero the dollar side inDropUnbilledSavingstoo. Reachability is low (an error body that carries usage), so the team may accept it — but then the docs and the code comment must say so.
- What fails:
Non-blocking
- dash/accountant_price.go:335 — the estimate hold is all-or-nothing. A first request whose net figure is a cost (the tooldefer names list, an expand or adjudicate injection) now books 0 instead of that cost, so net saved is overstated by it. The docs and the issue speak only of savings that are too high. The held figure can be negative and renders as "Estimated saving (not counted): ≈ $-x". The cost side is measured, not estimate-dependent. Small figures; deliberate and test-covered (proxy/accountant_inject_test.go). Name it in the docs, or hold out only the removal half.
- proxy/accountant_envelope.go:291 — the comment on
messageOpssays "Any other change returns nil". The equal-length case returns one op per changed message, so a two-message change is explained, not refused. That is safe (it explains more), but the comment overstates the refusal. Fix the comment. - dash/event.go:866 —
PriceresetsPriceEstimatedandAccountantPricedbut notEstimatedSavedUSD. No production path callsPriceafterPriceModetoday (pings and the import path build fresh events), but a future caller could store a held figure on a row withprice_estimated = 0. Reset it there too, for symmetry withPriceMode.
What I checked
- The full diff
189c169d..HEAD(9 fix commits, 3 merges), every changed file, and the code around each change:apply/apply.go,apply/ops.go,proxy/proxy.go,proxy/accountant_wire.go,proxy/accountant_envelope.go,proxy/dashcapture.go,dash/accountant_price.go,dash/event.go,dash/keepalive.go,dash/adapttools.go,dash/toolsuggest.go,dash/overview.go,dash/overview_lean.go,dash/query.go,dash/schema.go,dash/store.go,dash/ui/app.js, both docs files, all new and changed tests. - The accountant library at the pinned version (
v0.1.1-0.20261104203351-5fa19864db4c, read from the module cache): howMessageSplicepositions are undone (compute.goundo/apply), whenpenalty_unknownis set (PhaseUnknown, whole result), whatisChangekeeps, and the ended-change rules. The tail op and the tooldefer op chain correctly with pipeline ops, and the ChangeID schemes (hash of before-bytes; position for an insert) match the library's replace-an-active-change semantics. - The insert SQL: 92 columns, 92 placeholders, 92 arguments; the new columns sit at the same place in all three lists.
tooldefer/rewrite.go placeList: it produces exactly the two shapesmessageOpsexplains (append to a system message at index 1, or insert one at index 1, including the one-message append). Anything else makesmessageOpsreturn nil, the sum check fails, and the event falls back — the safe direction.accBilledTTLagainstentryFactsand the library's TTL use: it only lowers 1h to 5m, only with an exact lifetime, only when a write was billed below the 1h tier, and it copies the breakpoints slice before mutating. SharedWarm is documented as judged with the asked tier.- The keep-alive gate change: every
kaSaved/kaSavedSincecaller aliasesrequestsasr., theacc_pricedcolumn is in both the CREATE TABLE andadditiveColumns, and the credited-row guard stays first. When the override applies,keepalive_saved_usdis always the library's figure (accOverridesets it), so trustingacc_priced = 1is sound. Ping events usePrice, neverPriceMode, so they never carry the marker. - The plausibility-bound exclusion (
toolSectionOperators):context-guru.expandandcontext-guru.adjudicateoperators are emitted only by the envelope reporter for tools-section declarations; pipeline message components use their own names, so no message operation escapes the bound. #911read-side changes:DeclFilterSavingsand thecarriedCostfilter scan skip dollars for accountant-priced rows and keep requests, tokens, dates andSince(set before the skip). The filter scan'spricedRequestsis not consumed by any reader that could misread a $0.- Fail-open: no change touches the forwarded body; all changes are reporting and pricing. A wrong or missing operation falls back to no saving, never to a wrong forward.
- Test coverage: each of the six issues (#906, #907, #908, #911, #912/#919, #914, #915) has a unit test and, where it needs the wire, an end-to-end test. I did not run them (live tests not approved); I read them and their helpers.
A 4xx or 5xx body can carry a usage object. The accountant then priced a saving beside the zeroed tokens. Skip the compute for a status that is not 0 and not 2xx, as PingSavings does. The row falls back with no_result. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
…old rows as held - The measured filter credit owns the filter dollars of a request whose figure was held out (declFilterInBaseline is false when estimate_held). - New column requests.estimate_held marks the hold-out. Rows from before keep their legacy meaning and are not counted as not counted. - Count a baseline-priced filtered row as priced in adapttools. - Price resets EstimatedSavedUSD and EstimateHeld. - Name what the hold-out covers (ttl legs, negative figure) in code, UI, docs. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: DAVID BOAZ <DAVIDBO@il.ibm.com>
|
Round 1 fixes are pushed to integ/batch1: dcd1b31, 359a770, 0a397f9. Review 1, finding 1 (estimated filter saving in no total): fixed in 0a397f9. Owner: the measured filter credit. Review 1, finding 2 (price_estimated changed meaning): fixed in 0a397f9. New column Review 1 finding 3 and Review 2 finding 1 (rejected request with usage books dollars): fixed in dcd1b31. The accountant compute is skipped for a status that is not 0 and not 2xx ( Non-blocking: the hold-out covers the head-TTL legs; the comment and the docs now say so, and say that the keep-alive rescue and split stay booked. A negative held figure renders as "a cost of $x" and the docs name it. Assisted-By: Claude (Anthropic AI) noreply@anthropic.com |
|
Round 1 replies. All findings are fixed in Reviewer 1, finding 1 (filter saving of an estimated request in no total). Fixed in Reviewer 1, finding 2 ( Reviewers 1 and 2, rejected request with usage. Fixed in Non-blocking, all done.
Known limits, stated plainly.
|
boazdavid
left a comment
There was a problem hiding this comment.
| Reviewer | Kind | Started by | Context | Review | Round | Head | Verdict |
|---|---|---|---|---|---|---|---|
| Reviewer 2 (glm, user of @boazdavid) | GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 | offload-review skill, @boazdavid's session | resumed | 2 of 2 | 2 of 5 | 0206b444ccbc |
approve |
Verdict: approve
Findings
None. Every round-1 finding is fixed and verified in the code. The new commits (#916, #917, #918, #920 and the round-1 fixes) contain no defect I could confirm.
Earlier findings, re-checked
- My finding 1 (rejected request with usage books dollars) — fixed in
dcd1b31f.capture.rejected()(proxy/dashcapture.go:880) returnsc.status != 0 && (c.status < 200 || c.status > 299). This matchesDropUnbilledSavingsexactly.finishnow skipscomputeSafeandPriceModefor such a row and setsaccNoResult = "rejected_status"(proxy/dashcapture.go:733-745). Tokens and dollars are both 0. The testTestFinishBooksNoDollarsOnARejectedRequestThatReportedUsagecovers a 429 and a 400 with a usage object, with a 200 control that still books. - Reviewer 1, finding 1 (estimated filter saving in no total) — fixed in
0a397f9d.declFilterInBaselinenow starts withr.estimate_held = 0 AND(dash/toolsuggest.go:199). The measured filter credit keeps its dollars for exactly the rows the books hold out.EstimatedSavedUSDstays outside every total. The testTestFilterCreditKeepsDollarsForAHeldEstimatedRequestcovers Realized,/api/adapt/toolsand the Overview. - Reviewer 1, finding 2 (
price_estimatedchanged meaning) — fixed in0a397f9d. New columnrequests.estimate_held, default 0, set only when the hold-out applied. The INSERT has 93 columns, 93 placeholders and 93 arguments; I counted them by script.Price,PriceMode,scanRequestandrequestColsall carry the field. The Overview (full and lean) and the request detail readestimate_held.TestOverviewCountsEstimatedRequestsholds an old-meaning row that must not count.
New commits, reviewed as a first review
2da3cb04(#916, stats cache key).cacheKeynow uses the exact query (dash/api.go). I traced the UI side:rangeMsmapsto === 'now'to 0,writeRangeomitsuntilwhen it is 0, andresolveTimeroundsnow-Xranges down to 30 s. So the common view still hits the cache, and two windows that differ by one second no longer share a key. Sound.6138171b(#917, state file).accStateVersion = 2with a{Version, Snapshot}wrap;restorerefuses other versions and bare snapshots.savesyncs the temp file before the rename, skips the write when the bytes are unchanged (with anos.Statre-check so a removed file is written again), and cleans stale temps.withoutCacheMarksnow finds cache marks nested in objects and arrays, up to 64 deletions, with gjson path escaping for. * ? | # @. A failed delete breaks the loop and the mark stays in the hash, so the hash only becomes more conservative. Fail-safe.TestSavingsStateSkipsAnUnchangedWriteis consistent: the first save from the fresh tracker writes (its snapshot differs), the second skips, a removed file is written again.a1b89cf1(#918, panic recovery).KeepAlivePing,PingSavingsandaccBegineach recover a panic and log it. I checked theaccBegindrop path:accEndguardsc.accReq == nil(proxy/accountant_wire.go:45) andsavingsTracker.dropguardsreq != nil. A hostile body can no longer reach the pre-forward recover and discard the pipeline rewrite. Sound.ab085e8e(#920). Pins that a first request books a cost when the price map is loaded. The old$0row was an unloaded price map, not a pricing defect. No production change.359a770d. Comment fix only, onmessageOps. Correct now.0206b444. Merge of the four above intointeg/batch1. No new content beyond the commits read above.
Non-blocking
- dash/ui/app.js:3711-3714 — the comment says the net saving of a
price_estimatedrow is "Marked '≈'", but no code marks it now. The base marked it:(e.price_estimated ? '≈ ' : '')plus the caveat "the net may be too high". The fix for reviewer 1's finding 2 switched the detail page toestimate_heldand dropped the old marking. A row written before #915 (price_estimated = 1,estimate_held = 0) keeps its booked figure and now shows an unmarked net. At head the two flags are always set together, so such rows exist only in databases written by the unreleased branch. The docs make no false claim about them. Suggest: mark the net with "≈" whenprice_estimated && !estimate_held, or fix the comment.
What I checked
- Last review: round 1,
850893277db4cfd5736e06a1bfc55de881693ab9, submitted 2026-11-05T05:55:34Z. - New commits read: 8 —
2da3cb04(#916 cache key),6138171b(#917 state file),a1b89cf1(#918 panic recovery),ab085e8e(#920 first-request test),dcd1b31f(rejected-status fix),359a770d(messageOps comment),0a397f9d(estimate_held),0206b444(merge). - Read in full: 2 reviews, 2 comments, 0 inline comments.
- New items since my last review, and what I did about each:
- The three round-1 fix commits and the merge: verified each fix in the code, as listed under "Earlier findings".
- Reviewer 1's two findings and both reviewers' non-blocking notes: checked the fixes (
r.estimate_held = 0 AND, theestimate_heldcolumn and its readers,pricedRequests++, the head-TTL legs named in the comment and docs,estUsdrendering a negative figure as a cost,PriceresettingEstimatedSavedUSDandEstimateHeld, themessageOpscomment). - The author's comment with the fix details: read in full; each claim matched the code.
- The author's "Known limits" list: read. The +11 % Overview gap with the default tail breakpoint is stated plainly and the cause is not found. I did not re-raise it; it is an accuracy question of the accountant line, not a defect in a changed line, and it is disclosed.
- The merge of #928: reviewed as new commits above.
- The full diff
8508932..0206b444(1240 lines), every changed file, and the code around each change:proxy/dashcapture.go,proxy/accountant_state.go,proxy/accountant_savings_body.go,proxy/accountant_ping.go,proxy/accountant_wire.go,proxy/accountant_state_test.go,proxy/rejected_savings_test.go,cmd/context-guru-proxy/main.go,dash/event.go,dash/schema.go,dash/store.go,dash/query.go,dash/accountant_price.go,dash/toolsuggest.go,dash/adapttools.go,dash/overview.go,dash/overview_lean.go,dash/api.go,dash/ui/app.js, all new and changed tests, both docs files. - The insert SQL after the merge: 93 columns, 93 placeholders, 93 arguments, counted by script.
- One design point I judged correct rather than a defect:
finishdoes not drop the ping facts for a rejected request (pings.dropstays in the non-rejected branch). The facts were never consumed by a compute that booked nothing, so the next request may still credit a rescue from them without double counting. - I did not run the tests (live tests not approved). I read them and their helpers.
boazdavid
left a comment
There was a problem hiding this comment.
| Reviewer | Kind | Started by | Context | Review | Round | Head | Verdict |
|---|---|---|---|---|---|---|---|
| Reviewer 1 (glm, user of @boazdavid) | GLM rits/zai-org/glm-5-3 via Claude Code 2.1.241 | offload-review skill, @boazdavid's session | resumed | 1 of 2 | 2 of 5 | 0206b444ccbc |
approve |
Verdict: approve
Earlier findings
-
My finding 1 (filter dollars of a held estimated request in no total) — fixed in
0a397f9d.declFilterInBaselineis now(r.estimate_held = 0 AND EXISTS (...))(dash/toolsuggest.go:204). Both users of the const share it:declFilterSavingsQuery(dash/toolsuggest.go:214),DeclFilterSavingsByTenant(dash/toolsuggest.go:274) andcarriedCostthroughinBaselineCol(dash/adapttools.go:341). So every credit path keeps the dollars of a held row.- The author's reason is sound: the measured credit is computed from the rate card and the billed tier. It does not depend on the unknown cache state, so it is the right owner.
accOverridesetsEstimateHeld = trueonly in the estimated branch (dash/accountant_price.go:354), andPriceModeresets it (dash/accountant_price.go:218). The stored figure stays out of every total. No path counts a held row twice.TestFilterCreditKeepsDollarsForAHeldEstimatedRequestproves the fix: 10 rows, 4 held, all carrying atoolfilterrow withpriced_by = accountant. It fails without theestimate_heldterm, because the EXISTS clause alone matches held rows too.
-
My finding 2 (
price_estimatedchanged meaning, UI claim false for old rows) — fixed in0a397f9d.- New column
requests.estimate_held(dash/schema.go:118,additiveColumnsdash/schema.go:781, default 0). The INSERT has 93 columns, 93 placeholders and 93 arguments, withboolInt(e.EstimateHeld)at the same position as the column (dash/store.go:596, dash/store.go:676).scanRequestreads it back (dash/query.go:196, dash/query.go:226). - Both Overview paths count it: full (dash/overview.go:682) and lean (dash/overview_lean.go:119). The UI text and the request detail key off
estimate_held(dash/ui/app.js:1834, dash/ui/app.js:3715-3718). A pre-#915 row hasestimate_held = 0, keeps its booked figure, and is no longer called "not counted". The docs sentence is replaced (docs/dashboard.md:162-166). TestOverviewCountsEstimatedRequestsnow holds one old-meaning row beside new-meaning rows and wants 2, not 3.
- New column
-
My finding 3 and reviewer 2's finding 1 (a rejected request that reported usage booked dollars beside zeroed tokens) — fixed in
dcd1b31f.- I checked this independently, not only from the author's comment.
finishnow testsc.rejected()before the compute (proxy/dashcapture.go:741): a status that is not 0 and not 2xx setsaccNoResult = "rejected_status", skipscomputeSafe, andPriceModefalls back withno_result.noResultWhyreportsaccNoResultbefore the rounds reason (proxy/dashcapture.go:884-887). Status 0 stays excluded, as before. - The ping facts are not dropped on a rejected request. That is correct: the request booked nothing, so the next request can consume them once. No double credit.
TestFinishBooksNoDollarsOnARejectedRequestThatReportedUsagesends a 429 and a 400 with ausageobject, and a 200 control that books a saving. It pins the fix.- I agree with reviewer 2's finding 1 after this check. The fixes match.
- I checked this independently, not only from the author's comment.
-
Non-blocking items — all fixed.
priceCarriedSessioncounts an in-baseline row as priced (dash/adapttools.go:480). The TTL legs are named as held out in the comment (dash/accountant_price.go:345-352) and the docs.PriceresetsEstimatedSavedUSDandEstimateHeld(dash/event.go:873). A negative held figure renders as "a cost of $x" (estUsd, dash/ui/app.js:105). ThemessageOpscomment matches the code (359a770d).
Findings
None in the new commits.
Non-blocking
- Rows written by the round-1 head read as not held. A database written by
8508932holds rows withprice_estimated = 1,estimated_saved_usd > 0andbaseline_cost_usd = cost_usd, but noestimate_heldvalue. The new column defaults to 0, so such a row reads as "counted", anddeclFilterInBaselinematches it: its filter dollars appear in no total, the defect of my round-1 finding 1. The author's comment covers only pre-#915 rows, which did book their figure; rows from the round-1 branch did not. Exposure is small: the branch is unreleased and the window is hours. A one-timeUPDATE requests SET estimate_held = 1 WHERE price_estimated = 1 AND estimated_saved_usd != 0in the migration would catch them, minus rows whose figure was exactly 0. - The +11% Overview figure, cause not found. The author states: with the default tail breakpoint the Overview reports 0.0817 against a true 0.0736. The direction is flattering, not conservative, and it is the headline number of the product. I could not prove the mechanism by reading the code, so I do not make it a finding. Track it before the release: find the cause, or change the default.
- docs/dashboard.md:169 says
acc_fallbackreasonrejected_status. The stored reason isno_result(dash/accountant_price.go:131).rejected_statusappears only in theacc_fallbacklog line, as thewhyattribute (proxy/accountant_shadow.go:20-23). Say "log reason" in the docs, so a user inspecting the database finds what the docs promise.
What I checked
- Last review: round 1,
850893277db4cfd5736e06a1bfc55de881693ab9, posted 2026-11-05T05:53:59Z. - New commits read: 8 (
2da3cb04#916,6138171b#917,a1b89cf1#918,ab085e8e#920,dcd1b31f,359a770d,0a397f9d, merge0206b444). Full diff8508932..0206b444read, and each changed file around the changes. - Read in full: 2 reviews (mine and reviewer 2's, round 1), 2 issue comments, 0 inline comments. New items since my last review: the 8 commits above and the author's two comments.
- The
estimate_heldwiring end to end:PriceMode/accOverride(set and reset),Price, the column DDL andadditiveColumns,insertBatch(93/93/93, counted by script),scanRequest, Overview full and lean, the UI text and the request detail, docs/dashboard.md, docs/data-dependence.md. declFilterInBaselineand its three users,accumulateDeclFilterRow'sskipUSD,priceCarriedSession's in-baseline branch, andTestFilterCreditKeepsDollarsForAHeldEstimatedRequest.dcd1b31f: therejected()gate, its place beforecomputeSafe, the ping-facts drop skip,noResultWhy's new order, and the test's 200 control.2da3cb04(#916):cacheKeykeeps since/until exact. The UI quantizesnowto 30 s inresolveTime(dash/ui/app.js:281) and omitsuntilwhile the window is live (dash/ui/app.js:300-302), so a repaint burst still shares a key.dashCacheTTLis 30 s, so the key changes no faster than the body expires.6138171b(#917): the version wrap (accStateVersion = 2, an unwrapped or other-version file is refused and the proxy starts empty),Syncbefore the rename, the skip-when-unchanged write (a stale mtime is then discarded by the one-hour age rule, which is consistent: an unchanged state describes entries no cache keeps),removeStaleTempsscoped topath + ".tmp*", andfindCacheMark/withoutCacheMarks: the walk enters objects and arrays only, strings stay data, keys are escaped for the gjson path, and the 64-mark cap is far above the wire's four. The hash change is why the version bumped.a1b89cf1(#918): the three recovers —KeepAlivePing(cmd/context-guru-proxy/main.go:976),PingSavingswith named returns and injectable library calls, andaccBegin, whose recover callsaccDropbefore the panic can reach the pre-forward recover and cost the rewrite.savings.dropis nil-safe (proxy/accountant_savings.go:34).ab085e8e(#920): test only; pins first-request pricing and the unloaded-price-map cause of the $0 row.- I did not run any build or test. Live tests are not approved; I read the code and the tests.
This PR stacks on the draft re-land #923. It must not merge unless the repo owner says yes to the re-land. Savings figures change.
Summary
What the merge changed
dash/accountant_switch_test.go: both branches added a test at the same place. Both tests stay.dash/schema.go,store.go,query.go,event.go,accountant_price.go: gates addsacc_priced, recording addsestimated_saved_usd. Both columns stay, in this order, everywhere.How it was tested
go test ./dash/...passes.go test ./proxy/...passes exceptTestALateFirstByteOnTheRealTransportNeverReachesTheClient. That test also fails on the re-land base189c169d. It expects an answer within 5 ms and this machine is loaded.apply,keepalive,components,tooldeferpass.Known limits
Assisted-By: Claude (Anthropic AI) noreply@anthropic.com