Fix: coalesce chatty RLM child usage attribution writes that saturate the worker (#1054) — credits @Lauritz-Timm's prior #1060/#1061 #1788
kaluli123123
started this conversation in
Bug reports
Replies: 1 comment
|
Independent journal measurements confirm the persistence amplification. Across 25 audited top-level journals, 16 contained 39,837
Current source still appends one durable attribution record for each child assistant completion. The aggregate usage is useful, but the per-completion rows become a substantial secondary source of journal growth and replay work during wide or long-running RLM trees. These measurements support preserving exact additive accounting while coalescing durable writes into bounded per-child or periodic deltas. No raw journals are attached. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
With active RLM subagents, every child assistant
message_endappends achild_usage_attributedentry to the parent session's transcript, synchronously and unconditionally. With a few fast-working subagents this becomes a firehose: one report measured 553 attribution entries in 20 minutes (peak 73/min), each a full ~600-byte transcript entry that also mutates the parent's assistant-message usage object and advances the transcript leaf/index on every single call. The worker process hosting the parent + children saturated (97-99% CPU, 3.1 GB RSS observed), so the supervisor'sattach/heartbeats_listcommands timed out for 25+ minutes and the session became completely unreachable from the UI. This is #1054, one of the "related reports" under the still-open tracker #1382.Prior art, credited upfront: @Lauritz-Timm already diagnosed this and proposed a fix in #1060/#1061 (both closed, not merged, via the maintainer's standard backlog-sweep note citing #1162 as "covering" this — but the underlying bug is still live on current
main, confirmed with a new regression test below). Their approach and mine converge on the same root cause and a similar shape (accumulate pending usage, flush at a natural boundary, force-flush on cleanup), which isn't surprising given there's really one place in the code this can be fixed. The concrete difference: #1060/#1061 flush onagent_endand whenever the attributionoriginchanges; this PR instead debounces on a fixed time window (RLM_CHILD_USAGE_ATTRIBUTION_COALESCE_MS, 1s) regardless of origin, plus the same forced flush on run settlement. Either is a reasonable design; I'm not claiming this supersedes their work, just documenting an independent pass with a different coalescing trigger, in case it's a useful alternative or point of comparison for whoever picks this up.Root cause
The RLM child event subscriber in
spawnRlmChild(agent-session.ts) callsattributeChildUsage()andsessionManager.appendChildUsageAttribution()directly and synchronously inside the"message_end"handler, once per child assistant message, with no batching of any kind.Fix
Track a per-run pending usage delta (summed via the existing
addAssistantUsage, not just "keep the latest" — each message's usage is an independent, additive cost, so skipping intermediate messages would undercount) and debounce the actual attribute+persist call to at most once perRLM_CHILD_USAGE_ATTRIBUTION_COALESCE_MS(1000ms) of inactivity for that child. The run's existingfinallyblock also force-flushes any still-pending delta before the child run settles, so a run that finishes inside the coalescing window never silently drops its last batch of usage. This lives entirely at the call site;SessionManager.appendChildUsageAttribution's own contract (synchronous JSONL append) is unchanged, since other tests (context-tree.test.ts) rely on it persisting immediately when called.Tests
packages/coding-agent/test/agent-session-recursion.test.ts:childUsagestill equals the exact total across all 20 turns — directly modeling the reported ~550-messages-to-~550-entries shape, now collapsed (21 entries for 20 turns on the pre-fix code, matching the bug's ratio almost exactly).Both verified to fail against the pre-fix code and pass after.
Validation
npx tsgo -p tsconfig.json --noEmitand the rootnpm run check(biome, tsgo, installer render, browser smoke) both pass. Added apackages/coding-agent/.changes/fragment per the changelog-fragment CI check.Patch
Branch: https://github.com/kaluli123123/prime-agent/tree/fix/coalesce-child-usage-attribution
Diff: main...kaluli123123:prime-agent:fix/coalesce-child-usage-attribution
All reactions