Skip to content

Retire finished passive sources and responses so long-running capture stays bounded (Refs #4640) - #4835

Merged
EHotwagner merged 12 commits into
mainfrom
routine/passive-state-retirement
Oct 23, 2026
Merged

EHotwagner merged 12 commits into
mainfrom
routine/passive-state-retirement

Conversation

@EHotwagner

@EHotwagner EHotwagner commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

A passive telemetry association kept at most 8 source cursors and 256 response identities and never retired either. On a host that writes about 150 Claude Code transcripts a day it deferred permanently after a few files. Measured capture-only on real 2.1.309 transcripts: two files filled the index in ten passes, and every later pass returned response-index-full-capture-deferred. This lands before the coherent113 candidate binds, so the fix ships in 0.113.0.

Change (SkillTelemetryAdapter.capturePassiveFromWithHook):

  • Closing a source. A source closes when a pass reads its descriptor to the end on a line boundary, with no pass-budget, partial-tail or correction deferral, and the file has been unmodified for 15 minutes. PassiveScan gains SourceLength and ModifiedAtUnixSeconds. The cursor moves to a compact retired entry, and every response identity from that source instance leaves the index.
  • Reselecting a closed source. A continuous append resumes from the retired entry. A rewritten file whose anchor no longer matches gets closed-source-discontinuity-skipped and is never re-read from zero.
  • Active Claude transcripts keep only their newest 24 identities. The writer finishes all content blocks of one response before the next begins. Active Codex rollouts keep the original rule.
  • Bound and horizon. At most 384 retired entries. Evicting the oldest sets a horizon one second after that file's modification time. Records from an unknown source at or before the horizon are suppressed (pre-horizon-record-suppressed) instead of being counted twice. Re-emitting an identity the receiver already holds would be refused as an identity conflict.
  • Crash safety. Retirement rides the existing intent as absolute values (retirementAfter), so a crash at any cut applies it exactly once.
  • Snapshot gains closedSources and retiredResponses.

Tests (WorkspaceTelemetryApplicationTests, 33/33; full FS.GG.Coord.Cli.Tests 754/754):

  • 600 responses in one active session with no re-emission; the index holds 24.
  • 20 files selected eight at a time: all close, resume reads only the appended responses, and a rewrite is skipped.
  • Every crash cut during a closing pass: retirement is applied once and nothing is emitted twice.
  • A three-day simulation: 451 files plus a main session growing each day, 1710 responses, each exactly once. State stays under 1 MiB, 384 retired entries, at most 256 identities. An evicted file reselected is suppressed, and a new file after the horizon is captured.

Also included:

  • the 0.113.0 CLI release-note line
  • roadmap paragraphs on retirement, re-read suppression and spool backpressure / drain cadence (utel-operational-completeness.md)
  • the Atlas lesson github/passive-capture-finite-state-retirement (digest f5e33dfd…) in docs/atlas and both seeds, with the driver manifest regenerated

Review repair (after REQUEST CHANGES at 757a7c8b). Every re-read path of a retired response now suppresses instead of re-emitting, because the receiver would refuse a re-emitted identity as an identity conflict and quarantine the whole batch:

  • source-alias-skipped: another active or retired entry already owns the same file (hard link, rename), meaning the same inode and the owner's anchored cursor still continues, so a reused inode is not taken as an alias. This also fixes a null-entry crash.
  • continuity-replay-suppressed: an active file re-read after a continuity gap, at or before its high-water timestamp.
  • retired-response-suppressed: an identity among the 4096 most recently retired (copies).
  • closed-source-discontinuity-skipped: such an entry is now marked and not rescanned.

Quiescence takes the mtime after the read. The roadmap corrects three things: drain throughput is one envelope per passive chain per drain, settled receipts (not only applied ones) rotate, and the loss paths are listed. The Atlas record is replaced (now facb45d3…; earlier in-PR records never reached main). The release notes gain a rollback warning. After critic ACCEPT at 3b58ec3f: the alias check now requires anchored continuity, the replaced-inode test renames over the original to guarantee a new inode, the roadmap states the about-two-day copy window and the obligation to keep selecting active paths, and origin/main is merged. Tests: 34/34 targeted (a new alias/copy/rewrite regression fails on the previous adapter) and 755/755 for the full CLI suite.

Limits: the 24-identity window relies on observed Claude writer ordering, and a correction to an older response would surface as a receiver conflict. A single active Codex rollout with more than 256 responses still stalls until it closes. Installed adoption and Host ingestion remain separate.

Merge is held: the .github main merge hold (Wizard 0.22.0) is in effect, and the fdev sequence puts this before #4833.

architecture-map: unaffected

Refs #4640

🤖 Generated with Claude Code

EHotwagner and others added 12 commits October 23, 2026 11:33
… stays bounded

A passive association kept at most 8 source cursors and 256 response
identities and never retired either, so on a host writing about 150
Claude transcripts a day it deferred permanently after a few files
(measured: two real transcripts saturated the index in ten passes).

A source now closes when a pass reads it to the end on a line boundary
and it has been unmodified for 15 minutes; its cursor moves to a compact
retired entry (resumable on append, never re-read on rewrite) and its
response identities leave the index. Active Claude transcripts keep their
newest 24 identities. Retired entries are capped at 384; eviction sets a
horizon below which unknown-source records are suppressed instead of
counted twice. Retirement rides the existing intent so a crash applies it
once. Tests cover >256 responses, >8 files, resume/rewrite, crash cuts and
a three-day 451-file simulation.

Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-add the passive retirement Atlas record to the merged catalogs and
regenerate the driver manifest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found that retirement made duplicate prevention depend on a
path-keyed cursor still lining up. A hard link, rename, copy or rewrite
could re-emit an identity the receiver already holds at revision 1, which
it refuses as an identity conflict and quarantines with the whole batch,
and a hard link closing in the same pass crashed on a null index entry.

Skip a path whose inode another active or retired entry already owns
(source-alias-skipped); keep a per-source high-water timestamp and
suppress records at or before it after a continuity gap
(continuity-replay-suppressed); keep the 4096 most recently retired
identities and suppress them on any re-read (retired-response-suppressed);
mark a discontinuous closed entry so it is not rescanned; guard the null
in the correction check; and take the quiescence time from the descriptor
after the read. Correct the roadmap claims about drain throughput, outcome
rotation and loss, and add a rollback note to the release notes.

Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d lesson

The earlier in-PR record overstated duplicate prevention; this one names
the re-read paths, their suppressions, drain throughput and the remaining
limits. The replaced record never reached main.

Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A deleted transcript's inode can be reused by a new file; the alias check
now also requires the candidate to continue the owning entry's anchored
cursor. Make the replaced-inode test rename a fresh file over the original
so the new inode is guaranteed, and state the remaining copy window and
the caller's obligation to keep selecting active paths until they close.

Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… alias check

Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wns the docs

The signature-doc-siting gate refuses /// comments in an implementation
file with a sibling .fsi; the field docs live in SkillTelemetryReaders.fsi.

Refs #4640

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EHotwagner
EHotwagner merged commit 5868f42 into main Oct 23, 2026
80 of 83 checks passed
@EHotwagner
EHotwagner deleted the routine/passive-state-retirement branch October 23, 2026 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant