Retire finished passive sources and responses so long-running capture stays bounded (Refs #4640)

.agents/skills/atlas/assets/seed/records/facb45d3084caa490d9b0ac2bcfcee89629a4c3db5926ec9256a9d27d25559ee.json

@@ -0,0 +1 @@
+{"citations":[{"path":"src/FS.GG.Coord.Cli/SkillTelemetryAdapter.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"c7b46e90e78b6690e63d31ab77ceca6a38daf94a991279cd51153ff076248586"},{"path":"src/FS.GG.Coord.Cli/SkillTelemetryReaders.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"f1fad4f089aa05a34b0d8a48c6c94f0005360580e59b70ab59cbb80d944b1feb"},{"path":"tests/FS.GG.Coord.Cli.Tests/WorkspaceTelemetryApplicationTests.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"a0a68520ba8b80f377c29e119be33a18035fca932f06fa65b42ef2003bc89570"},{"path":"docs/roadmaps/utel-operational-completeness.md","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"318b2deacb9283ef3343c2703fe4fe8966f2cbaa2434ab1ea06dfbb53134a85e"},{"path":"src/FS.GG.Coord.Cli/WorkspaceTelemetryApplication.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"7c1c889ceb8022f66650623016e6e71493b0eba6dc755272a62800095c8ef150"},{"path":"src/FS.GG.Coord.Cli/TelemetryStoreApplication.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"a7eca3888fc4fb7a1c3160d4e01ec8cd1b90daa8fee17db1f61c6fcd46d65ba1"}],"conflictsWith":[],"id":"github/passive-capture-finite-state-retirement","kind":"lesson","recordedAt":"2026-10-23T10:11:04Z","repository":"FS-GG/.github","schema":"fsgg.atlas/1","status":"recorded","summary":"Through coord 0.112 and the 0.113 source of #4802, one passive association kept at most 8 source cursors and 256 response identities and never removed either. A new path past 8 gave `source-index-full-capture-deferred`; a pass whose new identities would exceed 256 deferred without advancing, so every later pass rescanned and deferred again. Measured 2026-10-23 on the fdev host (capture-only, no network): two real Claude Code 2.1.309 transcripts reached 235 observations in ten passes and then stalled; the host wrote about 152 transcripts in 24 hours. Any passive producer with many files or one long session hits this.\n\nRetiring finished state fixes the stall but moves duplicate prevention from a permanent identity index to cursors, and that is the reusable lesson: the receiver refuses a repeated identity at a revision it already holds with a different content digest (`native fact identity conflict`) and quarantines the whole batch, and passive facts carry capturedAt and sourceGeneration, so any re-read differs. Independent review found five re-read paths after retirement: a hard link, a rename, a copy under a new inode, an active file replaced by a new inode, and a hard link closing in the same pass (which also crashed on a null index entry). Each needs explicit suppression with a visible gap.\n\nThe fix (Outcome #4640, this revision): a source closes at end of descriptor on a line boundary after 15 minutes unmodified (mtime taken after the read); its cursor becomes a compact retired entry (cap 384) and its identities leave the index; an active Claude transcript keeps its newest 24 identities. Suppression: `source-alias-skipped` (same inode and continuing the owning entry's anchored cursor, so a reused inode is not an alias), `closed-source-discontinuity-skipped` (marked, not rescanned), `continuity-replay-suppressed` (at or before a per-source high-water timestamp after a continuity gap), `retired-response-suppressed` (4096 most recently retired identities, 16-hex prefixes) and `pre-horizon-record-suppressed` (unknown source at or before one second after the newest evicted entry's mtime). Retirement rides the existing intent, so a crash applies it once.\n\nDrain throughput: a drain sends at most one envelope per passive chain, because later envelopes wait behind the earliest pending head, so drain must run at least as often as capture emits batches (at most 24 responses each). At 128 ready envelopes or 8 MiB, enqueue returns `spool-overload`, the intent and cursors are retained and capture stops; nothing queued is dropped.\n\nLimits: the 24-identity window relies on observed, undocumented Claude writer ordering; a copy of an old closed file made after its identities left the recent set and before its entry is evicted can still repeat responses (about two days at 150 transcripts and 10,000 responses a day); the caller must keep selecting active paths until they close; records without timestamps are suppressed where a timestamp bound applies; active Codex rollouts keep the no-retirement rule; rolling a retired association back to 0.112 would re-read closed sources. Tests are synthetic; installed adoption and receiver ingestion (Host built from 61526e94 or later) are separate.","supersedes":[],"tags":["utel","telemetry","passive","retirement","claude","transcript","finite-profile","index-full","re-emission","drain","coord-0.113"],"title":"Passive telemetry needs bounded retirement, and retirement needs explicit suppression of every re-read path"}

.claude/skills/atlas/assets/seed/records/facb45d3084caa490d9b0ac2bcfcee89629a4c3db5926ec9256a9d27d25559ee.json

@@ -0,0 +1 @@
+{"citations":[{"path":"src/FS.GG.Coord.Cli/SkillTelemetryAdapter.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"c7b46e90e78b6690e63d31ab77ceca6a38daf94a991279cd51153ff076248586"},{"path":"src/FS.GG.Coord.Cli/SkillTelemetryReaders.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"f1fad4f089aa05a34b0d8a48c6c94f0005360580e59b70ab59cbb80d944b1feb"},{"path":"tests/FS.GG.Coord.Cli.Tests/WorkspaceTelemetryApplicationTests.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"a0a68520ba8b80f377c29e119be33a18035fca932f06fa65b42ef2003bc89570"},{"path":"docs/roadmaps/utel-operational-completeness.md","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"318b2deacb9283ef3343c2703fe4fe8966f2cbaa2434ab1ea06dfbb53134a85e"},{"path":"src/FS.GG.Coord.Cli/WorkspaceTelemetryApplication.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"7c1c889ceb8022f66650623016e6e71493b0eba6dc755272a62800095c8ef150"},{"path":"src/FS.GG.Coord.Cli/TelemetryStoreApplication.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"a7eca3888fc4fb7a1c3160d4e01ec8cd1b90daa8fee17db1f61c6fcd46d65ba1"}],"conflictsWith":[],"id":"github/passive-capture-finite-state-retirement","kind":"lesson","recordedAt":"2026-10-23T10:11:04Z","repository":"FS-GG/.github","schema":"fsgg.atlas/1","status":"recorded","summary":"Through coord 0.112 and the 0.113 source of #4802, one passive association kept at most 8 source cursors and 256 response identities and never removed either. A new path past 8 gave `source-index-full-capture-deferred`; a pass whose new identities would exceed 256 deferred without advancing, so every later pass rescanned and deferred again. Measured 2026-10-23 on the fdev host (capture-only, no network): two real Claude Code 2.1.309 transcripts reached 235 observations in ten passes and then stalled; the host wrote about 152 transcripts in 24 hours. Any passive producer with many files or one long session hits this.\n\nRetiring finished state fixes the stall but moves duplicate prevention from a permanent identity index to cursors, and that is the reusable lesson: the receiver refuses a repeated identity at a revision it already holds with a different content digest (`native fact identity conflict`) and quarantines the whole batch, and passive facts carry capturedAt and sourceGeneration, so any re-read differs. Independent review found five re-read paths after retirement: a hard link, a rename, a copy under a new inode, an active file replaced by a new inode, and a hard link closing in the same pass (which also crashed on a null index entry). Each needs explicit suppression with a visible gap.\n\nThe fix (Outcome #4640, this revision): a source closes at end of descriptor on a line boundary after 15 minutes unmodified (mtime taken after the read); its cursor becomes a compact retired entry (cap 384) and its identities leave the index; an active Claude transcript keeps its newest 24 identities. Suppression: `source-alias-skipped` (same inode and continuing the owning entry's anchored cursor, so a reused inode is not an alias), `closed-source-discontinuity-skipped` (marked, not rescanned), `continuity-replay-suppressed` (at or before a per-source high-water timestamp after a continuity gap), `retired-response-suppressed` (4096 most recently retired identities, 16-hex prefixes) and `pre-horizon-record-suppressed` (unknown source at or before one second after the newest evicted entry's mtime). Retirement rides the existing intent, so a crash applies it once.\n\nDrain throughput: a drain sends at most one envelope per passive chain, because later envelopes wait behind the earliest pending head, so drain must run at least as often as capture emits batches (at most 24 responses each). At 128 ready envelopes or 8 MiB, enqueue returns `spool-overload`, the intent and cursors are retained and capture stops; nothing queued is dropped.\n\nLimits: the 24-identity window relies on observed, undocumented Claude writer ordering; a copy of an old closed file made after its identities left the recent set and before its entry is evicted can still repeat responses (about two days at 150 transcripts and 10,000 responses a day); the caller must keep selecting active paths until they close; records without timestamps are suppressed where a timestamp bound applies; active Codex rollouts keep the no-retirement rule; rolling a retired association back to 0.112 would re-read closed sources. Tests are synthetic; installed adoption and receiver ingestion (Host built from 61526e94 or later) are separate.","supersedes":[],"tags":["utel","telemetry","passive","retirement","claude","transcript","finite-profile","index-full","re-emission","drain","coord-0.113"],"title":"Passive telemetry needs bounded retirement, and retirement needs explicit suppression of every re-read path"}

docs/atlas/records/facb45d3084caa490d9b0ac2bcfcee89629a4c3db5926ec9256a9d27d25559ee.json

@@ -0,0 +1 @@
+{"citations":[{"path":"src/FS.GG.Coord.Cli/SkillTelemetryAdapter.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"c7b46e90e78b6690e63d31ab77ceca6a38daf94a991279cd51153ff076248586"},{"path":"src/FS.GG.Coord.Cli/SkillTelemetryReaders.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"f1fad4f089aa05a34b0d8a48c6c94f0005360580e59b70ab59cbb80d944b1feb"},{"path":"tests/FS.GG.Coord.Cli.Tests/WorkspaceTelemetryApplicationTests.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"a0a68520ba8b80f377c29e119be33a18035fca932f06fa65b42ef2003bc89570"},{"path":"docs/roadmaps/utel-operational-completeness.md","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"318b2deacb9283ef3343c2703fe4fe8966f2cbaa2434ab1ea06dfbb53134a85e"},{"path":"src/FS.GG.Coord.Cli/WorkspaceTelemetryApplication.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"7c1c889ceb8022f66650623016e6e71493b0eba6dc755272a62800095c8ef150"},{"path":"src/FS.GG.Coord.Cli/TelemetryStoreApplication.fs","repository":"FS-GG/.github","revision":"6e8167ec4d7b4fd6f0cecf932a6f0811cf65b50f","sha256":"a7eca3888fc4fb7a1c3160d4e01ec8cd1b90daa8fee17db1f61c6fcd46d65ba1"}],"conflictsWith":[],"id":"github/passive-capture-finite-state-retirement","kind":"lesson","recordedAt":"2026-10-23T10:11:04Z","repository":"FS-GG/.github","schema":"fsgg.atlas/1","status":"recorded","summary":"Through coord 0.112 and the 0.113 source of #4802, one passive association kept at most 8 source cursors and 256 response identities and never removed either. A new path past 8 gave `source-index-full-capture-deferred`; a pass whose new identities would exceed 256 deferred without advancing, so every later pass rescanned and deferred again. Measured 2026-10-23 on the fdev host (capture-only, no network): two real Claude Code 2.1.309 transcripts reached 235 observations in ten passes and then stalled; the host wrote about 152 transcripts in 24 hours. Any passive producer with many files or one long session hits this.\n\nRetiring finished state fixes the stall but moves duplicate prevention from a permanent identity index to cursors, and that is the reusable lesson: the receiver refuses a repeated identity at a revision it already holds with a different content digest (`native fact identity conflict`) and quarantines the whole batch, and passive facts carry capturedAt and sourceGeneration, so any re-read differs. Independent review found five re-read paths after retirement: a hard link, a rename, a copy under a new inode, an active file replaced by a new inode, and a hard link closing in the same pass (which also crashed on a null index entry). Each needs explicit suppression with a visible gap.\n\nThe fix (Outcome #4640, this revision): a source closes at end of descriptor on a line boundary after 15 minutes unmodified (mtime taken after the read); its cursor becomes a compact retired entry (cap 384) and its identities leave the index; an active Claude transcript keeps its newest 24 identities. Suppression: `source-alias-skipped` (same inode and continuing the owning entry's anchored cursor, so a reused inode is not an alias), `closed-source-discontinuity-skipped` (marked, not rescanned), `continuity-replay-suppressed` (at or before a per-source high-water timestamp after a continuity gap), `retired-response-suppressed` (4096 most recently retired identities, 16-hex prefixes) and `pre-horizon-record-suppressed` (unknown source at or before one second after the newest evicted entry's mtime). Retirement rides the existing intent, so a crash applies it once.\n\nDrain throughput: a drain sends at most one envelope per passive chain, because later envelopes wait behind the earliest pending head, so drain must run at least as often as capture emits batches (at most 24 responses each). At 128 ready envelopes or 8 MiB, enqueue returns `spool-overload`, the intent and cursors are retained and capture stops; nothing queued is dropped.\n\nLimits: the 24-identity window relies on observed, undocumented Claude writer ordering; a copy of an old closed file made after its identities left the recent set and before its entry is evicted can still repeat responses (about two days at 150 transcripts and 10,000 responses a day); the caller must keep selecting active paths until they close; records without timestamps are suppressed where a timestamp bound applies; active Codex rollouts keep the no-retirement rule; rolling a retired association back to 0.112 would re-read closed sources. Tests are synthetic; installed adoption and receiver ingestion (Host built from 61526e94 or later) are separate.","supersedes":[],"tags":["utel","telemetry","passive","retirement","claude","transcript","finite-profile","index-full","re-emission","drain","coord-0.113"],"title":"Passive telemetry needs bounded retirement, and retirement needs explicit suppression of every re-read path"}

docs/roadmaps/utel-operational-completeness.md

@@ -56,6 +56,42 @@ a 256-response revision index and 1 MiB private state. Existing spool bounds rem
 envelopes/8 MiB with bounded outcome retention. Partial tails defer; truncation/replacement creates a
 new source generation and continuity gap. Cumulative turn snapshots never become response spend.
 
+Finished-source retirement (2026-10-23, Outcome #4640) keeps one long-running association inside
+these bounds. A source closes when a pass reads its descriptor to the end on a line boundary and the
+file has been unmodified for 15 minutes. Its cursor moves to a compact retired entry and its response
+identities leave the index. A later append resumes from that entry. An active Claude transcript keeps only
+its newest 24 response identities, relying on the observed (undocumented) writer order in which every content
+block of one API response precedes the next response; active Codex rollouts keep the original rule.
+
+Re-emitting a retired identity would be refused by the receiver as an identity conflict and quarantine the
+whole batch, so each re-read path suppresses records instead, with a visible gap:
+`source-alias-skipped` for the same file under another path (same inode and continuing the owning entry's
+anchored cursor, so a reused inode is not mistaken for an alias); `closed-source-discontinuity-skipped` for a
+closed file that no longer continues its entry (marked, and not rescanned); `continuity-replay-suppressed`
+for an active file re-read after a continuity gap, at or before its high-water timestamp;
+`retired-response-suppressed` for any identity in the bounded set of the 4096 most recently retired; and
+`pre-horizon-record-suppressed` for an unknown source's records at or before the eviction horizon, which is
+one second after the modification time of the newest evicted entry (at most 384 entries are kept).
+Records without a timestamp are suppressed wherever a timestamp bound applies. One case remains: a copy of
+an old closed file under a new inode, made after its identities left the recent set but before its entry is
+evicted, can repeat responses and would be quarantined at the receiver. At about 150 transcripts and
+10,000 responses a day, the recent set covers roughly nine hours and the retired entries roughly two and a
+half days, so that window is about two days. A source leaves the active set only when a pass closes it, so
+the caller must keep selecting each active path until it closes; eight abandoned active paths (for example,
+deleted while active) block new sources with `source-index-full-capture-deferred`. Retirement travels in the existing
+intent, so a crash applies it once. Measured before the change, two real transcripts filled the
+256-identity index in ten passes and every later pass deferred.
+
+Spool backpressure never drops queued observations. At 128 ready envelopes or 8 MiB the local enqueue
+returns `spool-overload`; the pass keeps its intent, records `enqueue-deferred-cursor-retained` and the
+failure, and advances no cursor, so capture stops and resumes from the same records after a drain. Ready
+envelopes are never evicted; only settled (applied, rejected or expired) outcome receipts rotate. A pass emits
+at most one batch of 24 responses, while a drain sends at most one envelope of each passive chain, because a
+later envelope waits behind its earliest pending predecessor. Drain must therefore run at least as often as
+capture produces a batch. Observations can still be lost, visibly, when a source file disappears while
+capture is held (`source-unavailable-cursor-retained`), through the suppressions above, or when a conflicting
+batch is quarantined by the receiver.
+
 Existing `activate-remote --capture-only` is an explicit initial local setup option. It retains
 configuration, permission, conflict and lock checks while skipping credential presence; its result is
 capture-configured/receiver-unverified. Default activation and later credential/receipt checks remain