[P2] Stop a catch-up sweep from swallowing a live command (including \stop) it read from history first #38

Activity

1. MatthewS65537

   added

   bugSomething isn't workingSomething isn't working

   on Oct 11, 2026
2. MatthewS65537 commented on Oct 11, 2026

   

   MatthewS65537

   on Oct 11, 2026

   OwnerAuthor

   More actions

   Proposed fix

   Approach: Give live delivery a short grace period. Neither history sweep classifies, retires or dispatches a message younger than LIVE_GRACE_MS (30s). Its socket event therefore makes the first claim, and history only acts on messages that genuinely never arrived. The race exists only because the scan's upper bound is "now". Lowering that bound removes the race in one place, with no new state, receipts or policy rules.

   Changes:

   src/slack/catchup.ts

   • Add the constant.
   • Set the bound-thread scan's upperTs to now - LIVE_GRACE_MS.
   • Skip the fetch entirely when nothing is outside the grace yet. Otherwise oldest > latest, e.g. a thread bound by a prompt a few seconds ago, whose cursor is prevTs(root).

   export const MAX_SNAPSHOT_CANDIDATES = 200;
   +/** History never classifies messages this young: their live event gets first claim (#38). */
   +export const LIVE_GRACE_MS = 30_000;
    …
        if (!scan) {
          const oldest = [thread.historyCursorTs, recovery.replayFloorTs, replayFloor(state.now())]
            .filter((ts): ts is string => !!ts).sort(compareTs).at(-1)!;
   -      scan = { upperTs: timestampFromMs(state.now()), afterTs: oldest, generation: recovery.bindingGeneration,
   +      const upperTs = timestampFromMs(state.now() - LIVE_GRACE_MS);
   +      if (compareTs(oldest, upperTs) > 0) continue; // nothing outside the live grace yet
   +      scan = { upperTs, afterTs: oldest, generation: recovery.bindingGeneration,
            intentVersion: recovery.intentVersion, candidates: [], complete: false };
        }

   • Apply the same horizon in sweepTopLevelMessages. A missed top-level command there is harmless, but a top-level prompt that history beats its live event to is currently run as a "history" replay and counted as a ghost replay. The watermark never passes the horizon, so younger messages are picked up next pass.

   -    const newer = page.messages.filter((m) => timestampMicros(m.ts) !== undefined && compareTs(m.ts, mark) > 0);
   +    const horizon = timestampFromMs(state.now() - LIVE_GRACE_MS);
   +    const newer = page.messages.filter((m) => timestampMicros(m.ts) !== undefined && compareTs(m.ts, mark) > 0
   +      && compareTs(m.ts, horizon) <= 0);

   The end-of-scan retireHistory(key, nextTs(scan.upperTs)) now moves the floor only to now - 30s. The live path's retired_history / owner_canceled checks therefore can no longer fire for a message whose event is still in flight.

   Tests:

   • test/router.test.ts (real router + sweepMissedMessages): a bound thread receives \verbose off posted 2s ago in a replies page.
     • After the sweep: fetchReplies got latest === timestampFromMs(NOW - LIVE_GRACE_MS), and no lastDecision or floor was recorded for it.
     • handleIncomingMessage(sameMsg) (live) returns "accepted" and the thread's verbose is "off".
     • A second case with \stop asserts the stop handler ran (abort called) instead of "canceled". This case is not prototyped; the \verbose case is.
   • test/catchup.test.ts: the same message is still classified (held historical_command) once the clock has moved past the grace. This guards that the grace delays recovery and does not disable it.
   • test/catchup.test.ts top-level: a root newer than the horizon is neither dispatched nor passed by the watermark.
   • Existing tests to update (prototype run: the four touched files plus the new tests pass):
     • catchup.test.ts "page-three stop": assert latest: timestampFromMs(NOW - LIVE_GRACE_MS).
     • "eight recent and two dormant slots": recent owner timestamps NOW - (LIVE_GRACE_MS + 1000).
     • "coalesces overlapping timer/reconnect calls": advance a mutable clock by LIVE_GRACE_MS before release, since a frozen clock now correctly skips the rerun.
     • start-reliability.test.ts "recovers a history-only prompt at ten seconds": it now recovers at the 40s tick.

   Risks / interactions:

   • A genuinely missed message is recovered about 30s later than today (first tick after ts + 30s). That is acceptable for a backstop.
   • Replay safety is unchanged. Claims, receipts, classifyRecovery and the one-task-at-a-time pendingRun gate are untouched; only the window moves.
   • A Slack redelivery more than 30s late (e.g. the 1-min retry after a socket blip) can still race. That case keeps today's "missed command is held" semantics.
   • The skipped-fetch continue still bumps scheduler.polled, so fairness rotation is unchanged.
   • Interacts with [P3] Count only owner-authored deliveries as live in the ghost detector #52: a ghost witness that counts only owner input would turn today's benign history-beats-live races into false ghost warnings. Land [P2] Stop a catch-up sweep from swallowing a live command (including \stop) it read from history first #38 first or together.
   • Interacts with [P3] Skip top-level roots the thread sweep already confirmed instead of re-dispatching them as replays #51 and [P2] Delete accepted receipts left behind when a sweep confirms a prompt whose run is still active #39: all three touch sweep() / sweepTopLevelMessages. Their hunks are disjoint and were prototyped together.

   Alternatives considered:

   • Write a durable "history held" tombstone receipt and let a live delivery of the same ts override it. This needs new disposition semantics in claimMessage / classifyRecovery and still leaves \stop canceled-but-not-aborted.
   • Make a history \stop actually abort the active run. Worth considering separately, but it does not fix \model / \verbose loss.

   Effort: S. Confidence: high

   ---

   Proposal only, not implemented. Sketches are against 22ce7ee.
3. MatthewS65537

   added a commit that references this issue on Oct 13, 2026

   Harden catch-up recovery and fix small command bugs (#33, #35, #36, #37…

   ...

   5fe4ac8