sync_service: bound AwaitingWarp wait with warp progress detection#3300
sync_service: bound AwaitingWarp wait with warp progress detection#3300lrubasze wants to merge 3 commits into
Conversation
A warp sync that stays alive but never completes kept queued subscribe_all requests waiting indefinitely (livelock re-arming the deadline, or a starve/re-trigger cycle between Deciding and AwaitingWarp). Track warp's verified finalized height across deadline firings; after 3 windows without progress, commit AllForksOnly so subscribers get the checkpoint header. Warp stays resumable, so a later completion still reaches subscribers via Stop + resubscribe. Resolves TODO from #3268 (discussion_r3319656011).
| /// Deadline firings (`MODE_DECISION_TIMEOUT` apart) without warp progress before committing | ||
| /// AllForksOnly, so queued `SubscribeAll` requests always receive a response (the checkpoint | ||
| /// header). | ||
| const MODE_DECISION_MAX_WARP_STALLS: usize = 3; |
There was a problem hiding this comment.
Open question: how many stalled windows to wait before falling back?
Current draft waits 3 stalled windows (~90s). I'm considering 1 (~30s): a healthy warp normally shows progress within one window, and committing "too early" is cheap - subscribers get the checkpoint header, warp keeps running, and on completion they get Stop + resubscribe at the warped head anyway. It also removes the stall counter and the AwaitingWarp → Deciding fallback entirely.
WDYT?
There was a problem hiding this comment.
Went with 1 window. Committing early costs nothing: warp keeps running and its completion still resets subscriptions at the warped head. So waiting longer has no benefit. The stall counter and the fallback to Deciding are removed.
Small edge that remains: if warp makes some progress every window, we never commit and subscribers wait until warp finishes. If we ever want a hard cap on that wait, we can commit on the first deadline regardless of progress.
A 30s window without warp progress now commits AllForksOnly directly, dropping the 3-window stall counter and the AwaitingWarp -> Deciding starve fallback. Committing early is cheap: warp stays unsuppressed and a later completion still resets subscriptions at the warped head. - keep the pending-completion check so a suppressed warp completion is never discarded by the commit - baseline last_warp_progress on AwaitingWarp entry so the window measures warp activity after entry, not progress left over from Deciding
1 participant
Summary
Resolves a TODO from #3268 (comment): a warp sync that stays alive but never completes kept queued
subscribe_allrequests waiting forever - theAwaitingWarpdeadline re-armed as long as warp looked alive, and the starve/re-trigger cycle throughDecidingreset the timer each round.Changes
warp_sync_progress()), baselined onAwaitingWarpentry; each 30s deadline firing with progress re-arms the deadline.AwaitingWarp→Decidingstarve fallback is removed; the stall commit covers that case.Known edge: a warp that advances at least one fragment per window never triggers the fallback, so subscribers wait until it completes.