fix: checkpoint sync follow-head fixes for epbs-devnet-1 - #9148
Closed
lodekeeper wants to merge 10 commits into
Closed
fix: checkpoint sync follow-head fixes for epbs-devnet-1#9148lodekeeper wants to merge 10 commits into
lodekeeper wants to merge 10 commits into
Conversation
…ock sync After checkpoint sync + range sync, the head block's execution payload envelope may be missing (already gossipped before we connected). This leaves the head in PENDING state without a FULL variant. When new gossip blocks arrive expecting a FULL parent, they fail with BLOCK_ERROR_PRESTATE_MISSING / REGEN_ERROR_BLOCK_NOT_IN_FORKCHOICE and the node gets permanently stuck. Fix: In the PRESTATE_MISSING error handler, detect when the failure is caused by a missing FULL variant (using the existing Gloas retry context check) and proactively fetch the parent's envelope via reqresp before retrying. This reuses the existing resolveEnvelopeForBlock method which tries gossip cache first, then falls back to ExecutionPayloadEnvelopesByRoot. Tested: Local node on epbs-devnet-1 with checkpoint sync - fix triggers correctly, envelope fetched via reqresp, head advances and tracks chain.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the BlockInputSync logic in packages/beacon-node/src/sync/unknownBlock.ts to handle missing parent envelopes for Gloas blocks. When a PARENT_UNKNOWN or PRESTATE_MISSING error occurs, the system now proactively attempts to resolve the parent's envelope via reqresp if specific retry conditions are met, facilitating recovery after checkpoint or range syncs. I have no feedback to provide.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9315d81ae0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Address review feedback: getGloasInvalidStateRootRetryContext reads the default (PENDING) variant, so wantsFullParent can be true even when the FULL variant already exists. Gate resolveEnvelopeForBlock on an explicit getBlockHex(parentRoot, PayloadStatus.FULL) check.
Allow range sync to advance through epochs with 0 blocks (poor liveness periods). Previously, validateBlockByRangeResponse threw on empty responses, stalling the sync chain indefinitely. Changes: - Return empty result with warning instead of throwing on 0-block responses - Skip data validation when no blocks exist in the range - Always advance chain on successful batch processing, including empty batches
Two changes to improve genesis sync through post-Gloas devnet stretches: 1. requestByRange: fetch blocks FIRST, then columns/envelopes only if blocks were returned. Previously all 3 requests fired in parallel, causing 6 concurrent RPCs per peer (3 RPCs × 2 batches) which overwhelmed Prysm peers and triggered rate limiting. Now empty epochs use only 1 RPC, and non-empty epochs use block-first then data in parallel. 2. shouldTreatAsTransientDownloadError: add TTFB_TIMEOUT and BODY_TIMEOUT to transient error list so they trigger backoff retry instead of counting toward MAX_BATCH_DOWNLOAD_ATTEMPTS.
lodekeeper
force-pushed
the
epbs-devnet-1
branch
from
April 1, 2026 13:09
fc1a4af to
b97e077
Compare
…batches Keep the block-first request sequencing and transient timeout handling, but do not process a batch when blocks succeeded and follow-up columns/envelopes requests failed. Post-Gloas blocks may still require envelopes for the correct EMPTY vs FULL state path; treating transport failures as completed batches can surface INVALID_STATE_ROOT during sync. With this change: - fetch blocks first - skip columns/envelopes entirely for 0-block responses - if data fetch fails after blocks succeeded, retry the batch instead of processing partial results
When finalized range sync receives a boundary batch whose first block is one slot after the batch start and its parent root is unknown, fetch the missing parent block by root from the same peers, import it, and retry the batch immediately. This targets the epbs-devnet-1 genesis-sync failure where epoch 3 was served as 31 blocks (97..127) without slot 96, causing slot 97 to fail with BLOCK_ERROR_PARENT_UNKNOWN on the finalized target root.
When a request spans the finalized boundary, the finalized block may still be in hot storage during archive transitions. In that case the old handler could stream nothing from the archive for slot N and then continue with the live canonical chain from slot N+1, producing a malformed range response that skips the first existing block. Fix the handler to explicitly include the canonical finalized block from local storage when it falls in-range but is absent from the archive stream. Add a regression test covering the boundary-gap case for request 96..127, where the archive misses slot 96 and the live chain starts at slot 97.
If a finalized range batch starts after a peer's own finalized boundary root, that response is malformed for boundary processing and will later explode with PARENT_UNKNOWN on the first child block. Detect that case before processing, quarantine the offending peer, reset the batch for download, and retry with another peer instead of letting the chain loop on the same malformed boundary response.
For finalized sync, peer.target.slot is the peer's finalized checkpoint boundary, not a by-range serving ceiling. Using it as an upper bound can exclude a usable low-range peer once batch.startSlot exceeds that finalized checkpoint, even though the peer can still serve the batch. Plumb headSlot through PeerSyncMeta and use it as the upper-bound check for RangeSyncType.Finalized while keeping target.slot for head sync. Add regression coverage proving the same peer is eligible for finalized sync but ineligible for head sync at startSlot=128, plus retry-path and negative headSlot cases.
…arget promotion Add a deterministic Fulu-era SyncChain regression covering the scenario we could not prove reliably on the live devnet: - finalized chain target promoted high by checkpoint-synced peers - low-range peer joins later with earliestAvailableSlot=0 and headSlot>128 - finalized sync should still request the later batch (startSlot=128) This demonstrates that the pushed headSlot-based eligibility fix can keep a promoted finalized chain moving once a genuinely usable low-range peer actually joins the sync peer set.
lodekeeper
changed the title
Contributor
Author
|
Superseded by #9156 which contains the same core fixes with cleaner scope (5 commits, 4 files vs 10 commits, 15 files). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
After checkpoint sync on epbs-devnet-1, Lodestar can fail to start finalized range sync and follow head due to two independent client-side bugs:
Peer classification: when the local node is stalled behind the wall clock, a peer with higher
finalizedEpochand higherheadSlotcan be incorrectly classified asFullySyncedinstead ofAdvancedif its head falls within the slot-import tolerance range. This prevents finalized range sync from starting.Missing parent envelope: during unknown-block processing,
PRESTATE_MISSINGcan occur because the parent block's FULL variant (execution payload envelope) is still absent. The existing sync path did not proactively resolve the missing parent envelope before retrying, leaving the block stuck in the download queue.Changes
Peer sync classification fix (
remoteSyncType.ts,sync.ts)currentSlotparameter togetPeerSyncTypeFullySyncedshortcut whenremote.finalizedEpoch > local.finalizedEpoch, check whether the local head is stalled behind the wall clockAdvancedso range sync can beginMissing parent envelope recovery (
unknownBlock.ts)PRESTATE_MISSINGerror during block import, check whether the parent block's FULL variant is absentEvidence & Limitations
Live testing on epbs-devnet-1 with a genuine Advanced erigon/caplin peer showed that the classification fix causes Lodestar to correctly enter finalized range sync instead of staying in the fully-synced path.
That same session then hit a separate outgoing
beacon_blocks_by_rangeV2INVALID_REQUEST(SSZ_SNAPPY_ERROR_UNDER_SSZ_MIN_SIZE), which is not addressed by this PR. The direct-host repro window later became unstable (the same host anchor alternated between behind / peerless / refused states), so this PR does not claim a complete end-to-end live-devnet fix.This PR is limited to the two client-side logic fixes that are currently best supported by the available evidence. The downstream req/resp interop failure with Caplin remains a separate follow-up investigation.