Skip to content

fix(epbs): always serve post-CL state for checkpoint sync, never FULL - #9030

Closed
lodekeeper wants to merge 6 commits into
ChainSafe:epbs-devnet-0from
lodekeeper:fix/epbs-missed-slot-checkpoint-serving
Closed

fix(epbs): always serve post-CL state for checkpoint sync, never FULL#9030
lodekeeper wants to merge 6 commits into
ChainSafe:epbs-devnet-0from
lodekeeper:fix/epbs-missed-slot-checkpoint-serving

Conversation

@lodekeeper

@lodekeeper lodekeeper commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes checkpoint state serving for post-Gloas finalized/justified lookups to always return the post-CL state (no payload applied). Ensures correct checkpoint sync behavior when epoch boundary slot 0 is missed.

Problem

Two issues combined to produce 404s on checkpoint state lookups:

  1. Regen caching bug: processSlotsToNearestCheckpoint() (called by prepareNextSlot) cached epoch boundary states using the head block's payload status (FULL). But processSlots() never processes payloads — the resulting state is always post-CL regardless of the source block's status.

  2. Missed slot edge case: When epoch slot 0 is missed, getCheckpointPayloadStatus() correctly reports EMPTY (processSlot clears executionPayloadAvailability). The API forces EMPTY lookups, but only the FULL variant existed in cache → 404.

Fix

Regen (regen.ts): Always cache post-Gloas epoch boundary states as EMPTY (payloadPresent=false). This is correct because processSlots() only does epoch transitions — never processes payloads.

API (utils.ts): For post-Gloas finalized/justified, force EMPTY lookup with no fallback to FULL. If EMPTY isn't found → 404 is the correct response. Serving a post-payload (FULL) state would break checkpoint sync for other clients. The regen fix guarantees the EMPTY variant is always cached.

Testing

  • pnpm lint
  • pnpm check-types ✅ (0 errors in beacon-node)
  • 7 unit tests covering:
    • getStateValidatorIndex — existing tests (restored)
    • getStateResponseWithRegen — 404 when only FULL variant exists (never serves post-payload state)
    • Post-Gloas missed-slot-0: verifies 404, not silent FULL fallback
    • Normal case (block at epoch boundary): EMPTY variant found on first try
    • Justified checkpoint: same EMPTY-only behavior
    • Regen caching: verifies EMPTY for Gloas epoch boundary states

Context

  • ethereum/beacon-APIs#572 — potuz, sproul, twoeths agree finalized/justified should always serve post-CL state
  • Discord #epbs discussion (2026-03-13): potuz flagged the missed-slot-0 edge case, barnabasbusa removed Lodestar from devnet checkpoint sync rotation

Notes

AI-assisted investigation and patch.

When epoch boundary slot 0 is missed (no block), the checkpoint state API
could 404 even though the state existed in cache:

Root cause: processSlotsToNearestCheckpoint (called by prepareNextSlot)
cached the epoch boundary state using the head block's payload status
(FULL). But getCheckpointPayloadStatus correctly reports EMPTY for
missed slots (processSlot clears executionPayloadAvailability). The API
forced EMPTY lookups, both the primary and defensive fallback searched
for EMPTY, and the FULL variant in cache was never tried.

Two fixes:
1. processSlotsToNearestCheckpoint: always cache post-Gloas epoch
   boundary states as EMPTY (false). processSlots() never processes
   payloads, so the result is always a post-CL block state regardless
   of the source block's payload status.
2. getStateResponseWithRegen: add opposite-variant fallback. When both
   forced EMPTY and original checkpoint status miss, try the opposite
   payload variant as a last resort before returning 404.

Ref: ethereum/beacon-APIs#572 discussion (potuz, sproul, twoeths agree
finalized/justified should always serve post-CL state)
@lodekeeper
lodekeeper requested a review from a team as a code owner March 13, 2026 11:55
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical bug in the beacon node's API where requests for finalized/justified checkpoint states would incorrectly return a 404 error when an epoch boundary slot (specifically slot 0 of an epoch) was missed. The root cause was an inconsistency in how epoch boundary states were cached and retrieved, particularly concerning their payload status (FULL vs. EMPTY). The changes ensure that states are cached with the correct payload status and that the API has a robust fallback to find the state even if the initial lookup variant is incorrect, thereby improving the reliability of checkpoint state serving.

Highlights

  • API State Retrieval: Extended the getStateResponseWithRegen function to include an additional fallback mechanism that attempts to retrieve checkpoint states with the opposite payload status (e.g., FULL if EMPTY failed) before returning a 404 error. This addresses a specific edge case where a state might exist in the cache with a different payload status than initially requested.
  • Checkpoint Caching Logic: Modified processSlotsToNearestCheckpoint to ensure that all post-Gloas epoch boundary states are consistently cached as EMPTY (payloadPresent=false). This prevents inconsistencies where states might be cached as FULL due to the head block's payload status, leading to cache misses for EMPTY lookups.
  • New Unit Tests: Added new unit tests in utils.test.ts to specifically cover the missed-slot-0 scenario, verifying the fix for the 404 bug. Another new test file, processSlotsCheckpointCaching.test.ts, was introduced to confirm the correct EMPTY caching behavior for Gloas epoch boundary states.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • packages/beacon-node/src/api/impl/beacon/state/utils.ts
    • Extended the getStateResponseWithRegen function to include a fallback for the opposite payload status when initial lookups miss, specifically addressing missed-slot-0 scenarios.
    • Updated comments to explain the new defensive fallback logic.
  • packages/beacon-node/src/chain/regen/regen.ts
    • Modified processSlotsToNearestCheckpoint to always cache post-Gloas epoch boundary states as EMPTY (payloadPresent=false).
    • Removed previous logic that allowed the caller's payloadPresent parameter to influence caching for post-Gloas states.
    • Updated comments to reflect the new caching strategy for post-Gloas epoch boundary states.
  • packages/beacon-node/test/unit/api/impl/beacon/state/utils.test.ts
    • Removed getStateValidatorIndex tests, focusing the file on getStateResponseWithRegen.
    • Added a new test suite for getStateResponseWithRegen specifically for post-Gloas checkpoint serving.
    • Introduced a test case to verify serving finalized state when slot-0 is missed and only the FULL variant exists in cache.
    • Added a test case to verify serving finalized state when slot-0 has a block (normal case, EMPTY variant exists).
    • Included a test case to verify serving justified state correctly for post-Gloas.
  • packages/beacon-node/test/unit/chain/regen/processSlotsCheckpointCaching.test.ts
    • Added a new test file to verify processSlotsToNearestCheckpoint post-Gloas checkpoint caching.
    • Included a test to confirm epoch boundary states are cached as EMPTY regardless of the caller's payloadPresent for post-Gloas.
    • Added a test to confirm pre-Gloas caching behavior remains unchanged (caches as FULL).
Activity
  • All linting and type checks (pnpm lint, pnpm check-types) passed successfully.
  • New unit tests were added for utils.test.ts to cover the missed-slot-0 scenario and for processSlotsCheckpointCaching.test.ts to verify EMPTY caching for Gloas.
  • All existing tests, including queued.closestHeadState.test.ts, passed.
  • The context for this fix includes discussions from ethereum/beacon-APIs#572 and a Discord #epbs discussion on March 13, 2026, where the missed-slot-0 edge case was flagged.
  • The investigation and patch for this issue were assisted by AI.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses a bug where checkpoint state APIs would return a 404 for missed epoch boundary slots. The two-pronged fix is sound: correctly caching post-Gloas epoch boundary states as EMPTY and adding a fallback mechanism to try the opposite payload variant in getStateResponseWithRegen. The changes are logical and well-supported by new unit tests that validate both the fix and the original bug scenario. My only concern is the removal of existing tests for an unrelated function, which I've flagged in a specific comment.

Comment thread packages/beacon-node/test/unit/api/impl/beacon/state/utils.test.ts Outdated
Restores the original getStateValidatorIndex unit tests that were
accidentally dropped when rewriting the test file for missed-slot
checkpoint serving tests. Also fixes a type error in the existing
getStateResponseWithRegen test by extracting the mock fn before
casting to never.
Remove both defensive fallbacks that could serve the post-payload (FULL)
state variant for finalized/justified checkpoint lookups.

For checkpoint sync, serving the wrong variant is worse than 404 — the
client retries with another server vs getting a state with payload
applied. The regen fix (always cache epoch boundary states as EMPTY)
ensures the EMPTY variant is always available, so the primary lookup
should always succeed.
@lodekeeper lodekeeper changed the title fix(epbs): serve correct checkpoint state when epoch slot-0 is missed fix(epbs): always serve post-CL state for checkpoint sync, never FULL Mar 13, 2026
…ed handler API

The handler was updated to use:
- forkChoice.getFinalizedBlock().slot (was getFinalizedCheckpointSlot)
- forkChoice.getHead() + getAllAncestorBlocks(head) (was getHeadRoot/getBlockHexDefaultStatus)
- db.executionPayloadEnvelopeArchive.getBinary() (was get(), returns raw bytes)
- chain.getSerializedExecutionPayloadEnvelope() (was db.executionPayloadEnvelope.get())

Update test mocks accordingly and rename test cases to reflect current behavior.
@lodekeeper

Copy link
Copy Markdown
Contributor Author

Squash-merged by Nico in c3cd2bb with lint/type fixes + payloadPresent propagation through processSlotsByCheckpoint. All core changes intact — closing.

@lodekeeper lodekeeper closed this Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant