fix: handle previous-epoch payload timeliness committees by lodekeeper · Pull Request #8983 · ChainSafe/lodestar

lodekeeper · 2026-03-04T16:39:47Z

Summary

Fixes an epoch-boundary bug in payload timeliness committee lookup.

EpochCache.getPayloadTimelinessCommittee(slot) previously only served committees for epochCtx.epoch.
At epoch boundaries, payload attestation validation can reference slot - 1 (previous epoch), which caused:

Payload Timeliness Committee is not available for slot=<prevEpochSlot>

This patch:

caches previous-epoch payload timeliness committees in epoch cache,
shifts current -> previous on epoch transition,
serves committees for current or previous epoch slots,
still throws for slots older than previous epoch.

Changes

packages/state-transition/src/cache/epochCache.ts
- add previous-epoch PTC cache handling
- update getPayloadTimelinessCommittee(slot) lookup logic
packages/state-transition/test/unit/cache/epochCache.test.ts
- add coverage for previous/current/too-old slot behavior

Validation

Local

pnpm --filter @lodestar/state-transition lint
pnpm --filter @lodestar/state-transition test:unit -- test/unit/cache/epochCache.test.ts

Devnet evidence (local)

Mixed Teku+Lodestar rerun with fixed image: original PTC error is gone in Lodestar logs.
Lodestar-only control run with this image reaches justification/finalization (justified=3, finalized=2 by slot 130), with no PTC errors and no VC misses.

Notes on remaining mixed-client instability

In my local mixed-client and Teku-only control runs, finalization remained unstable/absent due separate interop/topology behavior (outside this patch scope). This PR is scoped to the PTC root-cause fix only.

AI-assisted: drafted and validated with OpenClaw; final patch and test assertions reviewed by me.

…tees

gemini-code-assist · 2026-03-04T16:41:16Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical bug in the EpochCache where PayloadTimelinessCommittee lookups would fail at epoch boundaries when attestation validation referenced slots from the previous epoch. By introducing a mechanism to cache and retrieve committees for both the current and previous epochs, the system can now correctly handle these cross-epoch validations, preventing errors and improving the stability of attestation processing.

Highlights

Payload Timeliness Committee (PTC) Caching: Introduced a new cache to store Payload Timeliness Committees for the previous epoch, addressing issues with lookups across epoch boundaries.
Epoch Transition Handling: Implemented logic to correctly shift the current epoch's PTCs into the previous-epoch cache when a new epoch begins.
Enhanced PTC Lookup Logic: Modified the getPayloadTimelinessCommittee method to accurately retrieve committees for both the current and the immediately preceding epoch.
Improved Error Handling: Ensured that requests for PTCs from epochs older than the previous one now correctly result in an error, preventing invalid lookups.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

packages/state-transition/src/cache/epochCache.ts
- Added a new property previousPayloadTimelinessCommittees to store committees from the prior epoch.
- Updated the IEpochCacheData interface, constructor, and data serialization methods (fromData, toData) to include the new previous epoch committee cache.
- Modified the newEpochCache method to initialize the previousPayloadTimelinessCommittees array.
- Implemented logic within the epoch transition handler to move the current payloadTimelinessCommittees to previousPayloadTimelinessCommittees.
- Revised getPayloadTimelinessCommittee to first check the current epoch's committees, then the previous epoch's, and throw an error for older epochs.
packages/state-transition/test/unit/cache/epochCache.test.ts
- Added a new test file to specifically cover the getPayloadTimelinessCommittee method's behavior across epoch boundaries.
- Included tests to verify correct retrieval for previous and current epoch slots, and proper error handling for slots older than the previous epoch.

Activity

The initial draft and validation of the changes were assisted by OpenClaw.
The final patch and test assertions were reviewed by the author.
Local validation included linting and unit tests for the state-transition package.
Devnet evidence showed the original PTC error was resolved in Lodestar logs during mixed Teku+Lodestar runs.
A Lodestar-only control run with the fixed image achieved justification and finalization without PTC errors or VC misses.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request addresses an epoch-boundary bug by caching and serving payload timeliness committees for the previous epoch. While the overall implementation is clean and includes good unit test coverage, a critical vulnerability exists: the initialization logic in EpochCache.createFromState is incomplete. It fails to populate the previous-epoch committees when the cache is created from a state, which could lead to validation failures and Denial of Service at epoch boundaries after a node restart.

lodekeeper · 2026-03-04T17:05:15Z

@gemini-code-assist good catch — you were right about EpochCache.createFromState() not initializing previous-epoch PTC.

I pushed commit 0a7af31949 to address this:

when creating epoch cache from state, if !isGenesis && previousEpoch >= GLOAS_FORK_EPOCH, we now precompute previousPayloadTimelinessCommittees from previousShuffling.committees
this closes the restart/fresh-cache epoch-boundary gap for slot N / N-1 payload attestation validation

Validation rerun:

pnpm --filter @lodestar/state-transition lint
pnpm --filter @lodestar/state-transition test:unit -- test/unit/cache/epochCache.test.ts

Thanks again for flagging it.

gemini-code-assist · 2026-03-04T17:07:06Z

That's great to hear, @lodekeeper! I'm glad the feedback was helpful and that you've addressed the initialization of previousPayloadTimelinessCommittees in EpochCache.createFromState(). The explanation of the fix and the rerun validation steps are much appreciated. Thanks for the update!

lodekeeper · 2026-03-04T17:07:41Z

Closing for now per latest scope decision (Option 2): continue debugging mixed-client finalization/peering until acceptance criteria are met, then reopen/retarget with consolidated evidence.

nflaig · 2026-03-04T17:22:17Z

@lodekeeper why did you close this?

lodekeeper · 2026-03-04T17:49:24Z

@nflaig I closed it because of the latest scope decision (Option 2): keep debugging mixed-client finalization/peering first and only reopen once acceptance criteria are met.\n\nYou're right that I should have replied to your question immediately — sorry about that delay.\n\nIf you prefer, I can reopen this now and keep it as draft while we continue the interop investigation.

lodekeeper · 2026-03-04T18:00:18Z

Reopening per maintainer request so other maintainers can review this scoped PTC fix while we continue mixed-client interop debugging in parallel.

lodekeeper · 2026-03-04T18:24:35Z

Interop update while this PR is open for scoped PTC review:\n\n- Re-validated that PTC regression remains fixed in mixed-client runs (0 occurrences of Payload Timeliness Committee is not available for slot on Lodestar CL logs).\n- Ran topology mitigation experiment (1×Teku + 3×Lodestar, 6s slots): Lodestar-majority side reached justified=3/finalized=2, while the Teku node continued showing No peers for message topics instability.\n- This continues to point at a mixed interop/Teku-side liveness issue rather than a Lodestar PTC-cache regression.\n\nEvidence: notes/epbs-devnet-0/runs/mitigation2-teku1-lodestar3-20260304T180824Z/SUMMARY.md

lodekeeper · 2026-03-04T18:52:27Z

Option-2 interop update (new mitigation run):

Ran mitigation Greg/refactor #3 in a fresh 2xTeku + 2xLodestar setup with denser topology (supernode=true on all CLs, Lodestar targetPeers=20).
Result: finalization still did not resume. One Teku node (cl-2-teku) isolated to peers=0 and stalled at slot 95, while other nodes advanced to ~180.
Teku still shows repeated "No peers for message topics" plus bid-parent import errors; Lodestar still shows 0 "Payload Timeliness Committee is not available for slot" (PTC fix remains intact).

Evidence: notes/epbs-devnet-0/runs/mitigation3-supernode-2x2-20260304T183130Z/SUMMARY.md

lodekeeper · 2026-03-04T19:03:58Z

Final scope/status update:

Lodestar-side PTC issue is resolved in this PR scope (epoch-boundary previous-epoch committee lookup + createFromState() initialization fix).
Validation is complete (unit tests + devnet evidence, including minimal preset with zero Payload Timeliness Committee is not available for slot errors).
Remaining mixed-client finalization instability appears external/Teku-side based on the mitigation matrix and is out of scope for this Lodestar PTC fix.

Per latest maintainer direction, treating this task as done-for-now on Lodestar side. I will only provide normal review follow-up on this PR if additional comments/questions arrive.

nflaig · 2026-03-05T11:50:43Z

+      // Populate previous-epoch committees when creating cache from an in-memory state.
+      // This is needed for epoch-boundary lookups (slot N block validating slot N-1 attestation)
+      // after restart / fresh cache initialization.
+      if (!isGenesis && previousEpoch >= config.GLOAS_FORK_EPOCH) {


why is isGenesis required here?

Good call — it is not strictly required.

I simplified this in fb9cbaeaac by removing isGenesis from the guard and keeping only previousEpoch >= config.GLOAS_FORK_EPOCH.

That keeps pre-fork behavior unchanged, and for genesis with fork-at-0 it just computes the same committee set (harmless) instead of special-casing.

what's better is to do the following, similar to shuffling

const previousShuffling =
cachedPreviousShuffling ??
(isGenesis
? currentShuffling
: computeEpochShuffling(state, new Uint32Array(previousActiveIndicesAsNumberArray), previousEpoch));

ah, maybe it was fine, let's re-add the is genesis check

Good call — agreed. The final version now uses exactly that shuffling-style fallback for previousShuffling (cachedPreviousShuffling ?? (isGenesis ? currentShuffling : computeEpochShuffling(...))). This is present in #8991 (the PR carrying this fix to unstable), and #8983 was closed in favor of that cherry-pick path.

Yep — agreed, and this ended up reflected in the follow-up path. This PR is now closed/superseded, but in the merged replacement (#8991) the genesis guard is present again in EpochCache.createFromState (previousShuffling = isGenesis ? currentShuffling : computeEpochShuffling(...)).

nflaig · 2026-03-05T11:54:41Z

+      // Populate previous-epoch committees when creating cache from an in-memory state.
+      // This is needed for epoch-boundary lookups (slot N block validating slot N-1 attestation)
+      // after restart / fresh cache initialization.


can we move this comment to previousPayloadTimelinessCommittees type

Done in fb9cbaeaac — I moved that explanatory comment up to the previousPayloadTimelinessCommittees declaration so it documents the field directly.

nflaig · 2026-03-05T12:33:08Z

@lodekeeper I cherry picked your changes in #8991 to bring them directly to our unstable branch. please review that PR to make sure it's correct with your implementation and if it is please close this PR, I will cherry-pick my commit to epbs-devnet-0 branch later

lodekeeper · 2026-03-05T16:44:25Z

@nflaig Confirmed #8991 carries the implementation correctly to , so I’m closing this PR as requested. Thanks for cherry-picking this over.

lodekeeper · 2026-03-05T16:44:44Z

@nflaig (typo fix from my prior close comment) Confirmed #8991 carries the implementation correctly to unstable; closing #8983 as requested was correct.

We need the PTC cached for previous epoch in case of epoch boundary lookups, eg. simple case is slot N block validating slot N-1 attestation ~~but also if there are deeper reorgs / missed slots.~~ (this isn't required as ptc messages are only valid for one slot). See #8983 for more details.

We need the PTC cached for previous epoch in case of epoch boundary lookups, eg. simple case is slot N block validating slot N-1 attestation ~~but also if there are deeper reorgs / missed slots.~~ (this isn't required as ptc messages are only valid for one slot). See ChainSafe#8983 for more details.

fix(state-transition): cache previous epoch payload timeliness commit…

ba3fb81

…tees

lodekeeper requested a review from a team as a code owner March 4, 2026 16:39

gemini-code-assist Bot reviewed Mar 4, 2026

View reviewed changes

fix(state-transition): init previous-epoch ptc on cache creation

0a7af31

lodekeeper closed this Mar 4, 2026

lodekeeper reopened this Mar 4, 2026

nflaig reviewed Mar 5, 2026

View reviewed changes

nflaig changed the title ~~fix(state-transition): handle previous-epoch payload timeliness committees~~ fix: handle previous-epoch payload timeliness committees Mar 5, 2026

nflaig reviewed Mar 5, 2026

View reviewed changes

refactor(state-transition): simplify previous-epoch PTC init guard

fb9cbae

nflaig mentioned this pull request Mar 5, 2026

fix: cache previous epoch payload timeliness committees #8991

Merged

lodekeeper closed this Mar 5, 2026

Uh oh!

Uh oh!

Conversation

lodekeeper commented Mar 4, 2026

Summary

Changes

Validation

Local

Devnet evidence (local)

Notes on remaining mixed-client instability

Uh oh!

gemini-code-assist Bot commented Mar 4, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

lodekeeper commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 4, 2026

Uh oh!

lodekeeper commented Mar 4, 2026

Uh oh!

nflaig commented Mar 4, 2026

Uh oh!

lodekeeper commented Mar 4, 2026

Uh oh!

lodekeeper commented Mar 4, 2026

Uh oh!

lodekeeper commented Mar 4, 2026

Uh oh!

lodekeeper commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lodekeeper commented Mar 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nflaig commented Mar 5, 2026

Uh oh!

lodekeeper commented Mar 5, 2026

Uh oh!

lodekeeper commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lodekeeper commented Mar 4, 2026 •

edited

Loading

lodekeeper commented Mar 4, 2026 •

edited

Loading