Backport #17222 to glamsterdam-devnet-7: fix PTC blob_data_available to use data column availability - #17316
Open
parithosh wants to merge 29 commits into
Open
Conversation
Adds a feature flag --reorg-late-payloads that currently defaults to false and it is hidden.
postPayloadTasks no longer takes the head root and the envelope root as parameters since they can be obtained from the node. Also, it splits up a helper to check if the incoming payload will be head under a head lock and thus avoids any race.
When the payload is for the current slot, will become head and we are proposing in the next slot, do not send FCU if it arrived late.
The following are the main branches: 1. If the head has not changed at 10 seconds, we may still need to send a new FCU if the PTC is telling us to reorg the payload (there was consensus that it was late) 2. If the head has changed but the head root hasn't, we are dealing with a payload content reorg, we need to understand it: if it went from full -> empty, this can only happen because of attestations and this can only happen for heads from the previous blocks, we do not reorg such blocks so we need to send FCU If it went empty->full, it may have been that we avoided sending FCU when the payload was late, we need to evaluate if the PTC already achieved consensus, if it hasn't then we can continue our bet of reorging. 3. If the headroot has changed, continue with the previous paths to reorg late blocks.
Update shouldReorgPayload expectations for the negated PTC-consensus logic, only emit head_v2 on the empty->full transition, and convert counting loops to range to satisfy the rangeint modernize analyzer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
…nnot be guaranteed (OffchainLabs#17174) **What type of PR is this?** > Bug fix **What does this PR do? Why is it needed?** Reported in Discord ([Message link](https://discord.com/channels/595666850260713488/892088344438255616/1526470317315457165)). https://github.com/OffchainLabs/prysm/blob/73439ebe08e6cee85e5e9dbc24ac60dd71d4294d/beacon-chain/execution/engine_client.go#L182-L196 Today, Prysm reconstructs the series of execution payload envelopes from multiple sources: 1. DB: Saves (and read) *blinded* payload envelope which deliberately omits large-sized transactions, withdrawals, and **BAL**. 2. Call Engine API (`engine_getPayloadBodiesByHashV2`) for getting payloads. Note that this can return `null` `blockAccessList` when BAL has been pruned from storage. (ref: https://github.com/ethereum/execution-apis/blob/main/src/engine/amsterdam.md#engine_getpayloadbodiesbyhashv2) 3. Call Engine API (`eth_getBlockByHash`) for getting execution blocks. So it is possible to construct a payload with empty BAL, when the envelope originally had non-empty BAL at that slot. The problem is that we uses the signature from the DB, which can be deviated as the message becomes different. (non-empty BAL -> empty BAL) This PR returns an error when reconstruction cannot guarantee the correctness, which means: - if EL cannot provide execution payloads or - length of BAL is zero - BAL shouldn't be zero-byted in terms of RLP. **Which issue(s) does this PR fix?** N/A but should solve the issue in `glamsterdam-devnet-6`. ([Discord link](https://discord.com/channels/595666850260713488/892088344438255616/1526470317315457165))) **Other notes for review** ~~`TestReconstructExecutionPayloadEnvelope_BlindedStorage` covers possible scenarios that I can think of.~~ **Acknowledgements** - [x] I have read [CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md). - [x] I have included a uniquely named [changelog fragment file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd). - [x] I have added a description with sufficient context for reviewers to understand this PR. - [x] I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).
…to glamsterdam-devnet-7
…amsterdam-devnet-7
The lookahead solved the shuffling stability problem on Fulu
…Labs#17222) - `BlobDataAvailable` in payload attestation data was populated from `HasFullNode`, which only flips after the envelope is fully imported - Per spec, `blob_data_available` is `is_data_available(beacon_block_root)`, independent of envelope import - New `DataAvailable` getter checks the non-blocking column store status first, and only reads the block's bid when no columns are stored, since an empty column summary can't distinguish a blobless payload from missing data and this order avoids a DB block read in the common case (cherry picked from commit 21b1d04)
terencechain
force-pushed
the
glamsterdam-devnet-7
branch
from
August 6, 2026 13:43
3cf9687 to
821840a
Compare
terencechain
force-pushed
the
glamsterdam-devnet-7
branch
from
August 8, 2026 21:45
821840a to
9415ddd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Backport of #17222 (
21b1d046b0) toglamsterdam-devnet-7. Clean cherry-pick, no conflicts, original authorship preserved.developgot this fix on 2026-07-22.glamsterdam-devnet-7branched fromdevelopat5420c085(2026-07-15) and never received it, so the branch the devnet is actually running still has the old behaviour.Why it matters on this branch
On devnet-7,
buildPayloadAttestationDatapopulates the PTC vote as:HasFullNodeonly reports whether a full payload node exists in fork choice — it is a payload-import check and never looks at a blob commitment or a column sidecar. Perspecs/gloas/validator.md, the field is:is_data_availableis independent of envelope import, and with zero blob commitments it is vacuously true (all([])).Observed on the live devnet
Slot 165502 (2026-08-06 11:40:24 UTC). The payload was orphaned and the bid declared
blob_kzg_commitments: []— zero blobs. All 500 PTC votes agreedpayload_present=false, then split onblob_data_available:blob_data_available=trueblob_data_available=falseWith zero blobs there is no data that can be unavailable, so
trueis correct and Prysm'sfalseis wrong.This is systematic, and the correlation is exact — the
(payload_absent, blob_data_available=true)aggregate appears on only the zero-blob slots and never when the orphaned payload carried blobs:That is precisely the behaviour
DataAvailablerestores, and the cherry-pickedTestDataAvailable/no_blob_commitmentscovers exactly this case.Fork-choice impact at 165502 was nil —
payload_present=falsewas unanimous, soptcVotedLatedecided it and the availability split was never consulted. The reason to backport is that on this branch a Prysm PTC vote carries no independent blob-availability signal at all: both fields derive from having the payload, so the vote cannot express "payload arrived, blob data did not".For reference, Lighthouse had the same class of bug on their
glamsterdam-devnet-7branch (fork_choice.is_payload_received) and have already replaced it with a custody-column check onunstable.Behaviour change reviewers should know
buildPayloadAttestationDatais now fallible. ADataAvailableerror returns an InternalRpcError, so agetBlockmiss (root not in cache or DB) suppresses the PTC vote, where the oldHasFullNode(root)could never fail and always produced one. This is upstream's own design from #17222, carried over unchanged — noting it because this branch runs on a live devnet.Verification
Run in a clean worktree at the devnet-7 tip:
go buildof all four touched packages — cleango test ./beacon-chain/rpc/core/... -tags develop— okgo test ./beacon-chain/rpc/eth/validator/... -tags develop— okTestDataAvailable(new, from the cherry-pick) — all three subtests pass, includingno_blob_commitmentsPre-existing CI failures on this branch, not caused by this PR
Two test packages already fail to build on the devnet-7 base. Both verified at
HEAD~1(3cf9687dd9) and neither file is touched here:beacon-chain/blockchain—head_test.go:169,269,300:notifyNewHeadEventtakes[32]byte, tests pass[]bytebeacon-chain/rpc/prysm/v1alpha1/validator—proposer_payload_envelope_test.go:271,282: undefinedethpb.GenericSignedExecutionPayloadEnvelope_SignedEnvelope, undefinedprotoTo run
TestDataAvailableI temporarily patchedhead_test.golocally; that patch is not part of this PR.