Pending attestations fail to be re-gossiped after an unknown-block `ValidationIgnore` because GossipSub already marked the message as seen

[zhaochonghe]

Describe the bug

Description

When Prysm receives an attestation whose preliminary checks pass, but hasBlockAndState is false for its beacon_block_root, the gossip validator returns pubsub.ValidationIgnore and stores the attestation in the pending-attestation queue. In v7.1.7, hasBlockAndState requires both a known block and either its state or state summary.

After the missing block and required state become available, Prysm validates the pending attestation, saves it to the local attestation pool, and calls BroadcastAttestation (or Broadcast for an aggregate). Source inspection indicates that this second publication is suppressed by go-libp2p-pubsub's seen-message cache when the topic bytes and encoded payload are unchanged.

As a result, the pending attestation can become usable locally, but the intended re-gossip does not actually send it to mesh peers.

Relevant code paths

Prysm uses content-addressed Ethereum gossip message IDs:

• beacon-chain/p2p/pubsub.go configures WithNoAuthor and WithMessageIdFn.
• beacon-chain/p2p/message_id.go derives the message ID from the topic and message payload.

For an unaggregated attestation referencing an unavailable block/state:

• validateCommitteeIndexBeaconAttestation calls savePendingAtt and returns pubsub.ValidationIgnore.

For an aggregate referencing an unavailable block/state:

• validateBlockInAttestation calls savePendingAggregate and returns false.
• validateAggregateAndProof then returns pubsub.ValidationIgnore.

For pending attestations:

• processPendingAttsForBlock processes the queue after the block and state become available.
• processVerifiedAttestation calls BroadcastAttestation.
• processAggregate calls Broadcast.

In the go-libp2p-pubsub version used by Prysm v7.1.7, the message ID is marked as seen before the application validator result is known. A later local Topic.Publish of the same topic and payload encounters the same message ID. ValidateLocal returns a duplicate error, while Topic.Publish converts that duplicate error to nil.

This means the Prysm broadcast call can report success even though the message was not sent to mesh peers.

The default go-libp2p-pubsub seen-message TTL is 120 seconds:

TimeCacheDuration = 120 * time.Second

Prysm v7.1.7 does not override this with WithSeenMessagesTTL, so a retry after a few hundred milliseconds is still suppressed by the same seen entry. Reducing this global TTL to a few hundred milliseconds would not be a safe fix because the cache also protects GossipSub from duplicate processing and forwarding.

References:

• https://github.com/OffchainLabs/prysm/blob/v7.1.7/beacon-chain/sync/validate_aggregate_proof.go
• https://github.com/OffchainLabs/prysm/blob/v7.1.7/beacon-chain/sync/validate_beacon_attestation.go#L120-L126
• https://github.com/OffchainLabs/prysm/blob/v7.1.7/beacon-chain/sync/pending_attestations_queue.go
• https://github.com/OffchainLabs/prysm/blob/v7.1.7/beacon-chain/p2p/pubsub.go
• https://github.com/OffchainLabs/prysm/blob/v7.1.7/beacon-chain/p2p/message_id.go
• https://github.com/libp2p/go-libp2p-pubsub/blob/41b11d5cb1a7/validation.go
• https://github.com/libp2p/go-libp2p-pubsub/blob/41b11d5cb1a7/topic.go

Why an attestation can arrive before its referenced block

This ordering is expected under normal network behavior and does not necessarily indicate a faulty validator or peer.

A beacon block is typically much larger than an individual attestation. Blocks and attestations are also propagated on different GossipSub topics and may reach a node through different peers, mesh paths, queues, and validation workloads. Consequently, a small attestation can overtake the larger block it references.

The honest-validator specification does not require every validator to wait until approximately four seconds into the slot. It instructs a validator to create and broadcast its attestation when either:

1. it receives a valid block from the expected proposer for its assigned slot; or
2. the attestation due time, approximately one-third of the slot, is reached;

whichever happens first:

• https://ethereum.github.io/consensus-specs/phase0/validator/#attesting

Therefore, a validator that receives the proposed block early may immediately create and broadcast its attestation, normally voting for that block, before the approximately four-second attestation due time. Another node can receive that smaller attestation before it receives and processes the referenced block.

Even when a validator waits until the normal attestation due time, the same reordering can occur. The attestation and block may traverse different peers or network paths, and the larger block may still be in transit, queued, or undergoing validation when the attestation arrives.

The missing-block/state path can therefore reflect a normal transient dependency inversion. For attestations that pass the local-view and resource-admission checks described below, a short bounded grace period allows the node to absorb this expected reordering without losing the original GossipSub propagation opportunity.

Mainnet observation

On July 12, 2026, we ran a standard Prysm v7.1.5 node located in Shenzhen, China, on Ethereum mainnet and observed it for approximately 21 hours (The behavior was observed with Prysm v7.1.5. We also inspected v7.1.7 and found the same relevant validation, pending-queue, re-broadcast, and GossipSub seen-cache logic; this issue had not been addressed in the latest version):

• Unique received attestations, counted by distinct GossipSub message ID: 5,313,195
• Attestations that entered the missing block/state path: 156,285
• Affected proportion: 2.9415%
• Average affected rate: approximately 2.07 attestations/second

This suggests that the issue is not limited to a rare edge case on the observed node.

Measurement definition:

• The affected path was counted when hasBlockAndState(ctx, beacon_block_root) was false during attestation gossip validation.
• The waiting interval was measured from receipt of the attestation until the referenced block and required state/state summary made hasBlockAndState true.
• The denominator counts distinct attestations by unique GossipSub message ID, rather than repeated deliveries of the same GossipSub message.

For each affected attestation, we measured how long it took for the referenced block and required state to become available so the attestation could be processed again. The following empirical cumulative distribution shows the proportion that became processable within each waiting interval. The waiting interval on the x-axis is in milliseconds.

Figure: Empirical cumulative proportion of affected attestations whose referenced block/state became available within the given waiting interval.

The measured threshold coverage was:

• Processable within 400 ms: 96.91%
• Processable within 600 ms: 99.48%

Increasing the grace period from 400 ms to 600 ms therefore adds 2.57 percentage points of coverage. Based on the rounded proportions, this corresponds to approximately 4,017 additional attestations in the observed sample. At 600 ms, approximately 0.52% of affected attestations remained unavailable.

This makes 600 ms a useful initial evaluation point while still allowing the benefit to be weighed against the additional validator occupancy.

Sequence of events

1. Node receives attestation A on topic T.
2. GossipSub computes message ID M and marks M as seen.
3. Prysm cannot find the referenced block/state.
4. Prysm queues A and returns ValidationIgnore.
5. The referenced block and required state become available shortly afterward (typically within approximately 600 ms in our observation).
6. Prysm validates A, saves it to the local pool, and calls BroadcastAttestation(A).
7. Because the publication uses the same topic bytes and encoded payload, the local publish computes the same message ID M.
8. GossipSub treats it as a duplicate and does not send it to mesh peers.
9. The publish call may still return nil, so the suppression is not visible to the caller.

Expected behavior

If the referenced block arrives shortly after the attestation, the attestation should be fully validated and forwarded through the original GossipSub validation pipeline, or there should be another supported mechanism that allows the pending message to be propagated after successful validation.

Actual behavior

The attestation is processed and stored locally after the block and required state become available, but the attempted re-gossip is suppressed by the seen-message cache.

The duplicate local publication is expected to return nil, but no outbound router publication is performed for that call. This is a statement about the local publish path, not a claim that a successful publish would guarantee receipt by any particular peer.

Impact

This affects cases where attestations and their referenced blocks/states become available out of order. Other nodes may still propagate the attestation if they already have the block and state, so this does not necessarily cause network-wide loss. However, the affected node's attempted re-gossip is suppressed, which can reduce propagation under latency, packet reordering, or asymmetric peer connectivity.

It also makes the current broadcast result misleading because a nil return does not mean that the message reached the GossipSub router.

Suggested approach: bounded validation grace period

Would it be reasonable to keep the original gossip validation pending for a short, bounded grace period when the referenced block/state is missing?

The grace period should apply only when the attestation is compatible with the node's current local view and the unavailable referenced block/state is the only validation condition that cannot yet be completed. It should not apply indiscriminately to every attestation carrying an unknown root.

More precisely, admission to the grace-period path should fail closed. The attestation target state must already be available, and every validation that does not depend on importing beacon_block_root must complete successfully before a waiter is created. If the target state is unavailable, a committee cannot be derived, a signature/proof cannot be checked, or a waiter-budget token cannot be acquired, the message should not enter the 600 ms wait.

For example:

1. Require AttestationTargetState(data.Target) to succeed and retain that target state for validation.
2. Complete all non-block-dependent checks before waiting: propagation time, slot/target epoch, known-bad roots, duplicate/seen checks, topic/subnet, committee and bitfield/index checks, and all applicable attestation, aggregator, and selection-proof signatures.
3. Acquire hard global, per-peer, per-root, and total-message waiter-budget tokens. If any budget is exhausted, fall back immediately to the existing pending/ValidationIgnore behavior.
4. Enter the grace-period path only if all mandatory checks above pass. At this point, the only deferred predicates should be availability of the referenced block and its state/state summary, fork-choice membership/ancestry, and LMD/FFG consistency checks that require the referenced block.
5. Register an event-driven waiter keyed by beacon_block_root. Readiness must mean that the block and the state/state summary required by hasBlockAndState are available, not merely that a block-import event occurred.
6. Re-check whether the block and state are available after registering the waiter, to avoid a lost wake-up.
7. Wait for either:
   • the block/state availability event;
   • a short timeout, with 600 ms proposed as an initial evaluation point based on the observation above;
   • validator context cancellation.
8. If the block/state becomes available within the grace period, run the remaining block-dependent checks, including fork-choice and LMD/FFG consistency, and return ValidationAccept only if all checks pass. GossipSub can then forward the original message without a second local publication.
9. If the timeout expires, release the waiter budgets, atomically hand the attestation to the pending queue, and return ValidationIgnore.

The 600 ms value is a provisional, data-motivated starting point from one approximately 21-hour observation, not a proposed protocol constant. It should be validated using the exact threshold counts and across networks, node locations, peer counts, and longer observation windows before becoming a default.

Safety and resource considerations

A plain time.Sleep(600 * time.Millisecond) for every unknown block root would introduce a denial-of-service risk by occupying validator concurrency with messages referencing arbitrary roots.

The bounded wait should therefore include:

• event-driven notification rather than polling;
• per-root deduplication;
• hard global, per-peer, per-root, and total-message waiter limits;
• cancellation through the validator context;
• strict eligibility checks before waiting, so an unknown block/state is the only unresolved dependency;
• mandatory target-state availability and signature/proof validation before waiting, to make random-root flooding more expensive for an attacker;
• metrics for wait duration, wake-ups, timeouts, and rejected waiters;
• one shared readiness primitive per root rather than one timer/poller per message;
• an atomic ownership/handoff rule so the same attestation cannot be processed concurrently by both the grace-period path and the existing pending queue.

It may also be useful to trigger or schedule the missing-block request immediately, rather than waiting for the grace period to expire.

Versions inspected

• Observed on Ethereum mainnet: Prysm v7.1.5
• Relevant logic rechecked and still present: Prysm v7.1.7
• go-libp2p-pubsub: `v0.16.1-0.20260611143718-41b11d5cb1a7

Has this worked before in a previous version?

🔬 Minimal Reproduction

No response

Error

Platform(s)

No response

What version of Prysm are you running? (Which release)

No response

Anything else relevant (validator index / public key)?

No response