PTC payload attestations deadlock when a slot has multiple PTC duties (random duty order + serial threshold signing)

[shane-moore]

Summary

On a 4-operator Gloas devnet, every PTC duty in a slot fails with SignatureCollectionFailed(QueueClosedError) whenever the slot carries two or more PTC duties and the operators split evenly on which duty to sign first. The duty order is random per process (a HashSet iteration in the LH duties service), signing is serial per duty, and under Anchor each signature blocks on the 3-of-4 partial-signature threshold, so a 2-2 split on the first duty deadlocks both collections. Observation data is byte-identical across all four operators, and nothing gets published for the slot.

The code involved is Lighthouse's PayloadAttestationService / DutiesService (spawned by Anchor since #1078), but the bug can only manifest under Anchor: a plain VC signs locally and instantly, so list order is irrelevant. Filing here because Anchor owns the integration and the LH pin.

Environment

• Anchor 4a4cfcf6, Lighthouse pin e58ec88fe
• 4 geo-distributed operators (Sydney, Montreal, Falkenstein x2), one cluster, threshold 3-of-4
• Self-genesis Gloas devnet (fork at epoch 3, 12s slots), RUST_LOG=debug,signature_collector=trace,message_receiver=trace

Mechanism

1. PTC duty order is random per process. poll_beacon_ptc_attesters builds local_indices by iterating a HashSet of pubkeys. With INITIAL_PTC_DUTIES_QUERY_SIZE = 1 the stored per-epoch Vec<PtcDuty> inherits that order, and get_ptc_duties_for_slot preserves it. HashSet iteration order differs per process, so each operator holds the same duties in a different order.

2. Signing is serial per duty. sign_and_publish awaits each duty's signature in a for loop. In a plain VC each iteration returns immediately. Under Anchor, sign_payload_attestation blocks until 3 of 4 operators contribute a partial signature for that duty, so duty 2 does not start until duty 1 resolves.

3. A 2-2 split is a deadlock. If two operators start with duty A and two with duty B, each collection holds 2 of the 3 required partials and nobody can advance. The hang is converted into an error by collector cleanup (SIGNATURE_COLLECTOR_RETAIN_SLOTS = 1, cutoff): at the next slot boundary the collectors are evicted and every pending wait resolves with QueueClosedError. Each operator then proceeds to its second duty on a fresh collector (round-1 peer partials already evicted). The second round is the complement of the first, so it splits 2-2 again by construction and fails one boundary later. This is why the errors surface 2-3 slots after the duty, with identical data everywhere.

Evidence: slot 803

duty_count=2, validator indices 129 (0x9482943c...) and 130 (0xb30bbcf3...). All four operators fetched identical PayloadAttestationData (beacon_block_root=0x514e73e4..., payload_present=true).

First and second sign_and_collect(kind: PTCAttester) per operator, from the trace:

operator	first duty (at 01:06:38)	second duty (at 01:06:53)
1 (Sydney)	130	129
2 (Montreal)	129	130
3 (Falkenstein)	130	129
4 (Falkenstein)	129	130

Exact 2-2 split. All four second calls fire at 01:06:53.002, which is the slot-805 boundary, i.e. the instant eviction unblocks the first await. Both duties fail on every operator; the second failure lands at the slot-806 boundary (01:07:05).

Operator 1 timeline (slot 803 runs 01:06:29 to 01:06:41, PTC production fires at +9s):

01:06:38.001352 DEBUG Producing payload attestations       slot=803 duty_count=2
01:06:38.001941 TRACE sign_and_collect called              kind: PTCAttester, threshold: 3, slot: Slot(803), pubkey: 0xb30bbcf3...
01:06:38.001969 TRACE Spawned signature collector          signing_root=0x8dc2efb4... validator_index=ValidatorIndex(130)
01:06:38.002398 TRACE Signature collector received message msg=PartialSignature { operator_id: OperatorId(1), ... }
01:06:38.112056 TRACE Spawned signature collector          signing_root=0x8dc2efb4... validator_index=ValidatorIndex(129)
01:06:38.112080 TRACE Signature collector received message msg=PartialSignature { operator_id: OperatorId(2), ... }
01:06:38.151622 TRACE Signature collector received message msg=PartialSignature { operator_id: OperatorId(4), ... }
01:06:38.155236 TRACE Signature collector received message msg=PartialSignature { operator_id: OperatorId(3), ... }
                      [nothing further for these collectors until the slot-805 boundary]
01:06:53.002014 WARN  Insufficient partial signatures for payload attestation  slot=803 validator=0xb30bbcf3... error=...QueueClosedError
01:06:53.002064 ERROR Failed to sign payload attestation   slot=803 validator=0xb30bbcf3...
01:06:53.002111 TRACE sign_and_collect called              kind: PTCAttester, threshold: 3, slot: Slot(803), pubkey: 0x9482943c...
01:06:53.002419 TRACE Spawned signature collector          signing_root=0x8dc2efb4... validator_index=ValidatorIndex(129)
01:07:05.001800 WARN  Insufficient partial signatures for payload attestation  slot=803 validator=0x9482943c... error=...QueueClosedError
01:07:05.001821 ERROR Failed to sign payload attestation   slot=803 validator=0x9482943c...

The four partials at 01:06:38.002 to .155 attribute exactly, because the collector task's span appends slot / validator_index / signing_root to every Signature collector received message line: ValidatorIndex(130) collected {OperatorId(1), OperatorId(3)} and ValidatorIndex(129) collected {OperatorId(2), OperatorId(4)}. Each collection held 2 of the required 3, confirming the 2-2 split at the collector level, not just from call ordering. (An earlier revision of this issue claimed the trace lacks signing_root; that was wrong, an artifact of truncated log excerpts. The span fields are appended at the end of each line and no logging change is needed.)

Both PTC duties in slot 803 share one signing_root (PayloadAttestationData contains no validator-specific field), so the two collectors are keyed apart only by validator_index. That is not the cause (other 2-duty slots with the same structure succeeded) but is worth knowing when reading the traces.

Run statistics match the model

Identical-data PTC failures occurred only on multi-duty slots:

slot	duty_count	outcome
141	2	failed, both duties, all 4 operators
392	3	failed, separate cause (see below)
698	2	ok
803	2	failed, both duties, all 4 operators
846	2	ok

Single-duty slots (the overwhelming majority, roughly 30 successes per 90-minute window) never failed outside a known infra outage and a separate divergence class where all four operators agree on the block root but split on payload_present (structural for DVT, discussed in #1079). With uniform random ordering, a 2-duty slot deadlocks with probability 6/16 = 37.5%; we observed 2 of 4.

Ruled out

• Envelope-reveal gap: data was identical with payload_present=true on all operators.
• MAX_MESSAGES_PER_ROUND pre-consensus cap: PTC partials are sent as SignatureRequester::SingleValidator, so the msg_id carries the validator pubkey and the two duties never share a count. Healthy 2-duty slots (698, 846) publish both messages without tripping it.
• Rejection on receipt: no validation failures in the window.

Possible fixes

Either alone removes the deadlock:

1. Order the duty Vec deterministically (for example, sort by validator index) so every operator agrees on the serial order.
2. Sign a slot's PTC duties concurrently in sign_and_publish (join_all over duties). Also a latency win.

For context, the proposer preferences service has the same serial-loop shape (publish_proposer_preferences) but does not deadlock, because its list derives from the BN proposer duties for the epoch, which every operator holds in the same slot order. Concurrent signing was proposed there in sigp/lighthouse#9617 and closed in favor of the simpler per-duty retry of sigp/lighthouse#9660. A retry-on-later-polls approach does not transfer to PTC (payload attestations are slot-local with a fixed due point, and the deadlock only surfaces after collector eviction, 2 slots too late), so option 1 is the minimal change in the same spirit; option 2 stands if signing concurrency is acceptable here.

Both options are diffs to Lighthouse files, so the likely flow is a Lighthouse PR plus an Anchor pin bump; there is no clean Anchor-side workaround because the ValidatorStore is called one duty at a time and never sees the slot's duty list.

Full per-operator log captures for slot 803 (about 1700 lines each, the complete 01:06:29 to 01:07:18 window at trace level) are archived and available on request.