Goal
Bound the PTC payload-attestation signature-collection await to the end of the duty slot, so a no-quorum collection resolves while publishing is still possible instead of hanging until collector eviction.
Context / motivation
sign_payload_attestation awaits collect_signature with no timeout (anchor/validator_store/src/lib.rs:3500-3541). A collection that never reaches threshold resolves only when the cleaner evicts it at the second slot boundary after the duty fires (SIGNATURE_COLLECTOR_RETAIN_SLOTS = 1), surfacing as QueueClosedError well after the [75%, 100%] publish window has closed. A slot-N payload attestation is consulted by fork choice only in slot N+1 and dropped on the wire once data.slot != current_slot, so any wait past slot end is pure loss. #1217 measured the blast radius under the current serial loop: ~15s for the first non-quorum validator plus ~12s per additional one, with reconstructed sibling messages held unpublished throughout.
The Lighthouse-side fix for #1217 (concurrent signing) removes the ordering deadlock but leaves a slot's batch publish coupled to its slowest collection. The residual case is complementary duty-set divergence: after a dependent-root flap with a 2-2 beacon-node-view split, operators 1,2 hold duties {A, X} and operators 3,4 hold {A, Y}; duty A reconstructs on all four operators while X and Y sit at 2 of 3 partials, every operator's concurrent join blocks on its phantom duty until eviction, and A's message misses the slot everywhere. With awaits bounded at slot end, the phantom awaits fail in time for A's batch to publish.
sign_proposer_preferences already ships this exact mitigation: tokio::time::timeout around the same collect_signature call, mapping elapse to CollectionError::CollectionTimeout (lib.rs:3560-3585, constant at :104). PTC is the only remaining Gloas signature-collection path with an unbounded await.
Suggested approach
- Wrap the
collect_signature await in sign_payload_attestation with tokio::time::timeout_at at the end of data.slot, via self.get_instant_in_slot(data.slot + 1, Duration::ZERO)? (lib.rs:814). An absolute deadline rather than ProposerPreferences' relative 2-slot constant, because PTC usefulness ends at a fixed point in the slot; a call arriving after slot end then fails fast, which is the desired behavior.
- Map elapse to
CollectionError::CollectionTimeout exactly as ProposerPreferences does (lib.rs:3580-3585), so it flows through report_ptc_collection_failure (lib.rs:873) into the existing NoSignature bucket (instrumentation.rs:54-61) and anchor_ptc_reconstruction_failures_total{reason="no_signature"} with no new telemetry.
- Update the instrumentation comment that says
CollectionTimeout is synthesized only by sign_proposer_preferences (instrumentation.rs:54).
- Bounding at exactly slot end (no subtracted margin) is deliberate: publishes slightly past the boundary can still land within
MAXIMUM_GOSSIP_CLOCK_DISPARITY (500ms), and an earlier cutoff wastes reconstruction chances. Tune at PR time if evidence suggests otherwise.
Acceptance criteria
- A PTC collection that never reaches threshold returns
CollectionTimeout by the end of the duty slot instead of QueueClosedError at eviction.
- The timeout classifies as NoSignature (warn +
reason="no_signature"), not infra; no new metric or log shape.
- A call made after the duty slot has ended fails fast instead of awaiting.
- Successful collections are unaffected; no wire-behavior change.
Tests
Mirror the ProposerPreferences timeout tests (anchor/validator_store/src/testing/proposer_preferences.rs:443-512): a no-quorum collection surfaces CollectionTimeout via the bound and increments the reconstruction-failure metric; a threshold-reaching collection is unaffected. Add the fail-fast case (duty slot already over at call time).
Notes
Follow-up from #1217 planning. Complements, and does not replace, the Lighthouse-side concurrent-signing fix: concurrency removes the ordering deadlock; this bound removes the residual coupling of a slot's publish to a genuinely sub-threshold collection. Related: #1079; surfacing no-quorum as CollectionTimeout at the PTC callsite narrows the QueueClosedError ambiguity #1079 describes, but collector-granularity divergence attribution remains its scope.
Issues are directionally correct, not prescriptive; verify symbols at PR time.
Goal
Bound the PTC payload-attestation signature-collection await to the end of the duty slot, so a no-quorum collection resolves while publishing is still possible instead of hanging until collector eviction.
Context / motivation
sign_payload_attestationawaitscollect_signaturewith no timeout (anchor/validator_store/src/lib.rs:3500-3541). A collection that never reaches threshold resolves only when the cleaner evicts it at the second slot boundary after the duty fires (SIGNATURE_COLLECTOR_RETAIN_SLOTS = 1), surfacing asQueueClosedErrorwell after the [75%, 100%] publish window has closed. A slot-N payload attestation is consulted by fork choice only in slot N+1 and dropped on the wire oncedata.slot != current_slot, so any wait past slot end is pure loss. #1217 measured the blast radius under the current serial loop: ~15s for the first non-quorum validator plus ~12s per additional one, with reconstructed sibling messages held unpublished throughout.The Lighthouse-side fix for #1217 (concurrent signing) removes the ordering deadlock but leaves a slot's batch publish coupled to its slowest collection. The residual case is complementary duty-set divergence: after a dependent-root flap with a 2-2 beacon-node-view split, operators 1,2 hold duties {A, X} and operators 3,4 hold {A, Y}; duty A reconstructs on all four operators while X and Y sit at 2 of 3 partials, every operator's concurrent join blocks on its phantom duty until eviction, and A's message misses the slot everywhere. With awaits bounded at slot end, the phantom awaits fail in time for A's batch to publish.
sign_proposer_preferencesalready ships this exact mitigation:tokio::time::timeoutaround the samecollect_signaturecall, mapping elapse toCollectionError::CollectionTimeout(lib.rs:3560-3585, constant at:104). PTC is the only remaining Gloas signature-collection path with an unbounded await.Suggested approach
collect_signatureawait insign_payload_attestationwithtokio::time::timeout_atat the end ofdata.slot, viaself.get_instant_in_slot(data.slot + 1, Duration::ZERO)?(lib.rs:814). An absolute deadline rather than ProposerPreferences' relative 2-slot constant, because PTC usefulness ends at a fixed point in the slot; a call arriving after slot end then fails fast, which is the desired behavior.CollectionError::CollectionTimeoutexactly as ProposerPreferences does (lib.rs:3580-3585), so it flows throughreport_ptc_collection_failure(lib.rs:873) into the existing NoSignature bucket (instrumentation.rs:54-61) andanchor_ptc_reconstruction_failures_total{reason="no_signature"}with no new telemetry.CollectionTimeoutis synthesized only bysign_proposer_preferences(instrumentation.rs:54).MAXIMUM_GOSSIP_CLOCK_DISPARITY(500ms), and an earlier cutoff wastes reconstruction chances. Tune at PR time if evidence suggests otherwise.Acceptance criteria
CollectionTimeoutby the end of the duty slot instead ofQueueClosedErrorat eviction.reason="no_signature"), not infra; no new metric or log shape.Tests
Mirror the ProposerPreferences timeout tests (
anchor/validator_store/src/testing/proposer_preferences.rs:443-512): a no-quorum collection surfacesCollectionTimeoutvia the bound and increments the reconstruction-failure metric; a threshold-reaching collection is unaffected. Add the fail-fast case (duty slot already over at call time).Notes
Follow-up from #1217 planning. Complements, and does not replace, the Lighthouse-side concurrent-signing fix: concurrency removes the ordering deadlock; this bound removes the residual coupling of a slot's publish to a genuinely sub-threshold collection. Related: #1079; surfacing no-quorum as
CollectionTimeoutat the PTC callsite narrows the QueueClosedError ambiguity #1079 describes, but collector-granularity divergence attribution remains its scope.Issues are directionally correct, not prescriptive; verify symbols at PR time.