Skip to content

fix(validator_store): bound PTC collection awaits to the duty slot end #1218

Description

@shane-moore

Goal

Bound the PTC payload-attestation signature-collection await to the end of the duty slot, so a no-quorum collection resolves while publishing is still possible instead of hanging until collector eviction.

Context / motivation

sign_payload_attestation awaits collect_signature with no timeout (anchor/validator_store/src/lib.rs:3500-3541). A collection that never reaches threshold resolves only when the cleaner evicts it at the second slot boundary after the duty fires (SIGNATURE_COLLECTOR_RETAIN_SLOTS = 1), surfacing as QueueClosedError well after the [75%, 100%] publish window has closed. A slot-N payload attestation is consulted by fork choice only in slot N+1 and dropped on the wire once data.slot != current_slot, so any wait past slot end is pure loss. #1217 measured the blast radius under the current serial loop: ~15s for the first non-quorum validator plus ~12s per additional one, with reconstructed sibling messages held unpublished throughout.

The Lighthouse-side fix for #1217 (concurrent signing) removes the ordering deadlock but leaves a slot's batch publish coupled to its slowest collection. The residual case is complementary duty-set divergence: after a dependent-root flap with a 2-2 beacon-node-view split, operators 1,2 hold duties {A, X} and operators 3,4 hold {A, Y}; duty A reconstructs on all four operators while X and Y sit at 2 of 3 partials, every operator's concurrent join blocks on its phantom duty until eviction, and A's message misses the slot everywhere. With awaits bounded at slot end, the phantom awaits fail in time for A's batch to publish.

sign_proposer_preferences already ships this exact mitigation: tokio::time::timeout around the same collect_signature call, mapping elapse to CollectionError::CollectionTimeout (lib.rs:3560-3585, constant at :104). PTC is the only remaining Gloas signature-collection path with an unbounded await.

Suggested approach

  • Wrap the collect_signature await in sign_payload_attestation with tokio::time::timeout_at at the end of data.slot, via self.get_instant_in_slot(data.slot + 1, Duration::ZERO)? (lib.rs:814). An absolute deadline rather than ProposerPreferences' relative 2-slot constant, because PTC usefulness ends at a fixed point in the slot; a call arriving after slot end then fails fast, which is the desired behavior.
  • Map elapse to CollectionError::CollectionTimeout exactly as ProposerPreferences does (lib.rs:3580-3585), so it flows through report_ptc_collection_failure (lib.rs:873) into the existing NoSignature bucket (instrumentation.rs:54-61) and anchor_ptc_reconstruction_failures_total{reason="no_signature"} with no new telemetry.
  • Update the instrumentation comment that says CollectionTimeout is synthesized only by sign_proposer_preferences (instrumentation.rs:54).
  • Bounding at exactly slot end (no subtracted margin) is deliberate: publishes slightly past the boundary can still land within MAXIMUM_GOSSIP_CLOCK_DISPARITY (500ms), and an earlier cutoff wastes reconstruction chances. Tune at PR time if evidence suggests otherwise.

Acceptance criteria

  • A PTC collection that never reaches threshold returns CollectionTimeout by the end of the duty slot instead of QueueClosedError at eviction.
  • The timeout classifies as NoSignature (warn + reason="no_signature"), not infra; no new metric or log shape.
  • A call made after the duty slot has ended fails fast instead of awaiting.
  • Successful collections are unaffected; no wire-behavior change.

Tests

Mirror the ProposerPreferences timeout tests (anchor/validator_store/src/testing/proposer_preferences.rs:443-512): a no-quorum collection surfaces CollectionTimeout via the bound and increments the reconstruction-failure metric; a threshold-reaching collection is unaffected. Add the fail-fast case (duty slot already over at call time).

Notes

Follow-up from #1217 planning. Complements, and does not replace, the Lighthouse-side concurrent-signing fix: concurrency removes the ordering deadlock; this bound removes the residual coupling of a slot's publish to a genuinely sub-threshold collection. Related: #1079; surfacing no-quorum as CollectionTimeout at the PTC callsite narrows the QueueClosedError ambiguity #1079 describes, but collector-granularity divergence attribution remains its scope.

Issues are directionally correct, not prescriptive; verify symbols at PR time.

Metadata

Metadata

Assignees

Labels

epbsePBS / EIP-7732 / Gloas implementation

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions