Skip to content

fix(validator_store): publish decided AggregatorCommittee aggregates from the metadata service - #1245

Draft
shane-moore wants to merge 3 commits into
sigp:unstablefrom
shane-moore:fix/boole-aggregate-authoritative-publish
Draft

fix(validator_store): publish decided AggregatorCommittee aggregates from the metadata service#1245
shane-moore wants to merge 3 commits into
sigp:unstablefrom
shane-moore:fix/boole-aggregate-authoritative-publish

Conversation

@shane-moore

Copy link
Copy Markdown
Member

Stacked on #1229. The first two commits are #1229's; review scope is the top commit (97e24a4d) only. Will rebase once #1229 merges.

Problem, Evidence, and Context (Required)

Post-Boole, Anchor completes every attestation selection proof but most aggregates are never published. Anchor's selection proofs are distributed signing rounds finishing ~250-500ms into the slot, while embedded Lighthouse's attestation service clones its duty view (including selection_proof: Option) when attestation production starts and never re-reads it. All-None clones make produce_and_publish_aggregates return Ok silently. Both Lighthouse trigger paths (head event and timer) share the clone, so delaying the head event cannot close the race.

  • Scale-100 ssv-mini evidence: 13/200 expected unique aggregates published (amplified build), 73/200 (PR-1229-only control), while proofs, QBFT, and post-consensus signing all succeeded.
  • The failure is silent: no log, no metric, on the skip path.
  • go-ssv is not affected: it publishes decided aggregates from the SSV node itself with no validator-client snapshot in its path.

Change Overview (Required)

Make Anchor the single authoritative Boole+ aggregate publisher, working from the decided value instead of Lighthouse's snapshot:

  • The Boole+ branch of sign_aggregate_and_proofs now yields one empty Ok batch (Lighthouse's publish loop drops empty batches silently, verified at pin b263df5), and sign_committee_aggregate_and_proofs is deleted.
  • start_aggregator_post_consensus returns its vacant-only insertions; the metadata service spawns one detached publisher per slot that joins each new execution, drains the decided aggregate signatures under the existing two-slot deadline, assembles SignedAggregateAndProof (v1/v2 by decided variant), and POSTs via first_success (Lighthouse's own aggregate publication policy).
  • New anchor_aggregator_committee_publish_total{result} metric whose labels partition every committee processed; per-aggregate publish logs reproduce Lighthouse's field set (type="aggregated").

Reading order: aggregator_post_consensus.rs (registration, resolve, publisher driver), then metadata_service.rs (spawn + POST), then the lib.rs deletion, then tests.

Did not change: pre-Boole per-validator aggregate signing (Lighthouse still publishes), the contributions callback, PR #1229's exactly-once registration and detached signing, and all head-event/timer plumbing.

Risks, Trade-offs, and Mitigations (Required)

  • Publication logic for one object class moves into Anchor. Accepted deliberately for go-ssv parity: the decided value, reconstructed signature, and submission now live in one place, and correctness no longer depends on Lighthouse's internal scheduling at each pin.
  • All four operators publish the same reconstructed aggregate (as in go-ssv); duplicate POSTs to shared beacon nodes were exercised in a mixed-client run with zero errors.
  • The empty-batch behavior of Lighthouse's publish loop is a pin-specific fact; re-verify on pin bumps (tripwire noted in the code comment).
  • Cold Boole-only code path; zero lines and zero latency on attestation production.

Validation (Required)

  • 72 crate tests (was 66), including recorder-based publisher tests, vacant-only registration, quorum-failure and deadline paths, and the previously uncovered pre-Boole per-validator path. make cargo-fmt-check, make lint clean.
  • Four ssv-mini campaigns on this exact build (evidence and per-run verdicts recorded in the campaign notes):
    • Scale-100 all-Anchor: 200/200 unique aggregates (control: 73/200) with the underlying race still firing (48-52 of 62 proof batches after Lighthouse's snapshot start).
    • Mixed 2 Anchor + 2 go-ssv: 200/200 under a harsher race (63/64 late batches); publishes from go-led decided values; duplicate POSTs harmless.
    • Fault matrix: single-operator restarts x3 (continuous publication on 3/4), two-operator quorum loss (correct null result, no hangs, instant recovery), hung-BN pause (zero publish failures, instant resume).
    • 250 validators, 513-slot soak: every decided aggregate published, zero publish failures, flat resources; a ~3% eligible-assignment shortfall traced to selection-proof collection timeouts under load (upstream signing pipeline, pre-existing class).
  • Known limitation: the no_signatures outcome (decided aggregates, no root reaches quorum) is unit-tested but was not reproducible live (local reconstruction outruns slot-precise fault injection).

Rollback (Required for behavior or runtime changes; optional otherwise)

Single revertible commit; reverting restores the Lighthouse callback path (and its snapshot race). No config, schema, or wire-format changes; the new metric disappears on revert.

Blockers / Dependencies (Optional)

Additional Info / Next Steps (Optional)

  • Follow-ups to be filed: AggregatorCommitteeDataValidator version-vs-duty-slot-fork validation (mirrors b680964; makes the endpoint variant-sniff provably redundant), and an upstream Lighthouse issue for the duty-snapshot race (bites their remote-signer/DVT users).
  • Alternatives considered and rejected: head-event readiness gate (structurally partial: the timer path shares the snapshot; couples attestation liveness to aggregate readiness), fixed post-fork head-event delay (constant-dependent), Lighthouse fork (pin cadence). A redundant dual-publisher rollout was debated and declined: it forfeits the callback-path deletion and makes the publish metric unmeasurable under races.

…nce per slot

Both Boole post-consensus callbacks sized the committee partial-signature
batch from the full decided value but each signed only its own object class,
so any decided entry whose class the local Lighthouse did not request left the
batch permanently under-filled and the operator's single post-consensus
message was never sent. The whole duty then failed cluster-wide.

Drive the signing from the decided value instead. The slot pipeline starts one
execution per committee when it publishes that value at 2/3 slot, so exactly
one message per (committee, slot) holds by construction rather than by locking,
and the two Lighthouse callbacks become lookups that filter the shared outcome
for the roots they asked for. The batch size is the number of entries actually
submitted, so it is always reachable by local filtering.

This reuses the crate's existing signing pipeline rather than duplicating it:
SigningRequest and resolve now key results by (validator index, signing root)
so one validator can hold five distinct roots, and collect_prepared_signatures
shares the drain helper with the new path. collect_signature is unchanged.

Refs sigp#1227
Registering an AggregatorCommittee execution overwrote any existing entry for
the same (committee, slot). The slot pipeline publishes once per slot today, so
this was not reachable, but it moved the exactly-once guarantee out of this
module and into another module's timing loop. Overwriting would spawn a second
QBFT round and a second set of detached signing tasks while the first set kept
running, putting two post-consensus messages on the wire for one slot, which
peers reject with a gossip penalty.

Register only into a vacant entry so exactly-once holds here regardless of how
often the pipeline publishes, and correct the field doc, which still described
the superseded callback-triggered design.

Raised in review of sigp#1229.
…from the metadata service

Lighthouse's attestation service clones its duty view before Anchor's
distributed selection proofs finish, so post-Boole aggregates were signed
but silently never published (13-73 of 200 expected in scale-100 ssv-mini
campaigns). Make Anchor the authoritative Boole+ aggregate publisher: the
Lighthouse aggregate callback returns one empty batch, and a detached
metadata-service publisher joins each newly registered post-consensus
execution, drains the decided aggregate signatures under the existing
two-slot deadline, and posts them via first_success. This matches go-ssv,
which publishes decided aggregates from the SSV node itself, and makes
publication independent of Lighthouse's snapshot timing.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.28035% with 93 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.89%. Comparing base (82bac14) to head (97e24a4).
⚠️ Report is 1 commits behind head on unstable.

Files with missing lines Patch % Lines
anchor/validator_store/src/metadata_service.rs 0.00% 50 Missing ⚠️
...r/validator_store/src/aggregator_post_consensus.rs 91.86% 34 Missing ⚠️
anchor/validator_store/src/lib.rs 92.50% 6 Missing ⚠️
...tor_store/src/testing/aggregator_post_consensus.rs 99.59% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #1245      +/-   ##
============================================
+ Coverage     65.42%   69.89%   +4.47%     
============================================
  Files           164      166       +2     
  Lines         28075    29803    +1728     
============================================
+ Hits          18367    20832    +2465     
+ Misses         9708     8971     -737     
Flag Coverage Δ
rust 69.89% <93.28%> (+4.47%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shane-moore
shane-moore marked this pull request as draft August 13, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants