Boole AggregatorCommittee can omit decided roots from post-consensus batches

[shane-moore]

Description

At Boole and later, one AggregatorCommitteeConsensusData decision can contain both aggregate-and-proof and contribution-and-proof objects.

Anchor computes the expected local committee batch size from the complete decided value. However, it creates signing requests only from the local Lighthouse callback vector for the current object class.

If a decided object is absent from that callback vector, Anchor counts the object but never adds its local partial signature. The committee batch can remain below its expected size and never broadcast.

This was observed in a mixed-client no-fault run and is reproduced by a focused validator-store regression.

Version

Deterministic local reproduction:

• Anchor unstable: 82bac14ec635bc28a68a6bba9799bd603ec0c082
• Rust: rustc 1.91.0 (f8297e351 2025-10-28)
• Platform: macOS arm64

Original mixed-client observation:

• Anchor: fd3c7f9677b2bbb04fe7d7c2d4eca81313db7b79
• Embedded Lighthouse diagnostic revision: b263df596671a2bd42bf1034e1cdc8188ba8a9b0
• go-ssv: 874ba316266aec6215bc21c6e2e929e195f08d6b
• ssv-mini: fef3b7d791384911078b2ce977100f87679e5c46
• Topology: two Anchor operators plus two go-ssv operators in one 3-of-4 committee
• Fork: Boole at slot 192
• Validators: 74 active validators

Current remote heads were also checked:

• Anchor unstable: 82bac14ec635bc28a68a6bba9799bd603ec0c082
• Anchor epbs: 36950b69f6f8e6ca362b46edd608b6b1037793f7
• go-ssv stage: a9008f3d0542f83291f2e9d18babf3edd3abd7b1

The current ePBS branch has the same callback-vector outer loops. PR #1128 changes proposal inputs, but it does not remove this post-consensus worklist gap.

Present Behaviour

Mixed-client evidence

In the no-fault run at slot 230, the decided AggregatorCommittee value contained three post-consensus objects:

• One aggregate for validator 68.
• Two sync contributions for validator 65.

go-ssv operators 3 and 4 each signed all three objects. Anchor operators 1 and 2 each signed only the two validator-65 contributions.

The validator-68 aggregate received only two shares. Both go-ssv runners classified the combined AggregatorCommittee duty as incomplete at slot end.

Ordinary attestations remained included, both beacon nodes stayed on the same chain, and finality advanced. The failure was isolated to AggregatorCommittee post-consensus completeness.

Relevant Anchor excerpt from operator 1:

requester=Committee { validator_partial_signature_batch_size: 3,
  base_hash: 0x08750ba150bc48eafb9ed326bf49073d82a0e18671dd79eb80de197b0cb09c03 }
root=0x6ef13e08ec0ba710a73f3966dbad3fcae9083f2b853e016b1b35978371d85797
index=ValidatorIndex(65)

requester=Committee { validator_partial_signature_batch_size: 3,
  base_hash: 0x08750ba150bc48eafb9ed326bf49073d82a0e18671dd79eb80de197b0cb09c03 }
root=0x22b763e9a915bcdb2351accba31fef28e5f141795be2576805293440d7a09095
index=ValidatorIndex(65)

# No local sign_and_collect call for validator 68 / root:
0x7c4e5f2ea1031c52ee2068e3eb66adfd98d5a64cb99a0309553bbcd8f1b47765

The same Anchor log received the validator-68 root from go-ssv operators 3 and 4. It never produced a local share for that root.

Relevant go-ssv excerpt:

got post-consensus message signer=3 validators=[68,65,65]
got post-consensus message signer=4 validators=[68,65,65]
duty did not complete before slot end (likely stuck)

Both Anchor nodes showed the same omission.

The preserved local report SHA-256 is:

9d6351de4b183f413faf7580ee3ac39ec5a861cda4e743da1bb7c814150451a9

Deterministic regression

A focused test uses one SSV committee with two locally owned validators:

• Validator 0 owns one decided aggregate.
• Validator 1 owns two decided contributions for subcommittees 0 and 1.
• Only the contribution callback is invoked.
• The signature collector mock records each SignatureRequester and signing root.

Expected: three signing calls with three unique roots. Every request must use batch size 3 and the same decided-data hash.

Actual: two signing calls. Both requests claim that the local batch size is 3.

Command:

cargo test -p anchor_validator_store contribution_callback_requests_complete_mixed_decided_batch -- --nocapture

Output on current unstable:

running 1 test

thread testing::committee_post_consensus::contribution_callback_requests_complete_mixed_decided_batch panicked at
anchor/validator_store/src/testing/committee_post_consensus.rs:75:5:
assertion `left == right` failed: the decided batch has one aggregate root and two contribution roots
  left: 2
 right: 3

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 53 filtered out

The production signature collector does not send a committee batch until the local count equals the requested count. Therefore, a two-of-three local batch remains buffered:

anchor/anchor/signature_collector/src/lib.rs

Lines 295 to 344 in 82bac14

	SignatureRequester::Committee {
	validator_partial_signature_batch_size,
	base_hash,
	} => {
	// Batch one locally produced partial signature per validator before
	// sending a single committee message for this round.
	let mut entry = match manager
	.committee_partial_signature_batches
	.entry((base_hash, metadata.committee_id))
	{
	Entry::Occupied(occupied) => occupied,
	Entry::Vacant(vacant) => vacant.insert_entry(CommitteePartialSignatureBatch {
	batched_validator_partial_signatures: Vec::with_capacity(
	validator_partial_signature_batch_size,
	),
	for_slot: metadata.slot,
	}),
	};
	let validator_partial_signature_batch =
	&mut entry.get_mut().batched_validator_partial_signatures;
	// Add the partial signature we just produced for this validator to the
	// local batch.
	validator_partial_signature_batch.push(message.clone());
	trace!(
	have = validator_partial_signature_batch.len(),
	need = validator_partial_signature_batch_size,
	"Checking whether the batch of validator partial signatures is ready to send"
	);
	// Once the local batch of validator partial signatures is complete,
	// create and send the committee message.
	if validator_partial_signature_batch.len()
	== validator_partial_signature_batch_size
	{
	let signatures =
	entry.remove().batched_validator_partial_signatures;
	let msg = match manager.create_message(
	&metadata,
	signatures,
	&DutyExecutor::Committee(metadata.committee_id),
	) {
	Ok(msg) => msg,
	Err(err) => {
	error!(%err, "Failed to create committee partial signature message");
	return;
	}
	};

Source-level mechanism

The contribution callback iterates only its local contributions vector, but computes the expected size from all locally owned decided aggregators and contributors:

anchor/anchor/validator_store/src/lib.rs

Lines 1372 to 1495 in 82bac14

	async fn sign_committee_sync_committee_contributions(
	&self,
	committee_id: CommitteeId,
	cluster: Cluster,
	contributions: Vec<(ValidatorMetadata, ContributionToSign<E>)>,
	) -> Result<Vec<SignedContributionAndProof<E>>, Error> {
	let Some((_, first)) = contributions.first() else {
	warn!("sign_committee_sync_committee_contributions called with empty contributions");
	return Ok(vec![]);
	};
	let slot = first.contribution.slot;
	let epoch = slot.epoch(E::slots_per_epoch());
	let decided_data = self
	.run_aggregator_committee_consensus(
	committee_id,
	slot,
	&cluster,
	metrics::SYNC_CONTRIBUTION_AND_PROOF,
	)
	.await?;
	let domain_hash = self.get_domain(epoch, Domain::ContributionAndProof);
	// Prepare all validators: find each in decided data, build ContributionAndProof
	// (metadata already resolved by `group_by_committee`)
	let mut prepared: Vec<SigningRequest<ContributionAndProof<E>>> =
	Vec::with_capacity(contributions.len());
	for (validator, contrib) in &contributions {
	let validator_index = match validator.index {
	Some(idx) => idx,
	None => {
	debug!(
	pubkey = ?contrib.aggregator_pubkey,
	"Validator missing index, skipping contribution"
	);
	validator_metrics::inc_counter_vec(
	&validator_metrics::SIGNED_SYNC_COMMITTEE_CONTRIBUTIONS_TOTAL,
	&[metrics::OTHER_ERROR],
	);
	continue;
	}
	};
	let subcommittee_index = contrib.contribution.subcommittee_index;
	// Find this validator+subcommittee in the decided data
	let Some(decided_contributor) = decided_data.contributors.iter().find(|c| {
	c.validator_index == validator_index && c.committee_index == subcommittee_index
	}) else {
	debug!(
	?validator_index,
	subcommittee_index,
	"Validator not in decided data, skipping due to divergent operator views"
	);
	validator_metrics::inc_counter_vec(
	&validator_metrics::SIGNED_SYNC_COMMITTEE_CONTRIBUTIONS_TOTAL,
	&[metrics::OTHER_ERROR],
	);
	continue;
	};
	// Find the contribution for this subcommittee
	let Some(decided_contribution) = decided_data
	.sync_committee_contributions
	.iter()
	.find(|c| c.subcommittee_index == subcommittee_index)
	else {
	debug!(
	subcommittee_index,
	pubkey = ?contrib.aggregator_pubkey,
	"Contribution not in consensus data - likely filtered due to beacon API failure"
	);
	validator_metrics::inc_counter_vec(
	&validator_metrics::SIGNED_SYNC_COMMITTEE_CONTRIBUTIONS_TOTAL,
	&[metrics::OTHER_ERROR],
	);
	continue;
	};
	let message = ContributionAndProof {
	aggregator_index: contrib.aggregator_index,
	contribution: decided_contribution.clone(),
	selection_proof: decided_contributor.selection_proof.clone(),
	};
	let signing_root = message.signing_root(domain_hash);
	prepared.push(SigningRequest {
	validator: validator.clone(),
	signing_root,
	duty_data: message,
	});
	}
	// Collect signatures — using `collect_signature` directly because a validator in
	// multiple subcommittees produces multiple `SigningRequest`s with different signing
	// roots, which `collect_prepared_signatures` cannot handle (it deduplicates by
	// `ValidatorIndex`).
	let committee_validator_indices = self.get_committee_validator_indices(&committee_id);
	let validator_partial_signature_batch_size = decided_data
	.post_consensus_signature_count(|idx| committee_validator_indices.contains(idx));
	let data_hash = decided_data.hash();
	let collection_mode = CollectionMode::Committee {
	validator_partial_signature_batch_size,
	base_hash: data_hash,
	};
	let sig_futures: Vec<_> = prepared
	.iter()
	.map(|req| {
	self.collect_signature(
	PartialSignatureKind::PostConsensus,
	Role::AggregatorCommittee,
	collection_mode.clone(),
	&req.validator,
	&cluster,
	req.signing_root,
	slot,
	)
	})
	.collect();
	let signature_results = join_all(sig_futures).await;

The aggregate callback has the same asymmetry. It iterates only its local aggregates vector, but uses the complete decided-data count:

anchor/anchor/validator_store/src/lib.rs

Lines 1830 to 1886 in 82bac14

	async fn sign_committee_aggregate_and_proofs(
	&self,
	committee_id: CommitteeId,
	cluster: Cluster,
	aggregates: Vec<(ValidatorMetadata, AggregateToSign<E>)>,
	) -> Result<Vec<SignedAggregateAndProof<E>>, Error> {
	let Some((_, first)) = aggregates.first() else {
	warn!("sign_committee_aggregate_and_proofs called with empty aggregates");
	return Ok(vec![]);
	};
	let slot = first.aggregate.data().slot;
	let signing_epoch = first.aggregate.data().target.epoch;
	let decided_data = self
	.run_aggregator_committee_consensus(
	committee_id,
	slot,
	&cluster,
	metrics::AGGREGATE_AND_PROOF,
	)
	.await?;
	let domain_hash = self.get_domain(signing_epoch, Domain::AggregateAndProof);
	// Prepare all validators: find each in decided data, decode aggregate, build message
	// (metadata already resolved by `group_by_committee`)
	let mut prepared: Vec<SigningRequest<(u64, AggregateAndProof<E>)>> =
	Vec::with_capacity(aggregates.len());
	for (validator, agg) in aggregates {
	match Self::resolve_decided_aggregate(&decided_data, validator, &agg, domain_hash) {
	Ok(request) => prepared.push(request),
	Err(e) => {
	debug!(pubkey = ?agg.pubkey, error = e, "Skipping aggregate");
	validator_metrics::inc_counter_vec(
	&validator_metrics::SIGNED_AGGREGATES_TOTAL,
	&[metrics::OTHER_ERROR],
	);
	}
	}
	}
	// Collect signatures and assemble results
	let committee_validator_indices = self.get_committee_validator_indices(&committee_id);
	let validator_partial_signature_batch_size = decided_data
	.post_consensus_signature_count(|idx| committee_validator_indices.contains(idx));
	let signatures = self
	.collect_prepared_signatures(
	Role::AggregatorCommittee,
	slot,
	&cluster,
	validator_partial_signature_batch_size,
	decided_data.hash(),
	&prepared,
	)
	.await?;

go-ssv handles this differently. AggregatorCommitteeRunner.ProcessConsensus runs once for the decision, enumerates both object classes from the decided value, filters them by local share and duty, then sends one post-consensus message:

https://github.com/ssvlabs/ssv/blob/a9008f3d0542f83291f2e9d18babf3edd3abd7b1/protocol/v2/ssv/runner/aggregator_committee.go#L696-L824

Expected Behaviour

After AggregatorCommittee QBFT decides, Anchor must derive the complete local post-consensus signing worklist from the decided value.

Any non-empty aggregate or contribution callback for the slot and committee must be able to trigger every locally signable decided root. Callback order must not determine which local partial signatures are emitted.

The aggregate callback must still return only requested aggregates. The contribution callback must still return only requested contributions.

Concurrent or later callbacks must reuse the same execution. They must not sign or broadcast the complete decision twice.

Steps to resolve

A likely minimal direction is:

1. Build one heterogeneous worklist from all decided aggregators and contributors.
2. Resolve local validator metadata, usable shares, and cluster state for each decided entry.
3. Preserve one entry per (validator_index, signing_root). Validator index alone is not unique because one validator can have several contribution roots.
4. Sign the complete local worklist with one CollectionMode::Committee batch.
5. Use base_hash = decided_data.hash() and validator_partial_signature_batch_size = prepared.len().
6. Make the execution once-only and cancellation-safe for (slot, committee_id, decided_hash).
7. Let both callback classes join the same result, then filter returned objects by callback class and requested identity.
8. Reject a conflicting decided hash for the same slot and committee without starting a second batch.

For ePBS compatibility, isolate this coordinator in AnchorValidatorStore or a small validator-store module. Avoid coupling the cache to AggregationAssignments or the ePBS metadata and SlotVote changes.

The aggregate decoder must use the decided data version. The ePBS branch already has DataVersion::decode_attestation, which supports the Gloas shape:

anchor/anchor/validator_store/src/lib.rs

Lines 1786 to 1820 in 36950b6

	fn resolve_decided_aggregate(
	decided_data: &AggregatorCommitteeConsensusData<E>,
	validator: ValidatorMetadata,
	agg: &AggregateToSign<E>,
	domain_hash: Hash256,
	) -> Result<SigningRequest<(u64, AggregateAndProof<E>)>, String> {
	let validator_index = validator.index.ok_or("Validator missing index")?;
	// Find this validator in the decided data
	let decided_aggregator = decided_data
	.aggregators
	.iter()
	.find(|a| a.validator_index == validator_index)
	.ok_or("Validator not in decided data")?;
	let committee_index = decided_aggregator.committee_index;
	// Find the position of this committee index in decided data
	let decided_aggregate_idx = decided_data
	.aggregator_committee_indexes
	.iter()
	.position(|&idx| idx == committee_index)
	.ok_or("Committee index not found in decided data")?;
	let decided_aggregate_bytes = decided_data
	.aggregated_attestations
	.get(decided_aggregate_idx)
	.ok_or("Aggregate attestation bytes not found in decided data")?;
	// Decode with the shape the decided version selects (Gloas merkleizes progressively,
	// so the shape drives the signing root computed below).
	let decided_aggregate: Attestation<E> = decided_data
	.version
	.decode_attestation(decided_aggregate_bytes)
	.map_err(|e| format!("Failed to decode decided aggregate: {e}"))?;

Changing the expected batch size from 3 to 2 is not a valid fix. That would let callback order produce incomplete and inconsistent local committee batches.

Acceptance criteria

• A contribution-only callback signs one decided aggregate plus all decided contributions owned by the operator.
• The mirror aggregate-only case also emits all locally signable decided roots.
• Concurrent aggregate and contribution callbacks emit one local committee message.
• A validator with several contribution subcommittees retains every distinct root.
• Foreign, missing, or unusable local entries are excluded from the expected local count.
• Dropping the first callback future does not abandon or duplicate local partial emission.
• A late callback reuses cached outcomes without another signature or network send.
• One failed root does not erase successful outcomes or restart already emitted partials.
• Existing pre-Boole behavior remains unchanged.
• The same design ports to ePBS without importing unrelated metadata, SlotVote, PTC, or Gloas work.
• A mixed two-Anchor plus two-go-ssv replay reconstructs the previously missing aggregate.

Scope boundary

This fix can start when at least one Lighthouse callback class is non-empty. It does not independently schedule or publish decided objects when both callback vectors are empty.

Full callback-independent scheduling and direct publication remain a separate architectural follow-up.

Related test-coverage issue: #929.