fix(runner): conclude benign ErrNoValidDutiesToExecute as not_required, not failed - #2988
fix(runner): conclude benign ErrNoValidDutiesToExecute as not_required, not failed#2988momosh-ssv wants to merge 2 commits into
Conversation
…d, not failed A committee or aggregator-committee duty that reaches post-consensus quorum but leaves this operator with no beacon objects to submit is a normal "nothing to do" terminal (divergent validator sets across the committee's operators), yet both runners classified it as a failed outcome — emitting a spurious "duty failed" warning and a false ssv.runner.duty.outcome=failed data point, while the queue simultaneously treated the same sentinel as a benign terminal drop. Conclude the branch as not_required instead, in both runners: - AggregatorCommitteeRunner: direct swap of markDutyFailed for markDutyNotRequired at the len(beaconObjects) == 0 branch. - CommitteeRunner: markDutyNotRequired before the sentinel return; the deferred markDutyFailed becomes a no-op via concludeDuty idempotency. - CommitteeRunner.ProcessConsensus: the zero-valid-duties sentinel had no marker at all (false "stuck"); conclude it as not_required too. The sentinel is still returned in every case, preserving committee_queue's terminal-drop handling. Genuine terminal failures (expected-roots errors, terminal BLS reconstruction, submit failures) remain classified failed. Closes #2903
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Greptile SummaryThis PR changes zero-work committee-runner terminals from failed or unconcluded to not_required while preserving ErrNoValidDutiesToExecute for queue termination.
Confidence Score: 4/5The PR should not merge until the committee post-consensus path distinguishes benign absence of duties from failures that prevent every beacon object from being constructed. CommitteeRunner can swallow all per-validator construction or beacon-data errors into an empty object set, and the changed branch then records that missed submission as a successful not_required outcome. Files Needing Attention: protocol/v2/ssv/runner/committee.go
|
| Filename | Overview |
|---|---|
| protocol/v2/ssv/runner/committee.go | Adds not_required conclusions to zero-duty and empty-object paths, but the latter also masks cases where all object-building operations fail. |
| protocol/v2/ssv/runner/aggregator_committee.go | Reclassifies a genuinely empty aggregator/contributor assignment as not_required while retaining propagated derivation errors as failures. |
| protocol/v2/ssv/runner/committee_postconsensus_classification_test.go | Covers guard-invalidated empty-object and zero-duty outcomes but not an empty result caused by object-construction or beacon-data failures. |
| protocol/v2/ssv/runner/aggregator_committee_test.go | Verifies that decided data with no aggregator or contributor assignment concludes not_required and preserves the queue sentinel. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Consensus decides] --> B{Valid local duties?}
B -->|No| C[Conclude not_required]
B -->|Yes| D[Collect post-consensus quorum]
D --> E[Build beacon objects]
E --> F{Objects produced?}
F -->|Yes| G[Submit to beacon node]
F -->|No| C
C --> H[Return ErrNoValidDutiesToExecute]
H --> I[Committee queue terminates runner]
Reviews (1): Last reviewed commit: "fix(runner): conclude benign ErrNoValidD..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| // the deferred markDutyFailed becomes a no-op. The sentinel still tells committee_queue to | ||
| // drop the message and terminate the runner. | ||
| r.markDutyNotRequired() |
There was a problem hiding this comment.
When every validator is skipped because duty validation, object construction, domain-data retrieval, or signing-root computation fails, this branch records the empty result as successful not_required, suppressing the failed outcome and warning for a missed submission.
Knowledge Base Used: Protocol v2 Duty Runners and QBFT Consensus
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
The committee_queue case that catches ErrNoValidDutiesToExecute logged "❗ could not handle message, dropping message and terminating committee-runner" at Error level and marked the trace span as Error — the same false alarm as the failed outcome classification, fired by the same benign trigger. The runner now concludes the duty as not_required on this sentinel, so the drop is a correct completion: log it at Debug with neutral wording and set the span status to Ok. The drop-and- terminate behavior is unchanged.
| // not_required — not failed — before returning the sentinel; concludeDuty is idempotent, so | ||
| // the deferred markDutyFailed becomes a no-op. The sentinel still tells committee_queue to | ||
| // drop the message and terminate the runner. | ||
| r.markDutyNotRequired() |
There was a problem hiding this comment.
Building on the existing P1 by Greptile from above (empty objects masking construction failures) rather than repeating it — two clarifications that should help decide the fix:
A template already exists in the sibling runner. AggregatorCommitteeRunner hits the same empty-objects terminal but deliberately surfaces construction/domain errors instead of swallowing them, so a genuine failure stays failed rather than becoming not_required — see the note at aggregator_committee.go#L1441-L1445. Here, expectedPostConsensusRootsAndBeaconObjects instead does logger.Debug(...); continue on every construct / DomainData / signing-root failure, so the two siblings disagree on what an empty map means. Returning an error when a validator is skipped for a non-benign reason (vs. guard-invalid) would realign them and preserve failed.
Severity is likely below P1 in practice. DomainData for this domain/epoch is already fetched successfully during consensus-phase signing (signBeaconObject) and is normally cached, so an all-validators failure surfacing only at post-consensus is an unlikely path. The guard-invalid case (divergent validator sets — the actual #2903 trigger) is the dominant real cause of an empty map.
Scope note: only this post-consensus branch is affected. The consensus-phase sibling change at L381-L387 is unambiguous — totalAttesterDuties is incremented before signing and signing errors return early via errCh, so a zero count there can only mean guard-invalid, never a swallowed failure.
2 participants
What this fixes
Fixes #2903.
Sometimes a committee reaches consensus but one of its operators ends up with nothing to submit to the beacon node. This is normal — it happens when the operators in a committee don't run the exact same set of validators. The node handled the situation correctly, but it reported it as a failure: operators saw a
⚠️ duty failedwarning in their logs and afaileddata point in the duty outcome metric, for something that wasn't a failure at all.This PR makes the node report that situation for what it is: a duty that completed correctly with nothing to submit (
not_required).What changed
First commit — how the duty outcome is classified:
not_requiredinstead offailed.not_requiredbefore the error-handling defer can mark itfailed(recording an outcome twice is safe — only the first one counts).not_requiredas well.Second commit — the log line in the message queue:
When this situation occurs, the queue also printed an error-level log line (
❗ could not handle message...) and marked the trace as an error. Same false alarm, same harmless trigger. It's now a debug-level line with neutral wording, and the trace is marked OK.What did NOT change
failed— beacon submission errors, signature reconstruction failures, and so on.Why
not_requiredand not a new labelThe issue asked for a decision here.
not_requiredalready exists and means exactly this ("completed correctly with nothing to submit") — it's what the aggregator runner uses when a validator turns out not to be an aggregator. Reusing it avoids adding a new metric label for the same idea.Testing
Three new regression tests, one per fixed spot. Each was verified to fail on the old code and pass with the fix. The full
runnerandvalidatortest suites pass.