Skip to content

Update proposal preparations when a consensus client becomes active - #431

Open
ThomasDalla wants to merge 1 commit into
attestantio:masterfrom
ThomasDalla:reconnect-prepare-proposer
Open

Update proposal preparations when a consensus client becomes active#431
ThomasDalla wants to merge 1 commit into
attestantio:masterfrom
ThomasDalla:reconnect-prepare-proposer

Conversation

@ThomasDalla

Copy link
Copy Markdown

Fixes #430.

What this changes

Beacon nodes hold proposal preparations in memory, so a node that restarts has none until
Vouch's next scheduled update. That update runs once per epoch, and a node that is inactive
when it runs is skipped via ErrNotActive and not retried until the following epoch — so a
node can be back up and synced for minutes with no fee-recipient preferences. A proposal in
that window, if the block is built locally, uses the beacon node's own default fee recipient.

This updates proposal preparations as soon as a consensus client becomes active, in addition
to the existing per-epoch update.

Why this shape

  • go-eth2-client already detects the transition. http.Hooks.OnActive is fired from
    CheckConnectionState when a client goes from inactive to active, on the existing 30-second
    connection-state poll. Vouch simply never passed hooks when constructing its clients, so the
    change is to provide them — no new polling, timers or connection tracking.

  • Callbacks are registered in clients.go because fetchClient is where every client is
    constructed and cached, and the hooks have to be supplied at construction time. The hook
    reads the callback list when it fires rather than capturing it, so it does not matter whether
    a client is constructed before or after a callback is registered.

  • UpdatePreparations is called rather than targeting the client that just became active.
    It already submits to every configured submitter in one call, which is cheap, and it means a
    reconnect that coincides with another node also being unavailable still ends up updating both
    once they are back. Targeting a single node would need address-matching against the submitter
    list for no practical gain.

  • The nil check on proposalPreparer is deliberate. initProposalPreparer returns a nil
    service (with a nil error) when the chain is not yet bellatrix-capable; without the check the
    first activation would panic inside a client-owned goroutine.

  • Callbacks are called synchronously. OnActive is already invoked in its own goroutine by
    the client, so there is nothing to gain from another one, and this keeps a panic attributable.
    The callback list is copied under the read lock before dispatch, so a callback that registers
    another callback cannot deadlock — covered by a test.

What this does not cover

  • Connection state is polled every 30 seconds, so a beacon node that restarts and starts
    answering again within a single poll interval is never observed as inactive and no update is
    triggered. This makes the exposure window small in the common case rather than removing it
    entirely. Tying preparation updates to known proposal duties would close it properly, but
    that is a larger change and I did not want to bundle it here.

  • Validator registrations (blockrelay) have the same once-per-epoch shape and could reuse
    the same callback, but they were not the cause of the issue, so I have left them alone to
    keep this focused. Happy to add that here or separately if you would like it.

Testing

  • go build ./..., go vet ./... and go test ./... all pass.
  • New unit tests cover callback dispatch, the no-callbacks case, dispatch on each activation,
    and registration from within a callback; they pass under -race.
  • I intend to run this build against mainnet beacon nodes and can report back on how it
    behaves after a real beacon node restart.

Happy to adjust naming, comments, or add a CHANGELOG entry if you would like this in a
different form.

@ThomasDalla
ThomasDalla force-pushed the reconnect-prepare-proposer branch 2 times, most recently from 2891fe6 to de30bd7 Compare August 18, 2026 21:04
@ThomasDalla

Copy link
Copy Markdown
Author

Confirmed working against mainnet. Three configured beacon nodes, one of them restarted deliberately; timings are relative to the moment its connection dropped:

T+ Event
0s Node goes down; event-stream subscription drops
0s – 2m39s client is not active on submissions to that node
~2m45s Node active again → proposal preparations submitted successfully to all configured nodes
2m51s client is not synced — the update landed while the node was still syncing, before it could be asked to propose

vouch_proposalpreparation_process_requests_total{result="succeeded"} incremented at that point, with no result="failed" series and no proposalpreparer error lines. The next scheduled update was ~3 minutes later, so the increment is unambiguously the reconnect path rather than the periodic one — without this change the node would have held no preparations for that entire window.

Duties were unaffected throughout: submissions to the restarted node failed with client is not active for the duration of the outage, but the multinode submitter covered them via the other two nodes and vouch_attestation_process_requests_total{result="succeeded"} kept climbing with no failures.

Two things worth flagging from the run:

  • The update also fires once shortly after startup, as clients first transition to active. Harmless and arguably useful, but easy to suppress if you would rather it only ran on genuine reconnections.
  • The callback deliberately uses the service context, not the one supplied to the hook. assertIsActive checks connection state on the way in to a request that finds the client inactive, so OnActive can be invoked with that request's context, which is cancelled as soon as the request completes — long enough to start the preparation POSTs and not long enough to finish them. If you would prefer that handled in go-eth2-client instead (passing a non-cancellable context to the hooks, or documenting that the hook's context is not for retained work), I am happy to do it there.

The node here took ~2m45s to become active again, well past the 30s connection-state poll, so this run does not exercise the limitation noted in the description — a restart fast enough to go unobserved between polls still gets no update.

Node identifiers and validator details omitted; happy to provide anything more specific that would help review.

Proposal preparations are held in memory by beacon nodes, so a node that
restarts loses them until vouch's next scheduled update, which runs once
per epoch.  A node that is inactive when that update runs is skipped, and
not retried until the following epoch.  A proposal in that window can be
made with the beacon node's own default fee recipient rather than the
configured one.

Provide hooks to the consensus clients so that proposal preparations are
updated as soon as a client becomes active, in addition to the existing
per-epoch update.

The updates use the service context rather than the one supplied to the
hook.  A client checks its connection state on the way in to a request that
finds it inactive, so the hook can be called with that request's context,
which is cancelled as soon as the request completes; using it cancels the
preparations mid-flight.
@ThomasDalla
ThomasDalla force-pushed the reconnect-prepare-proposer branch from de30bd7 to ccdd500 Compare August 19, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal preparations are only resubmitted once per epoch, leaving a restarted beacon node with a stale fee recipient

1 participant