Proposal preparations are only resubmitted once per epoch, leaving a restarted beacon node with a stale fee recipient

[ThomasDalla]

Summary

Proposal preparations (POST /eth/v1/validator/prepare_beacon_proposer) are submitted by
Vouch on a fixed once-per-epoch schedule. Beacon nodes hold that state in memory, so a beacon
node restart discards it and the node falls back to its own CLI-level default fee recipient
until Vouch's next scheduled resubmission.

If a proposal duty falls in that window and the block ends up being built locally by the
beacon node, the block is produced with the wrong fee recipient. This happened to us on
mainnet, and nothing in Vouch's logs or metrics at default settings indicated a problem.

Two behaviours combine to make the window larger and quieter than it first appears:

1. The resubmission is time-based only. startProposalsPreparer
   (services/controller/standard/proposalspreparer.go) schedules the job at
   epochDuration*3/4 + slotDuration/2 — 294 s into each epoch on mainnet. Nothing else
   triggers a resubmission, so there is no reaction to a beacon node coming back.

2. A submission aimed at a node Vouch currently considers inactive is silently dropped.
   In updateProposalPreparations
   (services/proposalpreparer/standard/updatepreparations.go), ErrNotActive is skipped
   with a Debug log and is deliberately not counted as a failure, so the cycle is still
   recorded as proposalPreparationCompleted(..., "succeeded"):

   if err := proposalPreparationsSubmitter.SubmitProposalPreparations(ctx, proposalPreparations); err != nil {
       if errors.Is(err, eth2client.ErrNotActive) {
           // If the client isn't ready we don't count it as a failure.
           s.log.Debug()....Msg("Client is no active; cannot update proposal preparations")
           continue
       }
       ...

   That is reasonable on its own — the node genuinely cannot accept the request — but the
   skipped submission is never retried ahead of schedule. A node that is unreachable at
   294 s into epoch N gets nothing until 294 s into epoch N+1, however long before that it
   came back.

The net effect is that the restarted node can be up, answering, and synced for several
minutes while holding no fee-recipient preferences at all.

Why the state is lost on the beacon node side

prepare_beacon_proposer is documented as needing periodic resubmission, and at least Prysm
keeps it purely in memory — its handler writes to a cache with no persistence
(beacon-chain/rpc/eth/validator/handlers.go, PrepareBeaconProposer):

s.ProposerPreferencesCache.SetDefault(cache.ProposerPreference{
    ValidatorIndex: primitives.ValidatorIndex(validatorIndex),
    FeeRecipient:   bytesutil.ToBytes20(feeRecipient),
})

So the preferences do not survive a restart, and until Vouch resubmits, the node uses its own
--suggested-fee-recipient for any block it builds.

Observed timeline (mainnet)

Vouch 1.13.1, multiple beacon nodes, blockrelay with a proposerConfig assigning
per-validator fee recipients. One beacon node (call it node B) was rebooted by an
unattended OS upgrade. Times are relative to the moment Vouch lost its connection to node B;
validator index, slot numbers and hostnames are redacted.

Time	Event
T+0 s	Node B goes down. Vouch's event-stream subscription drops.
T+41 s … T+162 s	Vouch reports node B as client is not active on attestation/aggregate submissions.
T+91 s	Scheduled preparation cycle for the current epoch runs. Node B is marked inactive, so its submission is skipped via ErrNotActive. Not logged at default level; cycle recorded as succeeded.
T+174 s	Node B is answering again (errors change from not active to not synced) — i.e. back up, ~1.5 min after the skipped cycle.
T+210 s	Last not synced line; node B fully back.
T+475 s	Next scheduled preparation cycle runs — the first one since node B returned, ~5 minutes after it came back.
T+481 s	Start of our proposal slot, ~6 s later.

For that proposal the relay unblinding failed, so the block was built locally by node B — and
it carried node B's CLI-level default fee recipient rather than the per-validator address from
proposerConfig.

The ~6 s gap between the resubmission and the slot is not a safe margin in practice: a beacon
node begins building the payload for slot N before slot N starts, so the fee recipient in
use for that block was fixed before the resubmission landed. Effectively there were no valid
preparations on node B for the entire ~6.5 minutes between its restart and the proposal.

Redacted log extract

Connection to node B drops (T+0):

{"level":"error","service":"client","impl":"http","id":"<redacted>","address":"http://<node-b>:5051","error":"unexpected EOF","time":"T+0.000","message":"Failed to subscribe to event stream"}

Node B marked inactive — this is the state in which the scheduled preparation cycle at T+91 s
was skipped (T+41 s onward):

{"level":"warn","strategy":"submitter","impl":"multinode","beacon_node_address":"<node-b>:5051","slot":<redacted>,"error":"client is not active","time":"T+41.474","message":"Failed to submit aggregate attestations"}
{"level":"warn","strategy":"submitter","impl":"multinode","beacon_node_address":"<node-b>:5051","slot":<redacted>,"error":"client is not active","time":"T+42.497","message":"Failed to submit attestations"}

Node B answering again ~1.5 minutes later (T+174 s), still no preparation resubmission until
T+475 s:

{"level":"warn","strategy":"submitter","impl":"multinode","beacon_node_address":"<node-b>:5051","slot":<redacted>,"error":"client is not synced","time":"T+173.957","message":"Failed to submit attestations"}

There are no proposal-preparation log lines anywhere in the window — the skip is Debug
and success is Trace, so at default log level the whole sequence is invisible.

Impact

• Any proposal duty falling between a beacon node restart and Vouch's next scheduled
  preparation cycle can be built with the wrong fee recipient, if that node builds the block
  locally (relay unavailable, unblinding failure, min_value not met, builder disabled, …).
• The exposure window is up to one full epoch (~6.4 min on mainnet) after a restart, and up to
  two if the restart straddles a scheduled cycle, as above — the node can be fully back and
  synced for most of that time.
• The proceeds go to the beacon node's default fee recipient, which is generally not
  recoverable.
• It is silent: no error or warning at default log level, and
  vouch_proposalpreparation_process_requests_total still increments result="succeeded"
  for a cycle in which every node was skipped.

I would expect this to be more likely to bite operators who run unattended OS upgrades on
their beacon nodes, and less visible for those whose blocks almost always come from a relay,
since the wrong fee recipient only materialises on a locally-built block.

Suggestion

Rather than only resubmitting on the epoch schedule, also resubmit proposal preparations when
a beacon node transitions back to active — i.e. push prepare_beacon_proposer on reconnect,
in addition to the existing per-epoch cycle. go-eth2-client's HTTP service already exposes
an OnActive hook (http/hooks.go, WithHooks) fired from its connection-state check, so
the transition is already detected; Vouch does not currently pass hooks when constructing its
clients.

Happy to put together a PR if that direction sounds right.

Environment

• Vouch 1.13.1
• go-eth2-client v0.29.0
• Mainnet, multiple beacon nodes configured, Dirk account manager
• blockrelay with a proposerConfig defining per-validator fee recipients