Lighthouse proposer scheduler can skip an assigned slot after an early timer wake

[shane-moore]

Description

Anchor can skip a locally assigned proposal when the pinned Lighthouse proposer-duty timer wakes at a slot boundary but the Lighthouse slot clock still returns the previous slot.

This was observed on one Anchor operator during a mixed 2 Anchor plus 2 go-ssv stress run. The affected operator processed slot 78 again, sent a stale slot-78 notification, and then slept until the next boundary without ever checking slot 79. The other three operators reached the 3-of-4 threshold and produced canonical block 79.

The bug is most likely owned by Lighthouse validator_services. It is tracked here because Anchor pins and embeds that code, needs a dependency integration path, and can ship a temporary patched Lighthouse revision if required. The observation occurred before the Boole activation in this run, so the failure is not Boole-specific.

Version

• Anchor unstable: fd3c7f9677b2bbb04fe7d7c2d4eca81313db7b79
• Anchor Lighthouse pin: b263df596671a2bd42bf1034e1cdc8188ba8a9b0
• Lighthouse current unstable checked on 2026-08-06: 02e8732a81a68784ce105609942570470a7989f2, with the same scheduler behavior still present
• ssv-mini: fef3b7d791384911078b2ce977100f87679e5c46
• go-ssv stage: 874ba316266aec6215bc21c6e2e929e195f08d6b
• Project Docker Rust toolchain: 1.93.1
• Runtime image: node/anchor:boole-proposer-diagnostic-fd3c7f9-b263df5
• Image ID: sha256:5a459c08cf1d8fc3d6a4b06f76249c6273f07c1d782f2be5a38d6c288a2aa7ee
• Anchor binary SHA-256: 41c7addc3ef97b8be130e0e34690b43c05fa03714114209c9ed1c108d13303d3

Anchor's pin is visible in Cargo.toml.

Test topology

• 2 Anchor operators, operator IDs 1 and 2
• 2 go-ssv operators, operator IDs 3 and 4
• 10 managed validators, indices 64 through 73
• 3-of-4 threshold
• 74 total preregistered validators
• Fulu at genesis
• Boole at epoch 6
• 12-second slots
• Enclave: boole-proposer-diagnostic-1
• Enclave UUID: 1ed99ef7ef154ddba6dc20a9c1b6e731

Present Behaviour

Validator index 64, pubkey:

0x8de7ec501d574152f52a962bf588573df2fc3563fd0c6077651208ed20f24f3d8572425706b343117b48bdca56808416

was assigned to propose at slot 79.

Anchor operator 2 logged:

2026-08-06T23:36:30.002481Z DEBUG Checked cached block proposers slot=78 epoch=2 epoch_cached=true exact_slot_cached_count=1 eligible_proposer_count=1
2026-08-06T23:36:30.045650Z DEBUG Sent block production notification current_slot=78 proposer_count=1
2026-08-06T23:36:30.045791Z DEBUG Received block production notification notification_slot=78
2026-08-06T23:36:30.045812Z WARN  Skipping block production for expired slot current_slot=79 notification_slot=78 info="Your machine could be overloaded"
2026-08-06T23:36:30.091232Z DEBUG Downloaded proposer duties epoch=2 num_relevant_duties=7
2026-08-06T23:36:30.095532Z DEBUG Downloaded proposer duties epoch=3 num_relevant_duties=1
2026-08-06T23:36:30.095688Z DEBUG Checked cached block proposers slot=78 epoch=2 epoch_cached=true exact_slot_cached_count=1 eligible_proposer_count=1

There was no slot-79 proposer-cache lookup, slot-79 notification, RANDAO entry, unsigned-block request, or proposal publication on this operator.

For comparison, Anchor operator 1 entered the correct path:

2026-08-06T23:36:30.016695Z DEBUG Checked cached block proposers slot=79 epoch=2 epoch_cached=true exact_slot_cached_count=1 eligible_proposer_count=1
2026-08-06T23:36:30.016750Z DEBUG Sent block production notification current_slot=79 proposer_count=1
2026-08-06T23:36:30.018371Z DEBUG Received block production notification notification_slot=79
2026-08-06T23:36:30.019433Z TRACE Proposer randao reveal entered slot=79
2026-08-06T23:36:30.266510Z TRACE checkpoint="randao_reveal_completed" outcome="success" slot=79
2026-08-06T23:36:30.266529Z INFO  Requesting unsigned block slot=79
2026-08-06T23:36:30.492848Z INFO  Received unsigned block slot=79
2026-08-06T23:36:30.970861Z DEBUG New head from beacon node block_root=0xaeca1f6a53a616e3eee636a55454a2acf87e5db2044153dfe253bddd2a4258d7 slot=79

go-ssv operator 4 independently showed that its boundary estimator still considered the current slot to be 78, then completed slot 79 successfully:

{"time":"2026-08-06T23:36:30.009266Z","msg":"duty did not complete before slot end (likely stuck)","runner_role":"PROPOSER","slot":79,"estimated_current_slot":"78"}
{"time":"2026-08-06T23:36:31.008601Z","msg":"successfully submitted block proposal","runner_role":"PROPOSER","slot":79,"estimated_current_slot":"79","block_root":"aeca1f6a53a616e3eee636a55454a2acf87e5db2044153dfe253bddd2a4258d7"}

The warning's overload explanation is not supported by the resource sample at 23:36:31Z:

Service	CPU	Memory	Restarts	OOM
Anchor operator 1	1.31%	42,561,700 bytes	0	false
Anchor operator 2	0.36%	43,830,477 bytes	0	false

There were no monitor alerts at this sample.

Failure mechanism

The pinned Lighthouse proposer task does this once per loop:

1. Call duration_to_next_slot().
2. Sleep for that duration.
3. Immediately call poll_beacon_proposers().
4. Read slot_clock.now() once and use that slot for both proposer-cache lookups and notifications.

See the proposer loop and the single current-slot read.

SystemTimeSlotClock::duration_to_next_slot() calculates a duration from SystemTime. The async timer then waits for that duration, but the proposer loop does not verify that the intended next slot has actually started before polling.

In this event, the timer woke at the boundary and now() still returned slot 78. The whole iteration therefore processed slot 78. By the time that iteration finished, the outer loop calculated its next wait from slot 79 to slot 80, so slot 79 was never polled on that operator.

The first missing stage is before Anchor RANDAO, QBFT, and block production. This is not a duty-cache miss, network quorum failure, Boole domain mismatch, or resource saturation event.

Reproduction

The original runtime reproduction is retained locally:

• Params: /Users/shanemoore/Code/dev/ethereum/ssv-mini-boole-stress/params-boole-proposer-diagnostic.yaml
• Logs and resource samples: /var/folders/4l/n1rhxk3x4ng6tylztyffl4g80000gn/T/ssv-mini-boole-proposer-diagnostic.XXXXXXXX.kSU2qnePGN
• Resource run ID: 0117faf6-d5e8-420f-956f-88c9fce5a4ce

Run from the pinned ssv-mini checkout:

make run PARAMS_FILE=params-boole-proposer-diagnostic.yaml ENCLAVE_NAME=boole-proposer-diagnostic-1

The diagnostic Lighthouse patch only adds logs around:

• exact-slot proposer-cache lookup,
• block-production notification send,
• block-production notification receive.

It does not change protocol or scheduling behavior.

A deterministic regression test should avoid waiting for a rare live clock boundary:

1. Use a controllable SlotClock, such as ManualSlotClock or a narrow test clock.
2. Start the proposer scheduler in slot S - 1.
3. Let its sleep complete while now() still returns S - 1.
4. Then advance the clock to S.
5. Assert that the scheduler waits or retries until S, emits exactly one notification for S, and does not sleep through to S + 1.

Expected Behaviour

After waiting for the next-slot boundary, the proposer scheduler must confirm that the intended target slot has started before polling duties.

An early timer wake must not cause:

• a stale previous-slot notification,
• an expired-slot warning,
• omission of the target-slot proposer-cache lookup,
• omission of a locally assigned proposal.

Steps to resolve

Likely Lighthouse fix:

1. Capture the intended target slot before sleeping.
2. After every wake, re-read slot_clock.now().
3. If the clock still reports a slot before the target, wait again using duration_to_slot(target_slot) or an equivalent bounded retry.
4. Poll proposer duties only after the target slot is visible.
5. Define conservative behavior when the slot clock is unavailable or moves backwards.
6. Add deterministic early-wake regression coverage.
7. Confirm that the retry cannot emit duplicate block-production notifications.

Anchor follow-up:

1. Track or open the corresponding Lighthouse issue and fix.
2. Temporarily pin a patched Lighthouse revision if the upstream release path is too slow.
3. Run focused Lighthouse tests and Anchor proposer-service tests.
4. Repeat the mixed-client proposer diagnostic and confirm there are no stale notifications or skipped assigned slots.
5. Update Anchor's Lighthouse pin after the upstream fix lands.

Acceptance criteria

• An early wake before slot S cannot cause the scheduler to skip S.
• The assigned proposer for S receives exactly one usable notification.
• No duplicate proposal work is introduced.
• A deterministic regression test fails on b263df5 and passes with the fix.
• Anchor integrates the fixed Lighthouse revision.
• The mixed 2 Anchor plus 2 go-ssv diagnostic completes repeated managed proposal duties without this failure signature.

Scope

This issue covers only the early-wakeup proposer scheduling failure at slot 79. The separate post-suspension partial-signature delivery failure observed later at slot 216 has a different mechanism and is intentionally out of scope.

[shane-moore]

2026-08-09 scale-100 follow-up

The targeted rerun confirmed the proposer mechanism and found a separate Anchor-owned metadata race with the same early-wake shape.

Pinned run

• Anchor: fix/1227-devnet8-port at dbaf89bc00cb09fd032b342fd502292399166b84
• Anchor base: upstream/devnet8 at 9bad97efb2af627f2f9fe0b8188029bb46ab23b6
• Lighthouse: 512f3035b93734dde422dda30f501912e9716770
• Image: node/anchor:devnet8-1227-targetdiag-dbaf89bc
• Image ID: sha256:18941d5610761638e1f366b92d4a325a7abcc9750952a1fad9b35ea3b2ab9e0a
• Anchor binary SHA-256: 449c38880e09e2221f1cfbf5694f5e14036e5a9a9c04d5ab0cefcc3a445365df
• Topology: four Anchor operators, 100 managed validators, 64 baseline validators, and two Lighthouse plus Geth pairs
• Scored slots: formal windows 256 through 511 and steady-state soak 512 through 1023

Lighthouse proposer scheduler recurrence

The 512-slot soak contained 312 managed proposal assignments and 308 canonical managed blocks. Managed proposal slots 536, 561, 621, and 668 missed.

Seven unique proposer early-wake targets appeared across the operators: 536, 558, 561, 586, 621, 668, and 843.

The results covered the complete threshold progression:

• Slot 561 and slot 668: zero operators processed the target slot. No RANDAO collector started, and each block was silently missed.
• Slot 536: one operator processed the target. Its collector ended with one of three required shares.
• Slot 621: two operators processed the target. Both collectors ended with two of three required shares.
• Slot 843: three operators processed the target. Anchor reached the threshold and published the canonical block.

This confirms that collector cleanup is a downstream symptom. No collector exists when every operator skips the target slot.

The current Lighthouse stable and unstable branches still contain the same proposer loop as of 2026-08-10. The loop sleeps, then polls the slot returned by a new now() read. It does not enforce the intended target slot.

Production severity boundary

Different beacon-node choices do not affect this trigger. The local process clock selects the slot before a relevant beacon-node response.

All four Anchor containers in this run shared one Docker VM and system clock. This setup can amplify correlated early wakes.

Production Anchor operators usually use independent hosts and clocks. One skipped operator remains masked in a healthy 3-of-4 committee. Failure requires a second skip or another unavailable operator.

This run proves the mechanism and threshold effect. It does not establish the production coincidence rate on independently hosted operators.

The stock Lighthouse validator client uses the same SystemTimeSlotClock and proposer scheduler. A standalone Lighthouse validator client has no DVT threshold to mask a skip.

Anchor will not pin a private Lighthouse fork. The proposer fix must land in the official Lighthouse repository before Anchor updates its official Lighthouse pin.

Separate Anchor metadata scheduler race

Anchor MetadataService has a separate instance of the same unsafe scheduling assumption.

Phase 1 calculates the next-slot duration and sleeps. It then calls update_voting_assignments() without a target argument. That helper reads slot_clock.now() again and publishes assignments for the reported slot.

If the clock still reports N - 1, Anchor republishes old assignments. The next loop sleeps toward N + 1, so waiters for N observe the channel jump past their slot and return MetadataSlotPassed.

Observed recurrences were:

• Warm-up target 218: one-operator local gap.
• Formal target 451: two operators skipped, causing cluster contribution gaps at slots 451 through 453.
• Formal target 508: one-operator local gap at slots 508 through 510.
• Soak targets 546, 586, and 823: two operators skipped each target, causing nine cluster-wide missing contribution slots in total.
• Soak targets 558, 668, and 843: one-operator local gaps, masked by the other three operators.

Formal Window 1 was clean for all 128 slots. All 158 managed proposals across both formal windows completed every instrumented stage.

The metadata failure occurs before partial-signature collection. No affected metadata event ended a live collector before reconstruction.

The Anchor fix must preserve target slot N, wait until the clock reports at least N, and publish assignments explicitly for N. A deterministic test must force an early wake and reject N - 1 or N + 1 publication.

This metadata fix belongs in Anchor. It is independent of the Lighthouse proposer fix and independent of issue #1227.

Health controls and evidence

• No Anchor restart, OOM event, non-running sample, or resource alert occurred.
• Both beacon nodes agreed on canonical, non-optimistic heads.
• Epoch 31 reported all 100 managed validators live.
• Epoch 30 had positive source, target, and head reward components for all managed validators.
• Local evidence root: /tmp/ssv-mini-devnet8-1227-targetdiag.nKOjXX
• Integrity manifest: /tmp/ssv-mini-devnet8-1227-targetdiag.nKOjXX/evidence-sha256.txt

The issue therefore tracks two ownership paths:

1. Lighthouse proposer scheduler: upstream Lighthouse fix, then an official Anchor pin update.
2. Anchor metadata scheduler: direct Anchor fix and regression coverage.