op-supernode: hold finalized instead of publishing activation anchor on zero FinalizedL1 - #22129
Open
claude[bot] wants to merge 3 commits into
Open
op-supernode: hold finalized instead of publishing activation anchor on zero FinalizedL1#22129claude[bot] wants to merge 3 commits into
claude[bot] wants to merge 3 commits into
Conversation
…on zero FinalizedL1
A fresh virtual node has a zero-valued FinalizedL1 in its StatusTracker
until the first L1 finality poll fires. VerifiedBlockAtL1 treated that
zero ref as "nothing verified" and returned the interop activation
anchor cap, so the engine controller published the activation anchor as
the finalized head, poisoned the EL finalized label via FCU, and forced
a ~44k-block safe rewind on the next reset.
Two-part fix:
- VerifiedBlockAtL1 now returns an ErrNotStarted-wrapped error for a
zero l1Block ("L1 finality view unknown"), so FinalizedL2Head holds
the previous value via the existing HoldPrevious path instead of
mapping to the activation-anchor cap.
- eth.PollBlockChanges now polls immediately on subscription instead of
waiting a full interval (384s with op-node's default
L1EpochPollInterval), closing the zero-FinalizedL1/SafeL1 window at
the source.
Inverts the interop test subtest that asserted the old zero-ref
behavior, and adds a regression test for the priming poll.
Fixes #22127
Co-Authored-By: Claude
Co-authored-by: Jacob Elias <jacob@oplabs.co>
jelias2
marked this pull request as ready for review
July 30, 2026 16:51
…L heads On a restart where the supernode's verifier state is lost (empty verifiedDB while the chain is far past interop activation), the empty verifier contributes the activation anchor and the engine controller resolved it unconditionally: its downward protections (superAuthorityFinalizedHead, crossSafeCache) are in-memory only and empty on a fresh process, so the anchor seeded them instead of being rejected, every FCU then published safe/finalized at the anchor, and the next engine reset used those EL labels as its floor — a full re-derivation from the anchor (observed on interop-reorg-4 as a reset to genesis after a pod restart). Harden the controller so a fresh process cannot be re-seeded below the EL's own labels: - initializeUnknowns checks localFinalizedHead instead of FinalizedHead(), so the super authority can no longer both skip the EL finalized-label load and pre-seed the cache with the anchor. - The loaded EL finalized label seeds superAuthorityFinalizedHead, and the loaded EL safe label seeds the cross-safe cache: they are exactly the heads this node last published, so the existing monotonicity checks now hold across restarts. - resolveAnchorAsSafe holds the cached cross-safe when the anchor is below it, mirroring applyFinalizedHeadCacheChecks on the finalized side. A genuine bootstrap still resolves the anchor: the guards only refuse downward movement relative to the labels the EL already has. Co-Authored-By: Claude Co-authored-by: Jacob Elias <jacob@oplabs.co>
… activation bootstrap An empty verifiedDB was always reported as the pre-activation anchor cap. That is only correct for a genuine activation bootstrap; on a process that cold-started with a lost datadir on a chain far past activation (interop-reorg-4 restart), the anchor contribution regressed the published safe/finalized heads and forced a full re-derivation on the next engine reset. LatestVerifiedL2Block and VerifiedBlockAtL1 now return an ErrNotStarted-wrapped error (hold-previous, same propagation as the zero-FinalizedL1 fix) for an empty verifiedDB when: - cold-start init has not completed yet (a0bbd93-style guard), or - cold start chose its verification origin past activation, meaning the chain has history this verifier never covered and the anchor cap would regress consumers already past it. Only a cold start whose origin is at activation still reports the anchor cap. The hold clears as soon as the first verification round commits. Co-Authored-By: Claude Co-authored-by: Jacob Elias <jacob@oplabs.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Jacob Elias · Slack thread
Description
Before: a freshly created virtual node reports the interop activation anchor (e.g. L2 block 12107) as its finalized head for up to one
L1EpochPollInterval(default 384s), because itsStatusTrackerhas a zeroFinalizedL1until the first finality poll fires. The anchor is pushed to the EL as the finalized label via forkchoice update, and the poisoned label becomes the walkback floor on the next reset — turning a transient startup glitch into a ~44k-block safe rewind.After: a fresh virtual node holds its previous finalized head (or publishes none at all when there is no previous value) until it has a real L1 finality view — and that first real poll now happens immediately on subscription instead of after a full interval.
How:
Interop.VerifiedBlockAtL1(op-supernode/supernode/activity/interop/interop.go): a zerol1Blocknow returns anErrNotStarted-wrapped error ("L1 finality view unknown") instead of the activation-anchor cap.FinalizedL2Headalready treats verifier errors as hold-previous, so the engine controller keeps its cached finalized head and never FCUs the anchor.eth.PollBlockChanges(op-service/eth/heads.go): polls once immediately on subscription before entering the ticker loop, so op-node's L1 safe/finalized subscriptions get their first signal right away rather than after 384s. This closes the zero-FinalizedL1/SafeL1window at the source (allPollBlockChangesusers get strictly earlier delivery of the same signal).Tests
TestVerifiedBlockAtL1subtest ("zero l1Block returns empty immediately"), which asserted the old zero-ref-returns-anchor behavior as intended; it now asserts the error/hold-previous behavior ("zero l1Block returns error, not the activation anchor").TestPollBlockChanges_PrimesImmediately(op-service/eth/heads_test.go): fails without the priming poll (no signal within 2s at a 384s interval), passes with it.Fixes #22127
Restart hardening (interop-reorg-4 devnet feedback)
A devnet test running this fix hit a second path to the same anchor: after an invalid-message invalidation (167-block rewind at 259685) and an external pod restart of supernode-0, an engine reset loaded
safe=finalized=<activation anchor>(genesis on that devnet) and fully re-derived both chains, while the non-restarted supernodes reset shallow to 259685.Root cause: the restarted process cold-started with an empty verifiedDB (the pod had a bound PVC, but every in-repo mechanism for emptying the DBs is ruled out by code — the invalidation flow never touches the verifiedDB and
SafeHeadResetonly truncates above the rewind point — leaving a data-dir vs. volume-mount mismatch as the trigger: the--data-dirdefault./datadiris CWD-relative and a bound PVC doesn't prove the process writes through it). The code then amplified that into a full re-derivation: an empty verifiedDB contributes the activation anchor viaLatestVerifiedL2Block/VerifiedBlockAtL1(paths this PR previously left intact), and the engine controller resolved it unconditionally — its downward protections (superAuthorityFinalizedHead,crossSafeCache) are in-memory and empty on a fresh process, andinitializeUnknownsconsulted the super authority before loading the EL's labels, so the anchor seeded the caches, skipped the finalized-label load, was published by every FCU, and became the reset's walkback floor.Before: a restart that loses (or cannot find) the supernode datadir republishes the interop activation anchor as safe/finalized, regressing the EL labels and forcing re-derivation from the anchor on the next reset.
After: the node holds/keeps the EL's own labels instead:
op-node/rollup/engine/engine_controller.go:initializeUnknownsloads the EL labels before consulting the super authority and seedssuperAuthorityFinalizedHead+ the cross-safe cache from them (they are exactly the heads this node last published, so the existing monotonicity checks now hold across restarts);resolveAnchorAsSafeholds the cached cross-safe when the anchor is below it. A genuine bootstrap (labels at/below the anchor) still resolves the anchor.op-supernode/.../interop.go:LatestVerifiedL2BlockandVerifiedBlockAtL1returnErrNotStarted(hold-previous) for an empty verifiedDB while cold-start init is incomplete (ports the a0bbd93 cold-start guard to develop) and when cold start chose a verification origin past activation — i.e. a resume with a history gap, where the anchor cap would regress consumers already far past it. Only a genuine activation bootstrap (origin at activation) still reports the anchor cap; the hold clears on the first verification-round commit.Tests:
TestSafeL2Head_Anchor_DoesNotRegressCachedCrossSafe,TestFinalizedHead_Anchor_DoesNotRegressSeededCache,TestInitializeUnknowns_SeedsSuperAuthorityStateFromELLabels,TestSafeL2Head_Anchor_GenesisBootstrap_StillResolvesAnchor(op-node/rollup/engine);TestEmptyVerifiedDB_ColdStartHold(hold while cold start incomplete / origin past activation, anchor cap on activation bootstrap, hold clears after first commit) plus updated empty-DB subtests (op-supernode interop).Follow-ups (out of scope here)
op-node/rollup/sync/start.go) — that amplification is what turned this into a ~44k-block rewind.--data-dir/OP_SUPERNODE_DATA_DIRactually resolves inside the PVC mount (the flag default./datadiris CWD-relative); optionally warn at startup when the datadir is empty but the EL labels are far past activation.--l2.engine-rpc-timeout(11s observed), and the retry can then hit a real-38002mid-unwind and escalate to a reset — raise the flag on invalidation-prone nodes and/or give FCU a dedicated longer timeout.Generated by Claude Code