Skip to content

op-supernode: hold finalized instead of publishing activation anchor on zero FinalizedL1 - #22129

Open
claude[bot] wants to merge 3 commits into
developfrom
fix/supernode-zero-finalized-anchor
Open

op-supernode: hold finalized instead of publishing activation anchor on zero FinalizedL1#22129
claude[bot] wants to merge 3 commits into
developfrom
fix/supernode-zero-finalized-anchor

Conversation

@claude

@claude claude Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Requested by Jacob Elias · Slack thread

Description

Before: a freshly created virtual node reports the interop activation anchor (e.g. L2 block 12107) as its finalized head for up to one L1EpochPollInterval (default 384s), because its StatusTracker has a zero FinalizedL1 until the first finality poll fires. The anchor is pushed to the EL as the finalized label via forkchoice update, and the poisoned label becomes the walkback floor on the next reset — turning a transient startup glitch into a ~44k-block safe rewind.

After: a fresh virtual node holds its previous finalized head (or publishes none at all when there is no previous value) until it has a real L1 finality view — and that first real poll now happens immediately on subscription instead of after a full interval.

How:

  • Interop.VerifiedBlockAtL1 (op-supernode/supernode/activity/interop/interop.go): a zero l1Block now returns an ErrNotStarted-wrapped error ("L1 finality view unknown") instead of the activation-anchor cap. FinalizedL2Head already treats verifier errors as hold-previous, so the engine controller keeps its cached finalized head and never FCUs the anchor.
  • eth.PollBlockChanges (op-service/eth/heads.go): polls once immediately on subscription before entering the ticker loop, so op-node's L1 safe/finalized subscriptions get their first signal right away rather than after 384s. This closes the zero-FinalizedL1/SafeL1 window at the source (all PollBlockChanges users get strictly earlier delivery of the same signal).

Tests

  • Inverted the existing TestVerifiedBlockAtL1 subtest ("zero l1Block returns empty immediately"), which asserted the old zero-ref-returns-anchor behavior as intended; it now asserts the error/hold-previous behavior ("zero l1Block returns error, not the activation anchor").
  • Added TestPollBlockChanges_PrimesImmediately (op-service/eth/heads_test.go): fails without the priming poll (no signal within 2s at a 384s interval), passes with it.

Fixes #22127

Restart hardening (interop-reorg-4 devnet feedback)

A devnet test running this fix hit a second path to the same anchor: after an invalid-message invalidation (167-block rewind at 259685) and an external pod restart of supernode-0, an engine reset loaded safe=finalized=<activation anchor> (genesis on that devnet) and fully re-derived both chains, while the non-restarted supernodes reset shallow to 259685.

Root cause: the restarted process cold-started with an empty verifiedDB (the pod had a bound PVC, but every in-repo mechanism for emptying the DBs is ruled out by code — the invalidation flow never touches the verifiedDB and SafeHeadReset only truncates above the rewind point — leaving a data-dir vs. volume-mount mismatch as the trigger: the --data-dir default ./datadir is CWD-relative and a bound PVC doesn't prove the process writes through it). The code then amplified that into a full re-derivation: an empty verifiedDB contributes the activation anchor via LatestVerifiedL2Block/VerifiedBlockAtL1 (paths this PR previously left intact), and the engine controller resolved it unconditionally — its downward protections (superAuthorityFinalizedHead, crossSafeCache) are in-memory and empty on a fresh process, and initializeUnknowns consulted the super authority before loading the EL's labels, so the anchor seeded the caches, skipped the finalized-label load, was published by every FCU, and became the reset's walkback floor.

Before: a restart that loses (or cannot find) the supernode datadir republishes the interop activation anchor as safe/finalized, regressing the EL labels and forcing re-derivation from the anchor on the next reset.

After: the node holds/keeps the EL's own labels instead:

  • op-node/rollup/engine/engine_controller.go: initializeUnknowns loads the EL labels before consulting the super authority and seeds superAuthorityFinalizedHead + the cross-safe cache from them (they are exactly the heads this node last published, so the existing monotonicity checks now hold across restarts); resolveAnchorAsSafe holds the cached cross-safe when the anchor is below it. A genuine bootstrap (labels at/below the anchor) still resolves the anchor.
  • op-supernode/.../interop.go: LatestVerifiedL2Block and VerifiedBlockAtL1 return ErrNotStarted (hold-previous) for an empty verifiedDB while cold-start init is incomplete (ports the a0bbd93 cold-start guard to develop) and when cold start chose a verification origin past activation — i.e. a resume with a history gap, where the anchor cap would regress consumers already far past it. Only a genuine activation bootstrap (origin at activation) still reports the anchor cap; the hold clears on the first verification-round commit.

Tests: TestSafeL2Head_Anchor_DoesNotRegressCachedCrossSafe, TestFinalizedHead_Anchor_DoesNotRegressSeededCache, TestInitializeUnknowns_SeedsSuperAuthorityStateFromELLabels, TestSafeL2Head_Anchor_GenesisBootstrap_StillResolvesAnchor (op-node/rollup/engine); TestEmptyVerifiedDB_ColdStartHold (hold while cold start incomplete / origin past activation, anchor cap on activation bootstrap, hold clears after first commit) plus updated empty-DB subtests (op-supernode interop).

Follow-ups (out of scope here)

  • Bound the reset walkback on the local finalized head instead of round-tripping own finality through the EL label (op-node/rollup/sync/start.go) — that amplification is what turned this into a ~44k-block rewind.
  • Consider op-reth rejecting a backwards finalized label in FCU.
  • Verify the devnet's --data-dir / OP_SUPERNODE_DATA_DIR actually resolves inside the PVC mount (the flag default ./datadir is CWD-relative); optionally warn at startup when the datadir is empty but the EL labels are far past activation.
  • Deep-unwind FCUs can exceed the 10s --l2.engine-rpc-timeout (11s observed), and the retry can then hit a real -38002 mid-unwind and escalate to a reset — raise the flag on invalidation-prone nodes and/or give FCU a dedicated longer timeout.

Generated by Claude Code

…on zero FinalizedL1

A fresh virtual node has a zero-valued FinalizedL1 in its StatusTracker
until the first L1 finality poll fires. VerifiedBlockAtL1 treated that
zero ref as "nothing verified" and returned the interop activation
anchor cap, so the engine controller published the activation anchor as
the finalized head, poisoned the EL finalized label via FCU, and forced
a ~44k-block safe rewind on the next reset.

Two-part fix:
- VerifiedBlockAtL1 now returns an ErrNotStarted-wrapped error for a
  zero l1Block ("L1 finality view unknown"), so FinalizedL2Head holds
  the previous value via the existing HoldPrevious path instead of
  mapping to the activation-anchor cap.
- eth.PollBlockChanges now polls immediately on subscription instead of
  waiting a full interval (384s with op-node's default
  L1EpochPollInterval), closing the zero-FinalizedL1/SafeL1 window at
  the source.

Inverts the interop test subtest that asserted the old zero-ref
behavior, and adds a regression test for the priming poll.

Fixes #22127

Co-Authored-By: Claude
Co-authored-by: Jacob Elias <jacob@oplabs.co>
@jelias2
jelias2 marked this pull request as ready for review July 30, 2026 16:51
@jelias2
jelias2 requested a review from a team as a code owner July 30, 2026 16:51
claude and others added 2 commits July 30, 2026 18:33
…L heads

On a restart where the supernode's verifier state is lost (empty
verifiedDB while the chain is far past interop activation), the empty
verifier contributes the activation anchor and the engine controller
resolved it unconditionally: its downward protections
(superAuthorityFinalizedHead, crossSafeCache) are in-memory only and
empty on a fresh process, so the anchor seeded them instead of being
rejected, every FCU then published safe/finalized at the anchor, and
the next engine reset used those EL labels as its floor — a full
re-derivation from the anchor (observed on interop-reorg-4 as a reset
to genesis after a pod restart).

Harden the controller so a fresh process cannot be re-seeded below the
EL's own labels:
- initializeUnknowns checks localFinalizedHead instead of
  FinalizedHead(), so the super authority can no longer both skip the
  EL finalized-label load and pre-seed the cache with the anchor.
- The loaded EL finalized label seeds superAuthorityFinalizedHead, and
  the loaded EL safe label seeds the cross-safe cache: they are exactly
  the heads this node last published, so the existing monotonicity
  checks now hold across restarts.
- resolveAnchorAsSafe holds the cached cross-safe when the anchor is
  below it, mirroring applyFinalizedHeadCacheChecks on the finalized
  side.

A genuine bootstrap still resolves the anchor: the guards only refuse
downward movement relative to the labels the EL already has.

Co-Authored-By: Claude
Co-authored-by: Jacob Elias <jacob@oplabs.co>
… activation bootstrap

An empty verifiedDB was always reported as the pre-activation anchor
cap. That is only correct for a genuine activation bootstrap; on a
process that cold-started with a lost datadir on a chain far past
activation (interop-reorg-4 restart), the anchor contribution regressed
the published safe/finalized heads and forced a full re-derivation on
the next engine reset.

LatestVerifiedL2Block and VerifiedBlockAtL1 now return an
ErrNotStarted-wrapped error (hold-previous, same propagation as the
zero-FinalizedL1 fix) for an empty verifiedDB when:
- cold-start init has not completed yet (a0bbd93-style guard), or
- cold start chose its verification origin past activation, meaning the
  chain has history this verifier never covered and the anchor cap
  would regress consumers already past it.

Only a cold start whose origin is at activation still reports the
anchor cap. The hold clears as soon as the first verification round
commits.

Co-Authored-By: Claude
Co-authored-by: Jacob Elias <jacob@oplabs.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant