feat(archiver): verify the stored tip is canonical on start and reconnect - #1347
DylanVerstraete wants to merge 1 commit into
Conversation
PR SummaryHigh Risk Overview Before (re)starting the stream it re-fetches the stored tip block and compares RPC hash to the persisted block hash. Match resumes normally; legacy rows without a hash warn and continue; mismatch fails closed with Wiring: new Reviewed by Cursor Bugbot for commit b5748c0. Bugbot is set up for automated code reviews on this repo. Configure here. |
…nect Audit finding 4. The store's reorg guard only fires when an already-stored height is rewritten. Resume-after-restart and reconnect-after-stream-death both continue from stored tip + 1, so a reorg past the finalization lag (or a lag set too small) while the archiver was away left the tail on an abandoned fork with canonical roots spliced on top, unnoticed. - New anchor module: re-fetch the block at the stored tip and compare its hash with the one persisted next to the root. Match → resume. Legacy entry without a hash → warn, resume. Mismatch → fail closed with StoreError::AnchorMismatch and the store untouched. - --reanchor-max-depth N (default 0) opts into bounded self-healing: walk back at most N stored blocks to the last canonical one, drop everything above it (new RootStore::truncate_above, count corrected and persisted) and resume from there. No canonical block within N still fails closed, so an RPC serving the wrong chain cannot wipe the archive. - Runs at startup (after the chain-id pin, before backfill) and on every reconnect against the fresh client, selected against SIGTERM. On reconnect an RPC error is a failed attempt and retried; a mismatch exits. Verified against reth --dev: archived 49 blocks, swapped in a fresh dev chain (same chain id, new hashes) past height 49. Default: exit 1 with the AnchorMismatch hint. Depth 3 and 100: 'fork is deeper than allowed', 49 entries still in the store. Same chain restarted: 'stored tip is canonical'.
65e8db6 to
8911551
Compare
f11e78e to
b5748c0
Compare
…ad; end a silent subscription Review (beqaabu): 1. Boundary::Attested released heights straight from the published bound with no regard for the head this subscription had delivered. Against a source node behind the attestors, a height above its head fails its fetch, which StreamRoots treats as a connection error: heap cleared, every in-flight root aborted, reconnect, and the same height fails again next round. Zero progress, including on heights the node could serve. Maturity::mature_height already keeps the invariant for block tags; the attested arm now does too. The bound is folded with the newest head and clamped, and nothing is released until both are known. 2. Under an attested bound a silent subscription was indistinguishable from "nothing new to fetch", so a dead socket hid behind the archiver's "no new attestation yet" log for 13 minutes in the Sepolia run. Both the roots and tip streams now end the head stream when it stays silent for head_silence_timeout (default 120 s) AND the node, asked directly, reports the chain moved on or cannot be reached. A chain that genuinely produced no block (dev chains mining on demand) is left alone, so this does not churn CI's anvil setups. Ending the stream is what makes the outer layer reconnect, so the failure surfaces as a reconnect instead of silence. 3. A comment cited a canonical-anchor check that does not exist on this branch; it arrives with the archiver liveness stack (#1347). Reworded to say so and to make clear that reconciling reverted roots is not the bound poller's job.
1 participant
Fourth PR of the archiver liveness audit (finding 4). Stacked on #1346 → #1345 → #1344; retarget to
usc-devas those merge.Problem
RootStore::put_rootsonly detects a reorg when a height that is already stored is written again with different content. Two paths never touch stored heights: resuming after a restart and reconnecting after a stream death both continue fromstored tip + 1. If the source reorged past the finalization lag while the archiver was away (or the lag is set too small), the tail sits on an abandoned fork and canonical roots get spliced on top of fork roots. Every proof built across that seam is wrong and nothing notices.Changes
archiver/src/anchor.rs(new):reconcile(store, max_depth, canonical_hash)re-fetches the block at the stored tip and compares hashes.Verified, resume fromtip + 1Unverifiable, warn and resumeStoreError::AnchorMismatch(store untouched) and an operator hint--reanchor-max-depth N(default0): opt-in bounded self-healing. Walk back at mostNstored blocks to the last canonical one,truncate_aboveit and resume. No canonical block withinNstill fails closed, so an RPC serving the wrong chain cannot wipe the archive. A legacy hash-less entry below a mismatching tip also stops the walk (fork point cannot be located).RootStore::truncate_above(height): deletes the tail, corrects and persists the entry counter.--end-heightearly-exit check moved after the anchor step so a truncation cannot be skipped by it.Verification
cargo test -p archiver: 35 tests (8 new anchor tests: empty, verified-without-extra-fetches, fail-closed default, reanchor-within-depth truncates exactly the fork tail, deeper-than-depth fails without truncating, legacy tip, legacy-below-mismatch stops the walk, RPC error propagates without truncating; 1 new store test fortruncate_aboveincl. count persistence across reopen). clippy-D warnings, fmt.reth --dev: archived 49 blocks, then swapped in a fresh dev chain (same chain id 1337, different hashes) and let it pass height 49.refusing to resume on top of a fork; re-run with --reanchor-max-depth N …/stored tip 49 is not canonical: stored block hash 0x44fe…, canonical block hash 0x8e10…--reanchor-max-depth 3and100: exit 1,no canonical block found within --reanchor-max-depth …; the fork is deeper than allowed, store still holds 49 entriesstored tip is canonical tip=49, resumes normallyFailed to get block 49), which is the conservative outcome for a lagging replica.