Skip to content

fix(stream_eth): seed the block stream from eth_blockNumber and poll heads alongside newHeads - #1345

Open
DylanVerstraete wants to merge 2 commits into
fix/archiver-liveness-afrom
fix/archiver-liveness-b
Open

DylanVerstraete wants to merge 2 commits into
fix/archiver-liveness-afrom
fix/archiver-liveness-b

Conversation

@DylanVerstraete

Copy link
Copy Markdown
Contributor

Stacked on #1344 (base is fix/archiver-liveness-a; retarget to usc-dev once that merges). Liveness audit finding 1.

Problem

stream_rpc waited for the first newHeads frame with no deadline before any historical work could start, and the archiver awaited that constructor at startup, in backfill and inside its reconnect loop. A subscription that acknowledges but never delivers (seen through proxies/load balancers, or simply an idle chain) wedged the process indefinitely while /status kept returning 200. The audit reproduced this: "suppress notifications on replacement sockets → stalled at height 24 with no further connection attempts".

Fix

  • Seed from eth_blockNumber: catch-up begins immediately and never depends on a future header. Each subscribe/head attempt is bounded by rpc_call_timeout (default 30 s; alloy transports have no default timeout).
  • Merged head sources: subscription head numbers are merged with a periodic eth_blockNumber poll (head_poll_interval, default 12 s). Observations that don't advance the frontier contribute nothing, so the two sources overlap freely. The merged stream still ends when the subscription ends, so a dead socket surfaces exactly as before and triggers reconnection.
  • expand_heads / merge_heads extracted as pure functions with unit tests (the first tests in stream_eth).
  • Archiver: --head-poll-interval-secs / HEAD_POLL_INTERVAL_SECS and --rpc-timeout-secs / RPC_TIMEOUT_SECS, plumbed into all three stream constructions. The attestor uses the same StreamRoots and picks up the defaults unchanged.

Verification

  • cargo test -p stream_eth -p archiver: 4 new + 20 passing; clippy -D warnings on stream_eth/archiver/attestor; fmt clean.
  • Reproduced the audit scenario with a frame-dropping WS proxy in front of reth --dev (1 s blocks, poll 3 s):
Scenario Result
Start with eth_subscription frames suppressed API up at t+12 s, archived 14 (head 16); t+18 s archived 21 → progress while silent
Restore notifications t+22 s archived 27 = head
Suppress again mid-run t+30 s archived 33 (head 34) → progress while silent
Watchdog reconnects during the run 0
SIGTERM graceful stop

Previously the first scenario left the process alive with no API listener and no progress (audit table, last row).

@cursor

cursor Bot commented Sep 11, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes core ETH head discovery and stream startup on the archiving hot path; behavior is safer when RPC/WS is flaky but mis-tuned timeouts or poll intervals could affect catch-up or reconnect timing.

Overview
Fixes archiver/stream liveness when newHeads acknowledges but stops delivering (or the chain is idle): the ETH root stream no longer blocks on the first subscription frame with no deadline.

stream_eth (stream_rpc) now seeds catch-up from eth_blockNumber right after subscribe, with each subscribe/head attempt bounded by rpc_call_timeout (default 30s). Tip tracking merges newHeads with periodic eth_blockNumber polls (head_poll_interval, default 12s) via new merge_heads / expand_heads helpers; the merged stream still ends when the subscription ends so dead sockets trigger reconnect as before. Poll ticks use Delay missed-tick behavior to avoid RPC floods after long catch-up. Four unit tests cover the new helpers.

Archiver adds --head-poll-interval-secs / --rpc-timeout-secs (and env vars), wired into gap backfill, initial stream, and WS reconnect paths. README documents the flags. Attestor keeps library defaults unchanged.

Reviewed by Cursor Bugbot for commit e5ce953. Bugbot is set up for automated code reviews on this repo. Configure here.

…heads alongside newHeads

Liveness audit finding 1: `stream_rpc` waited for the first `newHeads` frame
with no deadline before any historical work could begin, and the archiver
awaited that constructor at startup, in backfill and inside its reconnect
loop. A subscription that acknowledges but never delivers (proxies, load
balancers, an idle chain) wedged the process while `/status` kept answering.

- The starting head now comes from `eth_blockNumber`; catch-up begins at once
  and never depends on a future header. Each subscribe/head attempt is bounded
  by `rpc_call_timeout` (default 30 s; alloy transports have none).
- Head observations from the subscription are merged with a periodic
  `eth_blockNumber` poll (`head_poll_interval`, default 12 s). Observations
  that do not advance the frontier contribute nothing, so the two sources
  overlap freely; the merged stream still ends when the subscription ends, so
  a dead socket surfaces as before and triggers reconnection.
- `expand_heads` / `merge_heads` are pure and unit-tested (first tests in
  stream_eth).
- Archiver: `--head-poll-interval-secs` / `--rpc-timeout-secs` plumbed into
  every stream construction. The attestor picks up the defaults unchanged.

Reproduced the audit scenario with a frame-dropping WS proxy against
`reth --dev` (poll 3 s): started with `eth_subscription` frames suppressed the
archiver came up, served `/status`, and advanced 14 -> 21 -> 27 through polls;
silenced again mid-run it advanced 27 -> 33 (head 34); zero watchdog
reconnects; SIGTERM still exited cleanly.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 12370dd. Configure here.

Comment thread common/streams/eth/src/roots.rs
The poll stream is not driven while expand_heads drains the seeded range;
with tokio's default Burst the ticks missed during a long catch-up fired
back-to-back as a flood of eth_blockNumber calls on the block-fetch socket.
@DylanVerstraete
DylanVerstraete requested review from a team, BradleyOlson64, beqaabu, creditcoinprotoclaw, didac-gluwa, jakerumbles and mdbig1 and removed request for a team September 11, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant