fix(archiver): pin chain identity, handle SIGTERM everywhere, decouple the tip window - #1344
Open
DylanVerstraete wants to merge 3 commits into
Open
DylanVerstraete wants to merge 3 commits into
DylanVerstraete wants to merge 3 commits into
Conversation
…e the tip window Liveness audit follow-ups (findings 3, 5 and 7). Chain identity (finding 3) - The archive pins its source chain_id in the sled meta tree on first run and refuses to start if the RPC now serves another chain (`StoreError::ChainIdMismatch`). - The archiver's reconnect loop rejects a replacement WS client whose chain_id differs from the pinned one and keeps retrying instead of archiving foreign roots. - `eth::Client::reconnect` fails closed (`Error::ChainIdChanged`) when the endpoint reports a different chain_id than the one it was created with, keeping the previous connection; every caller already treats reconnect errors as "retry later". Shutdown (finding 5) - SIGTERM (what systemd / Kubernetes send) is handled like Ctrl+C. - The shutdown signal is a `watch` selected on in the main loop, the reconnect backoff, WS reconnect, stream construction and the backfill poll loop, so a SIGTERM during an outage or a stuck stream constructor exits through the final-batch / final-flush path instead of needing a kill. Flush policy (finding 7) - New `--tip-window` (`TIP_WINDOW`, default 256): roots are written per block only within this many blocks of the target, instead of within `FLUSH_EVERY` (10,000) blocks, which made the last 10k blocks of every catch-up run with an fsync per block (~9x slower in the audit's smoke test). - Durability flushes at the tip are throttled to one per second; sled's own 500 ms timer flushes anyway and the write already makes the root visible. Verified locally against reth --dev and anvil: SIGTERM exits cleanly in <10 s with the final flush logged; a pinned archive started against another chain exits 1 with the mismatch error; 20 archiver unit tests incl. pin/mismatch and flush-throttle cases.
PR SummaryMedium Risk Overview Chain identity is pinned in sled metadata on first successful startup (after WS/HTTP and optional Creditcoin checks). Later runs, backfill dials, and WS reconnects must match that Graceful shutdown treats SIGTERM like Ctrl+C via a reusable Tip vs catch-up I/O: new Reviewed by Cursor Bugbot for commit 943c929. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 662af5b. Configure here.
…fy chain id on backfill dials Bugbot follow-ups on #1344: - `pin_chain_id` now runs after the ws/http agreement and the Creditcoin registration check, so a first start against the wrong RPC exits on the registry mismatch without recording that chain id into an archive that never stored a root. - The backfill path dials a fresh WS client; it now refuses one whose chain_id differs from the pinned id. Gaps have no reorg-guard second line of defence, so this was the one path that could write foreign roots.
…constructors StreamRoots::new retries the initial subscribe without bound while the source is down, and the backfill path dials a fresh client per gap. Neither was selected against the shutdown watch, so a SIGTERM during a source outage hung until SIGKILL and skipped the final flush. Both now yield to cancellation the same way the reconnect path already does. Verified against a fake RPC that rejects eth_subscribe: the archiver spun in the retry loop and exited within the same second of SIGTERM with 'shutdown requested before the root stream connected'.
This was referenced Sep 11, 2026
DylanVerstraete
requested review from
a team,
BradleyOlson64,
beqaabu,
creditcoinprotoclaw,
didac-gluwa,
jakerumbles and
mdbig1
and removed request for
a team
September 11, 2026 10:46
Contributor
Overview
Labels (1 changes)
-org.opencontainers.image.created=2026-06-27T04:19:04.617438+00:00
+org.opencontainers.image.created=2026-08-17T09:02:45.677319+00:00
org.opencontainers.image.description=The Ubuntu container image maintained by Canonical
Ubuntu is a Debian-based Linux operating system that runs from the desktop to the cloud, to all your internet connected things.
It is the world's most popular operating system across public clouds and OpenStack clouds.
It is the number one platform for containers; from Docker to Kubernetes to LXD, Ubuntu can run your containers at scale.
Fast, secure and simple, Ubuntu powers millions of PCs worldwide.
org.opencontainers.image.title=ubuntu
org.opencontainers.image.version=26.04Policies (2 improved, 0 worsened)
Packages and Vulnerabilities (47 package changes and 15 vulnerability changes)
Changes for packages of type
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | base-files | 14ubuntu6.1 |
14ubuntu6.2 |
| ♾️ | bsdutils | 1:2.41.3-3ubuntu2 |
1:2.41.3-3ubuntu2.2 |
| ♾️ | curl | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | diffutils | 1:3.12-1 |
1:3.12-1ubuntu0.1 |
| ♾️ | gnu-coreutils | 9.7-3ubuntu2 |
9.7-3ubuntu2.1 |
| ♾️ | gpgv | 2.4.8-4ubuntu3 |
2.4.8-4ubuntu3.1 |
| ♾️ | libattr1 | 1:2.5.2-4 |
1:2.5.2-4ubuntu0.1 |
| ♾️ | libaudit-common | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libaudit1 | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libblkid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libbz2-1.0 | 1.0.8-6build2 |
1.0.8-6ubuntu0.1 |
| ♾️ | libc-bin | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc-gconv-modules-extra | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc6 | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libcurl4t64 | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | libgcrypt20 | 1.12.0-2ubuntu1 |
1.12.0-2ubuntu1.1 |
| ♾️ | libmount1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libpam-modules | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-modules-bin | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-runtime | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam0g | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpq5 | 18.4-0ubuntu0.26.04.1 |
18.6-0ubuntu0.26.04.1 |
| ♾️ | libsmartcols1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libssh2-1t64 | 1.11.1-1ubuntu0.26.04.3 |
1.11.1-1ubuntu0.26.04.4 |
| ♾️ | libssl3t64 | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | libsystemd0 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libudev1 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libuuid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | login | 1:4.16.0-2+really2.41.3-3ubuntu2 |
1:4.16.0-2+really2.41.3-3ubuntu2.2 |
| ♾️ | mount | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | openssl | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | openssl-provider-legacy | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | perl-base | 5.40.1-7ubuntu0.1 |
5.40.1-7ubuntu0.3 |
| ♾️ | util-linux | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | zlib1g | 1:1.3.dfsg+really1.3.1-1ubuntu3 |
1:1.3.dfsg+really1.3.1-1ubuntu3.1 |
Changes for packages of type golang (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | github.com/canonical/pebble | 1.31.1-0.20260528050051-33f10658d3fd |
1.32.2-0.20260721212935-faa1696b477d |
| ♾️ | github.com/gorilla/websocket | 1.5.1 |
1.5.3 |
    |
|||
| Removed vulnerabilities (1): |
|||
| ➖ | golang.org/x/net | 0.40.0 |
|
   |
|||
| Removed vulnerabilities (10): |
|||
| ♾️ | golang.org/x/sys | 0.33.0 |
0.46.0 |
  |
|||
| Removed vulnerabilities (1): |
|||
| ♾️ | golang.org/x/term | 0.32.0 |
0.44.0 |
| ♾️ | stdlib | 1.26.3 |
1.26.5 |
   |
      ?labelColor=lightgrey){kind=icon} |
||
| Removed vulnerabilities (5): |
|||
Changes for packages of type npm (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | @types/node | 26.1.2 |
22.7.5 |
| ♾️ | node-gyp | 13.0.1 |
13.0.2 |
| ♾️ | picomatch | 4.0.5 |
4.0.7 |
| ♾️ | undici | 8.10.0 |
8.10.2 |
| ♾️ | undici-types | 8.3.0 |
6.21.0 |
| ♾️ | ws | 8.21.2 |
8.21.3 |
beqaabu
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups to the archiver liveness/performance audit of 2026-09-11 (findings 3, 5, 7). Findings 1 (watchdog over stream construction) and 2 (health/freshness) follow in separate PRs.
Finding 3 – chain identity pinned across restarts and reconnects
RootStore::pin_chain_id: the sled meta tree records the sourcechain_idon first run; every later start must match or the archiver exits witharchive was built from chain_id X but the source RPC now serves chain_id Y; refusing to write foreign roots.chain_idis refused (logged at error) and retried, never archived.eth::Client::reconnectnow fails closed withError::ChainIdChangedinstead of adopting the new chain id; the previous connection is kept and callers already treat reconnect errors as retryable (continuity, tip stream, attestor).Finding 5 – SIGTERM and shutdown during reconnect
watchand is selected on in the main loop, the reconnect backoff sleep, the WS reconnect, theStreamRootsconstructor wait, and the backfill poll loop. The audit's "SIGINT while replacement constructor is waiting → required a kill" case now exits through the final-batch/final-flush path.Finding 7 – flush window decoupled from
FLUSH_EVERY--tip-window/TIP_WINDOW(default 256 blocks): per-block writes only within this many blocks of the target. Previously (feat(archiver): write and flush every block once within a flush window of the head #1335) the window wasFLUSH_EVERY= 10,000, so the last 10k blocks of any catch-up ran with a write + fsync per block (the audit measured ~9× slower over a 2,000-root catch-up).TIP_FLUSH_INTERVAL(1 s). The write already makes the root visible to/roots; sled's own 500 ms flusher runs regardless.FLUSH_EVERYkeeps its meaning as the catch-up batch size. Docs/README updated.Verification
cargo test -p archiver: 20 passed (new: chain-id pin/mismatch, flush throttle, tip window).cargo clippy -p archiver -p eth -p continuity -p stream_eth -p attestor -p proof-gen-api-server --all-targets -- -D warnings,cargo fmt --check: clean.reth --dev(chain 1337) + anvil (31337): SIGTERM while following →shutting down… / flushing final state… / archiver stoppedwithin 8 s; tip flush lines at ~1 s spacing; restarting the same sled against anvil → exit 1 with the mismatch error.