feat(archiver): follow the latest attested height instead of resolving maturity - #1366
DylanVerstraete wants to merge 7 commits into
Conversation
PR SummaryMedium Risk Overview
The archiver adjusts stall detection, flush-at-tip, and ETA to the attested bound and fresh HTTP head reads. CI waits for the archiver to reach the last on-chain checkpoint before builder CSV comparison. Reviewed by Cursor Bugbot for commit a8327a8. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
e72ad1b to
5c2d38a
Compare
…g maturity Phase 2a of making the attested set the single maturity boundary. The archiver is a cache of roots for the prover. Until now it decided for itself which heights were mature, by resolving the chain's on-chain MaturityStrategy (or a FINALIZATION_LAG override) against its own RPC, and walked to that. That is a second opinion on maturity formed against a different node than the attestors use, and the two can disagree in both directions: an archiver ahead of the attestors holds roots for blocks that can still reorg, which is what the canonical-anchor failure on restart reports; one behind them starves the prover. Following the tip with CHAIN_KEY set, the archiver now follows the latest attested height for its chain key, read from Creditcoin (LastDigest, falling back to the last checkpoint) every ATTESTED_POLL_SECS (default 6). It is a high-water mark for a cache, so polling is enough and the archiver gains no second streaming dependency. It no longer parses the maturity strategy in this mode, and a FINALIZATION_LAG is warned about and ignored rather than obeyed: running ahead of the attestors is exactly the state this removes. Explicit ranges (END_HEIGHT), gap backfill and tip-following without a chain key keep resolving maturity against the source node. They walk history that may have no attestations yet (the BSC sweep computes roots the attestors only reach later) and must not become attestation-gated. stream_eth: roots::Config takes a Boundary instead of an eth::Maturity. Boundary::Source(maturity) is the existing per-head resolution and is what the attestor runs; Boundary::Attested(watch) follows an externally published bound, acting on each source head and on each change of the bound, so an attestation landing between heads releases its range at once. The range walk is factored into heights_to_fetch and tested on its own. The field is named bound rather than boundary because the builder derive names typestate parameters after fields and a Boundary parameter would shadow the enum. Under an attested bound a quiet stream is the normal state between attestations, so the archiver's stall detector only reconnects when the bound has moved past what it holds and the blocks still did not arrive. Progress logging reports distance to the bound rather than to the source head.
…ght poll fails Bugbot: the poller retried fetch_last_finalized on the same CcClient clone after an error and never reconnected. CcClient is a value clone whose dead socket never heals on its own, so a dropped Creditcoin WebSocket froze the bound until the process restarted. On a read error the poller now calls reconnect(), which swaps the shared connection atomically under its own backoff, so an outage costs one dial per poll.
…ain checkpoint The archiver follows the latest attested height now, so it starts fetching only once the first attestation lands and trails the attestors while they catch up on the 10k-block anvil state. The test read the archiver's range while it was still ~200 blocks behind the checkpoints the attestors had produced, and then compared builder output against checkpoints the archiver had not stored yet. The archiver was fully synced five minutes later in the same run. Being ahead of the attestors was never a property the test meant to assert; convergence is. Wait, bounded at 15 minutes, for the archiver to reach the last on-chain checkpoint before reading its range.
Bugbot: a successful read of a lower height overwrote the published bound. Stall detection and flush-at-tip both read that value, so a revert or a lagging Creditcoin node could hide a dead stream as "no new attestation yet" and shrink the flush target after roots for the higher range were already released. The bound now only advances; a lower reading is logged and ignored, and reverted roots stay the canonical-anchor check's job on the next start. advance_bound is factored out and tested.
5c2d38a to
5e0efe0
Compare
Overview
Labels (1 changes)
-org.opencontainers.image.created=2026-06-27T04:19:04.617438+00:00
+org.opencontainers.image.created=2026-08-17T09:02:45.677319+00:00
org.opencontainers.image.description=The Ubuntu container image maintained by Canonical
Ubuntu is a Debian-based Linux operating system that runs from the desktop to the cloud, to all your internet connected things.
It is the world's most popular operating system across public clouds and OpenStack clouds.
It is the number one platform for containers; from Docker to Kubernetes to LXD, Ubuntu can run your containers at scale.
Fast, secure and simple, Ubuntu powers millions of PCs worldwide.
org.opencontainers.image.title=ubuntu
org.opencontainers.image.version=26.04Policies (2 improved, 0 worsened)
Packages and Vulnerabilities (47 package changes and 15 vulnerability changes)
Changes for packages of type
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | base-files | 14ubuntu6.1 |
14ubuntu6.2 |
| ♾️ | bsdutils | 1:2.41.3-3ubuntu2 |
1:2.41.3-3ubuntu2.2 |
| ♾️ | curl | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | diffutils | 1:3.12-1 |
1:3.12-1ubuntu0.1 |
| ♾️ | gnu-coreutils | 9.7-3ubuntu2 |
9.7-3ubuntu2.1 |
| ♾️ | gpgv | 2.4.8-4ubuntu3 |
2.4.8-4ubuntu3.1 |
| ♾️ | libattr1 | 1:2.5.2-4 |
1:2.5.2-4ubuntu0.1 |
| ♾️ | libaudit-common | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libaudit1 | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libblkid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libbz2-1.0 | 1.0.8-6build2 |
1.0.8-6ubuntu0.1 |
| ♾️ | libc-bin | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc-gconv-modules-extra | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc6 | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libcurl4t64 | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | libgcrypt20 | 1.12.0-2ubuntu1 |
1.12.0-2ubuntu1.1 |
| ♾️ | libmount1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libpam-modules | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-modules-bin | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-runtime | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam0g | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpq5 | 18.4-0ubuntu0.26.04.1 |
18.6-0ubuntu0.26.04.1 |
| ♾️ | libsmartcols1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libssh2-1t64 | 1.11.1-1ubuntu0.26.04.3 |
1.11.1-1ubuntu0.26.04.4 |
| ♾️ | libssl3t64 | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | libsystemd0 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libudev1 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libuuid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | login | 1:4.16.0-2+really2.41.3-3ubuntu2 |
1:4.16.0-2+really2.41.3-3ubuntu2.2 |
| ♾️ | mount | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | openssl | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | openssl-provider-legacy | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | perl-base | 5.40.1-7ubuntu0.1 |
5.40.1-7ubuntu0.3 |
| ♾️ | util-linux | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | zlib1g | 1:1.3.dfsg+really1.3.1-1ubuntu3 |
1:1.3.dfsg+really1.3.1-1ubuntu3.1 |
Changes for packages of type golang (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | github.com/canonical/pebble | 1.31.1-0.20260528050051-33f10658d3fd |
1.32.2-0.20260721212935-faa1696b477d |
| ♾️ | github.com/gorilla/websocket | 1.5.1 |
1.5.3 |
    |
|||
| Removed vulnerabilities (1): |
|||
| ➖ | golang.org/x/net | 0.40.0 |
|
   |
|||
| Removed vulnerabilities (10): |
|||
| ♾️ | golang.org/x/sys | 0.33.0 |
0.46.0 |
  |
|||
| Removed vulnerabilities (1): |
|||
| ♾️ | golang.org/x/term | 0.32.0 |
0.44.0 |
| ♾️ | stdlib | 1.26.3 |
1.26.5 |
   |
      ?labelColor=lightgrey){kind=icon} |
||
| Removed vulnerabilities (5): |
|||
Changes for packages of type npm (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | @types/node | 26.1.2 |
22.7.5 |
| ♾️ | node-gyp | 13.0.1 |
13.0.2 |
| ♾️ | picomatch | 4.0.5 |
4.0.7 |
| ♾️ | undici | 8.10.0 |
8.10.2 |
| ♾️ | undici-types | 8.3.0 |
6.21.0 |
| ♾️ | ws | 8.21.2 |
8.21.3 |
| // too, so an attestation landing between heads is acted on without waiting. | ||
| let on_heads = { | ||
| let rx = rx.clone(); | ||
| heads.map(move |_| *rx.borrow()) |
There was a problem hiding this comment.
The attested bound isn't clamped to the observed source head. common/streams/eth/src/roots.rs:358 does heads.map(move |_| *rx.borrow()) — the head is fetched and then thrown away. Right next door, common/eth/src/maturity.rs:82 does Ok(Some(tagged.min(head))) with a doc comment spelling out exactly why: "the streams must never run ahead of the heads they have seen." The Attested path drops that invariant. Against a lagging source node, released heights above its head fail get_block, and StreamRoots treats that as a connection error — clearing the reorder heap and abort_all()-ing every in-flight root. So one above-head height tears down the whole batch each round and the archiver makes zero progress, including on heights it could serve. Structurally impossible under Boundary::Source
There was a problem hiding this comment.
Right, and thanks for tracing it through to the abort_all(): one above-head height would have torn down every in-flight root each round. Fixed in 1cab1b2. The attested arm now folds the bound with the newest head this subscription delivered and clamps, and nothing is released until both are known, so it keeps the same invariant mature_height keeps for tags. Pure helper clamp_to_head with a test covering the lagging-node case.
| // A lower reading is a revert or a lagging Creditcoin node. Neither | ||
| // may shrink the bound: roots for the higher range are already | ||
| // released, and stall detection and flush-at-tip both read this | ||
| // value. Reverted roots are reconciled by the canonical-anchor | ||
| // check on the next start, not by moving the bound. |
There was a problem hiding this comment.
the comment at archiver/src/main.rs:602 justifies never shrinking the bound by citing a "canonical-anchor check on the next start" — no such check exists; grep -i anchor archiver/ only hits find_gaps' unrelated low-side arg. The decision not to shrink is fine, the stated safety net is fictional. And a dead source WS subscription is no longer self-detected under an attested boundary (select never terminates), so the "no new attestation yet" log is misleading during an outage.
There was a problem hiding this comment.
Both points valid. The comment cited a check that lives on the archiver liveness stack (#1347), not on this branch; reworded in 1cab1b2 to say that reconciling reverted roots is not this poller's job and where that check actually arrives. On the second point, that is exactly what I watched happen in the Sepolia run: a dead socket hid behind "no new attestation yet" for 13 minutes. Same commit adds a silence watchdog to both the roots and tip streams: when newHeads stays quiet for head_silence_timeout (120 s default) the node is asked for its head directly; if the chain moved on or the node is unreachable the stream ends and the outer layer reconnects, if the chain genuinely produced nothing (on-demand dev chains) it keeps waiting. So the log is now truthful: a quiet stream that survives it really has nothing to fetch.
…ad; end a silent subscription Review (beqaabu): 1. Boundary::Attested released heights straight from the published bound with no regard for the head this subscription had delivered. Against a source node behind the attestors, a height above its head fails its fetch, which StreamRoots treats as a connection error: heap cleared, every in-flight root aborted, reconnect, and the same height fails again next round. Zero progress, including on heights the node could serve. Maturity::mature_height already keeps the invariant for block tags; the attested arm now does too. The bound is folded with the newest head and clamped, and nothing is released until both are known. 2. Under an attested bound a silent subscription was indistinguishable from "nothing new to fetch", so a dead socket hid behind the archiver's "no new attestation yet" log for 13 minutes in the Sepolia run. Both the roots and tip streams now end the head stream when it stays silent for head_silence_timeout (default 120 s) AND the node, asked directly, reports the chain moved on or cannot be reached. A chain that genuinely produced no block (dev chains mining on demand) is left alone, so this does not churn CI's anvil setups. Ending the stream is what makes the outer layer reconnect, so the failure surfaces as a reconnect instead of silence. 3. A comment cited a canonical-anchor check that does not exist on this branch; it arrives with the archiver liveness stack (#1347). Reworded to say so and to make clear that reconciling reverted roots is not the bound poller's job.
… stalls on the clamped bound Bugbot, two findings on the previous commit: 1. The roots stream consumed the first head to seed the pipeline before wrapping the subscription in end_on_silence, so the watchdog started without a baseline and judged every probe as a quiet chain. A subscription that died right after that first head was never caught while eth_blockNumber still answered, which is the failure the watchdog exists for. The first head is now handed in as the baseline, and a stream that is silent from the very start adopts its first probe as the baseline so the second silent round catches a chain that moved. 2. The archiver's stall check and flush-at-tip read the raw published bound, while the stream fetches only up to that bound clamped to the observed source head. With a source node lagging the attestors and the archiver caught up to that node, a healthy quiet stream looked like a stall and the WS client was torn down every STREAM_TIMEOUT_SECS. Both now use the clamped value and the log names both numbers.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8930f04. Configure here.
…ment Bugbot: the stall check and flush-at-tip capped the attested bound with the HTTP head tracker, which is 0 until its first successful read and is never repaired on later failures. min(bound, 0) put the fetchable bound below the next wanted height, so a hung fetch read as "nothing new to fetch" and the stream was never rebuilt; a lagging rpc_http node did the same. The tracker now records when it last succeeded, and a head is trusted only if read at least once and refreshed within three polls (36 s). Otherwise it does not clamp and the stream is judged against the raw bound, which errs towards a reconnect, never towards hiding a stall. Pure helpers, tested.
3 participants
Phase 2a of the plan to make the attested set the single maturity boundary: https://claude.ai/code/artifact/400f8668-7b43-4b11-86c8-b8922d8c85ee
What was there
The archiver decided for itself which heights were mature: it resolved the chain's on-chain
MaturityStrategy(or aFINALIZATION_LAGoverride) against its own RPC and walked to that. That is a second opinion on maturity formed against a different node than the attestors use, and the two disagree in both directions. Ahead of the attestors, it holds roots for blocks that can still reorg, which is what the canonical-anchor failure on restart reports. Behind them, it starves the prover.What is there now
Following the tip with
CHAIN_KEYset, the archiver follows the latest attested height for its chain key, read from Creditcoin everyATTESTED_POLL_SECS(default 6, new flag). It is a high-water mark for a cache, so polling is enough and the archiver gains no second streaming dependency. In this mode it no longer parses the maturity strategy at all.Unchanged on purpose: explicit ranges (
END_HEIGHT), gap backfill, and tip-following without a chain key still resolve maturity against the source node. They walk history that may have no attestations yet (the BSC sweep computes roots the attestors only reach later) and must not become attestation-gated. That resolution is now lazy, so a tip follower never touches it.FINALIZATION_LAGIn attested mode it is warned about and ignored, not obeyed. Obeying it would mean running ahead of the attestors, which is exactly the state this removes. It still applies to the three source-resolved paths above. The
creditcoin-archiverchart'sfinalizationLagFromChaindefault already omits the variable, so no chart change is needed for the default case.stream_ethchange (touches the attestor, mechanically)roots::Configtakes aBoundaryinstead of aneth::Maturity:Boundary::Source(maturity)is the existing per-head resolution, and is what the attestor runs, because attestors are the maturity decision.production.rschanges from.with_maturity(m)to.with_bound(Boundary::Source(m)), nothing else.Boundary::Attested(watch)follows an externally published bound. It acts on each source head and on each change of the bound, so an attestation landing between heads releases its range immediately rather than waiting for the next head.The range walk is factored into
heights_to_fetchand tested on its own. The field is calledboundrather thanboundarybecause the builder derive names typestate parameters after their fields, and aBoundaryparameter shadowed the enum inside the generated setter. That cost me one confusing compile error and is worth a comment for the next person.Stall detection
Under an attested bound a quiet stream is the normal state between attestations. The archiver's timeout now only reconnects when the bound has moved past what it holds and the blocks still did not arrive; otherwise it logs "no new attestation yet" and keeps waiting. Progress logging reports distance to the bound, which is the meaningful lag for a cache, rather than distance to the source head.
Not in this PR
/readyand the readiness lag arithmetic live in #1346 on the archiver liveness stack, not on this branch. When the stacks meet, that lag becomes distance from the attested bound, which is knowable under both a tag and an attested policy. Theusc-dev-wipmerge already carries theMaturity-typed interim version of that reconciliation.Tests
stream_eth: the range walk releases each height once and ignoresNoneand non-advancing bounds; nothing below start; an attested bound drives the walk on its own through a live watch channel, holds everything back whileNone, ignores a backwards move, and ends when the sender is gone.archiver: mode selection for all four(CHAIN_KEY, END_HEIGHT)combinations. Existingresolve_maturitytests unchanged. Clippy-D warningsacross stream_eth, attestor, stream_attestation, archiver, eth including benches and examples; fmt clean.Since opened: rebased onto
usc-dev, which already carries #1335 (flush every block near the tip). The one line this PR keeps from that reconciliation is that "near the tip" is measured against the attested bound rather than the source head, folded into the first commit. Found live: without it the bound block itself sat in the batch buffer and the prover answered 503 with 119 of 120 roots. Also: the CI attestor-network job now waits for the archiver to reach the last on-chain checkpoint, since the archiver now trails the attestors during catch-up rather than racing ahead of them.