feat(eth): resolve block tags by cross-provider agreement, keyed on the block hash - #1367
DylanVerstraete wants to merge 3 commits into
Conversation
PR SummaryHigh Risk Overview
The archiver tracks when the HTTP chain head was last seen ( Continuity drops Tests in Reviewed by Cursor Bugbot for commit c8fb095. Bugbot is set up for automated code reviews on this repo. Configure here. |
eae8a5d to
fc7bed9
Compare
fc7bed9 to
22c84c2
Compare
22c84c2 to
37971a6
Compare
|
pre-commit.ci run |
Overview
Labels (1 changes)
-org.opencontainers.image.created=2026-06-27T04:19:04.617438+00:00
+org.opencontainers.image.created=2026-08-17T09:02:45.677319+00:00
org.opencontainers.image.description=The Ubuntu container image maintained by Canonical
Ubuntu is a Debian-based Linux operating system that runs from the desktop to the cloud, to all your internet connected things.
It is the world's most popular operating system across public clouds and OpenStack clouds.
It is the number one platform for containers; from Docker to Kubernetes to LXD, Ubuntu can run your containers at scale.
Fast, secure and simple, Ubuntu powers millions of PCs worldwide.
org.opencontainers.image.title=ubuntu
org.opencontainers.image.version=26.04Policies (2 improved, 0 worsened)
Packages and Vulnerabilities (47 package changes and 15 vulnerability changes)
Changes for packages of type
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | base-files | 14ubuntu6.1 |
14ubuntu6.2 |
| ♾️ | bsdutils | 1:2.41.3-3ubuntu2 |
1:2.41.3-3ubuntu2.2 |
| ♾️ | curl | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | diffutils | 1:3.12-1 |
1:3.12-1ubuntu0.1 |
| ♾️ | gnu-coreutils | 9.7-3ubuntu2 |
9.7-3ubuntu2.1 |
| ♾️ | gpgv | 2.4.8-4ubuntu3 |
2.4.8-4ubuntu3.1 |
| ♾️ | libattr1 | 1:2.5.2-4 |
1:2.5.2-4ubuntu0.1 |
| ♾️ | libaudit-common | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libaudit1 | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libblkid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libbz2-1.0 | 1.0.8-6build2 |
1.0.8-6ubuntu0.1 |
| ♾️ | libc-bin | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc-gconv-modules-extra | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc6 | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libcurl4t64 | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | libgcrypt20 | 1.12.0-2ubuntu1 |
1.12.0-2ubuntu1.1 |
| ♾️ | libmount1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libpam-modules | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-modules-bin | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-runtime | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam0g | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpq5 | 18.4-0ubuntu0.26.04.1 |
18.6-0ubuntu0.26.04.1 |
| ♾️ | libsmartcols1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libssh2-1t64 | 1.11.1-1ubuntu0.26.04.3 |
1.11.1-1ubuntu0.26.04.4 |
| ♾️ | libssl3t64 | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | libsystemd0 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libudev1 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libuuid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | login | 1:4.16.0-2+really2.41.3-3ubuntu2 |
1:4.16.0-2+really2.41.3-3ubuntu2.2 |
| ♾️ | mount | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | openssl | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | openssl-provider-legacy | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | perl-base | 5.40.1-7ubuntu0.1 |
5.40.1-7ubuntu0.3 |
| ♾️ | util-linux | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | zlib1g | 1:1.3.dfsg+really1.3.1-1ubuntu3 |
1:1.3.dfsg+really1.3.1-1ubuntu3.1 |
Changes for packages of type golang (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | github.com/canonical/pebble | 1.31.1-0.20260528050051-33f10658d3fd |
1.32.2-0.20260721212935-faa1696b477d |
| ♾️ | github.com/gorilla/websocket | 1.5.1 |
1.5.3 |
    |
|||
| Removed vulnerabilities (1): |
|||
| ➖ | golang.org/x/net | 0.40.0 |
|
   |
|||
| Removed vulnerabilities (10): |
|||
| ♾️ | golang.org/x/sys | 0.33.0 |
0.46.0 |
  |
|||
| Removed vulnerabilities (1): |
|||
| ♾️ | golang.org/x/term | 0.32.0 |
0.44.0 |
| ♾️ | stdlib | 1.26.3 |
1.26.5 |
   |
      ?labelColor=lightgrey){kind=icon} |
||
| Removed vulnerabilities (5): |
|||
Changes for packages of type npm (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | @types/node | 26.1.2 |
22.7.5 |
| ♾️ | node-gyp | 13.0.1 |
13.0.2 |
| ♾️ | picomatch | 4.0.5 |
4.0.7 |
| ♾️ | undici | 8.10.0 |
8.10.2 |
| ♾️ | undici-types | 8.3.0 |
6.21.0 |
| ♾️ | ws | 8.21.2 |
8.21.3 |
| for (err_label, err) in errors.drain(..) { | ||
| if let Some((candidate_label, candidate)) = reported | ||
| .iter() | ||
| .min_by_key(|(_, block)| block.number) |
There was a problem hiding this comment.
A stale fallback provider permanently freezes maturity for the whole pipeline. common/eth/src/lib.rs:1044 picks the candidate via min_by_key(block.number) across all responders. A backup RPC that stops syncing but keeps serving answers safe = N forever; its block at N hashes identically to canonical, so the agreement check passes and returns Ok(N). common/streams/eth/src/tip.rs:81 only advances on tip_new > tip, so the tip pins at N indefinitely even though the primary is healthy. Verified both ends. This inverts what a fallback means — before this PR a fallback was only consulted when the primary errored, so a stale backup was harmless; now it's a hard brake. There's no staleness bound, no warning (only debug!), and no metric.
There was a problem hiding this comment.
You're right, and it's the more serious of the two: lowest-height turned every fallback into a brake. Fixed in e87dee4. The primary's answer is the candidate (the freshest fallback's when the primary has none), and the others confirm its identity: same height, compare hashes; further along, fetch the block at the candidate height and compare. A provider that hasn't reached the candidate cannot confirm and cannot veto; it is logged at warn with how far behind it is, so a persistently stale fallback is visible. Hash mismatch at the candidate height stays a hard BlockTagDisagreement. Resolution without full agreement now logs at warn, not debug. Tests: fallback ahead confirms, fallback behind steps aside, stale fallback cannot pin with and without a primary answer.
There was a problem hiding this comment.
No timeout anywhere — one black-holed provider hangs the tip stream forever. get_block_by_tag does join_all over all providers, and grep -E 'timeout|Timeout' common/eth/src/lib.rs returns nothing; init_rpc builds providers with a default reqwest client. A fallback that accepts TCP but never responds makes the lookup never return, and tip.rs awaits it inline in the stream loop — no reconnect, no error, no log. A per-provider tokio::time::timeout treated like the tolerated Err branch at lib.rs:1106 fixes it
There was a problem hiding this comment.
Fixed in e87dee4: every provider call in the lookup goes through a timed wrapper bounded by Client::call_timeout (15 s default, one Ethereum slot; with_call_timeout to override), and a timeout is handled exactly like the tolerated Err branch. Test uses a mock that answers eth_chainId and black-holes everything else; the lookup returns the primary's answer in under the timeout. Worth saying: init_rpc still has no timeout, so a black-holed URL hangs client construction; that is pre-existing and not on the tag path, I'd take it as a separate fix if you want it.
37971a6 to
e87dee4
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The merge-base changed after approval.
e87dee4 to
9009fc6
Compare
The merge-base changed after approval.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9009fc6. Configure here.
…ment Bugbot: the stall check and flush-at-tip capped the attested bound with the HTTP head tracker, which is 0 until its first successful read and is never repaired on later failures. min(bound, 0) put the fetchable bound below the next wanted height, so a hung fetch read as "nothing new to fetch" and the stream was never rebuilt; a lagging rpc_http node did the same. The tracker now records when it last succeeded, and a head is trusted only if read at least once and refreshed within three polls (36 s). Otherwise it does not clamp and the stream is judged against the raw bound, which errs towards a reconnect, never towards hiding a stall. Pure helpers, tested.
…he block hash
Phase 1c, and the second half of 1a, of making the attestor's maturity
decision worth following.
get_block_number_by_tag walked the provider list and returned the first
block any provider served, and it received the full block and returned only
its number. For a safety boundary that is the wrong shape twice over: a
single load-balanced peer on a stale fork could set the boundary for the
whole attestor, and the one datum that pins block identity was thrown away
at the one place it was free.
New get_block_by_tag returns TaggedBlock { number, hash } and asks every
configured provider at once. The lowest reported height is the candidate,
because a provider further along has by definition already passed it. Every
other responder must then have the candidate block under the same hash:
same-height responders already said so, further-along responders are asked
for the block at that height. A different hash is
Error::BlockTagDisagreement, and the lookup refuses to pick a side; the tip
and roots streams already log and retry on the next head. A provider that
reports the tag past a height it cannot serve is broken, not forked, and
fails the lookup outright. One that errors on the confirmation read has
already voted the candidate mature and is warned about, as tag-call errors
were before.
Providers that error or answer null still do not take part, so with one
provider this degenerates to that provider's answer: per-replica provider
diversity in the deployment is what gives the check its teeth, and this is
the code side of that. get_block_number_by_tag stays as a thin wrapper.
The hash stops at the boundary decision on purpose. Fetching the blocks
themselves by hash would serialise get_block and get_block_receipts and
roughly halve archiver throughput; #1362's receipts-to-block hash check
already pins each fetched block's integrity at no cost.
Also removed: EthRpcProvider::get_block_number_by_tag in continuity, dead
since #1365 took the prover's maturity opinion away.
Tests: agreement takes the lowest height and confirms it against the further
provider (both are asked every time); a forked provider is a disagreement at
different heights and at the same height; the disagreement surfaces through
Maturity as a per-head error. The mock chain now salts block hashes so two
mocks can be on one chain or on different forks.
… aside; every call is bounded Review (beqaabu): 1. Taking the lowest reported height made any fallback a brake on the whole pipeline: a backup that stopped syncing keeps answering safe = N forever, its block at N hashes identically to the canonical one, agreement passes, and the tip stream, which only ever advances, pins at N while the primary is healthy. That inverts what a fallback is for. The primary's answer is now the candidate (the freshest fallback's when the primary has none), and the other providers confirm its identity: same height, compare hashes; further along, ask for the block at that height. A provider that has not reached the candidate cannot confirm and cannot veto; it is logged at warn and skipped, as is one that claims the tag past a height it cannot serve. A different hash at the candidate height is still a hard BlockTagDisagreement. Resolution without full agreement is now a warn, not a debug line. 2. No provider call had a timeout, so one black-holed provider hung the lookup and the tip stream awaiting it inline, with no error and no log. Every provider call in the lookup is now bounded by Client::call_timeout (default 15 s, one Ethereum slot; Client::with_call_timeout overrides) and a timeout is handled like any other transport error. Tests: primary leads with a fallback ahead (confirms) and behind (steps aside); a stale fallback cannot pin maturity, with and without a primary answer; a black-holed fallback times out instead of hanging. The mock gained a black-hole mode that answers eth_chainId only.
9009fc6 to
c8fb095
Compare
3 participants
Phase 1c, plus the second half of 1a, of the plan to make the attested set the single maturity boundary: https://claude.ai/code/artifact/400f8668-7b43-4b11-86c8-b8922d8c85ee
What was there
get_block_number_by_tagwalked the provider list and returned the first block any provider served. It also received the full block from the RPC and returned only its number. For the one call that decides the attestor's maturity boundary that is the wrong shape twice: a single load-balanced peer on a stale fork could set the boundary for the whole attestor, and the one datum that pins block identity was discarded at the one place it was free.What is there now
get_block_by_tagreturnsTaggedBlock { number, hash }and asks every configured provider at once.Error::BlockTagDisagreementand the lookup refuses to pick a side. The tip and roots streams already log and retry on the next head, so a transient fork view costs one head, not an attestation.Two edge cases are decided explicitly. A provider that reports the tag past a height it cannot serve is broken rather than forked, and fails the lookup. A provider that errors on the confirmation read has already voted the candidate mature by reporting a higher tag, so it is warned about and skipped, the same treatment a tag-call error got before.
Providers that error or answer
nullstill do not take part. With one provider this degenerates to that provider's answer, so this PR is the code half of a two-part change: per-replica provider diversity in the AttestorSet (1b, cc-networks-iac) is what gives it teeth. Today every replica in a set reads the boundary from oneethUrlSecretRef.get_block_number_by_tagstays as a thin wrapper, soMaturityand the streams are untouched.Why the hash stops here
The plan said "carry the hash through
Maturityand fetch byBlockId::Hash". I stopped short deliberately, as in #1362: fetching by hash serialisesget_blockandget_block_receiptsand roughly halves archiver throughput, and #1362's receipts-to-block hash check already pins each fetched block's integrity for free. The hash is used where it decides something, the boundary identity across providers, and not threaded through code that has no use for it.Also removed
EthRpcProvider::get_block_number_by_tagin the continuity crate, its three impls and its test. Dead since #1365 took the prover's maturity opinion away.Tests
common/eth/tests/block_tag.rsmock now salts block hashes, so two mocks can be on one chain or on different forks. New: agreement takes the lowest height and confirms it against the further provider, and both are asked every time; a forked provider is a disagreement at different heights and at the same height; the disagreement surfaces throughMaturityas a per-head error. Existing fall-through and clamping tests unchanged. Clippy-D warningsacross eth, continuity, proof-gen, archiver, attestor, stream_eth; fmt; 189 tests green across those crates.