feat(eth): raw-RLP block fetch mode (debug_getRawBlock/Receipts) + archiver --fetch-mode - #1328
Draft
DylanVerstraete wants to merge 4 commits into
Draft
DylanVerstraete wants to merge 4 commits into
DylanVerstraete wants to merge 4 commits into
Conversation
PR SummaryMedium Risk Overview The archiver exposes Reviewed by Cursor Bugbot for commit 70c9b97. Bugbot is set up for automated code reviews on this repo. Configure here. |
Contributor
Overview
Labels (1 changes)
-org.opencontainers.image.created=2026-06-27T04:19:04.617438+00:00
+org.opencontainers.image.created=2026-08-17T09:02:45.677319+00:00
org.opencontainers.image.description=The Ubuntu container image maintained by Canonical
Ubuntu is a Debian-based Linux operating system that runs from the desktop to the cloud, to all your internet connected things.
It is the world's most popular operating system across public clouds and OpenStack clouds.
It is the number one platform for containers; from Docker to Kubernetes to LXD, Ubuntu can run your containers at scale.
Fast, secure and simple, Ubuntu powers millions of PCs worldwide.
org.opencontainers.image.title=ubuntu
org.opencontainers.image.version=26.04Policies (2 improved, 0 worsened)
Packages and Vulnerabilities (47 package changes and 15 vulnerability changes)
Changes for packages of type
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | base-files | 14ubuntu6.1 |
14ubuntu6.2 |
| ♾️ | bsdutils | 1:2.41.3-3ubuntu2 |
1:2.41.3-3ubuntu2.2 |
| ♾️ | curl | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | diffutils | 1:3.12-1 |
1:3.12-1ubuntu0.1 |
| ♾️ | gnu-coreutils | 9.7-3ubuntu2 |
9.7-3ubuntu2.1 |
| ♾️ | gpgv | 2.4.8-4ubuntu3 |
2.4.8-4ubuntu3.1 |
| ♾️ | libattr1 | 1:2.5.2-4 |
1:2.5.2-4ubuntu0.1 |
| ♾️ | libaudit-common | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libaudit1 | 1:4.1.2-1build1 |
1:4.1.2-1ubuntu0.1 |
| ♾️ | libblkid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libbz2-1.0 | 1.0.8-6build2 |
1.0.8-6ubuntu0.1 |
| ♾️ | libc-bin | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc-gconv-modules-extra | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libc6 | 2.43-2ubuntu2.3 |
2.43-2ubuntu2.4 |
| ♾️ | libcurl4t64 | 8.18.0-1ubuntu2.3 |
8.18.0-1ubuntu2.5 |
| ♾️ | libgcrypt20 | 1.12.0-2ubuntu1 |
1.12.0-2ubuntu1.1 |
| ♾️ | libmount1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libpam-modules | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-modules-bin | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam-runtime | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpam0g | 1.7.0-5ubuntu3.1 |
1.7.0-5ubuntu3.2 |
| ♾️ | libpq5 | 18.4-0ubuntu0.26.04.1 |
18.6-0ubuntu0.26.04.1 |
| ♾️ | libsmartcols1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | libssh2-1t64 | 1.11.1-1ubuntu0.26.04.3 |
1.11.1-1ubuntu0.26.04.4 |
| ♾️ | libssl3t64 | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | libsystemd0 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libudev1 | 259.5-0ubuntu3.3 |
259.5-0ubuntu3.4 |
| ♾️ | libuuid1 | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | login | 1:4.16.0-2+really2.41.3-3ubuntu2 |
1:4.16.0-2+really2.41.3-3ubuntu2.2 |
| ♾️ | mount | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | openssl | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | openssl-provider-legacy | 3.5.5-1ubuntu3.3 |
3.5.5-1ubuntu3.5 |
| ♾️ | perl-base | 5.40.1-7ubuntu0.1 |
5.40.1-7ubuntu0.3 |
| ♾️ | util-linux | 2.41.3-3ubuntu2 |
2.41.3-3ubuntu2.2 |
| ♾️ | zlib1g | 1:1.3.dfsg+really1.3.1-1ubuntu3 |
1:1.3.dfsg+really1.3.1-1ubuntu3.1 |
Changes for packages of type golang (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | github.com/canonical/pebble | 1.31.1-0.20260528050051-33f10658d3fd |
1.32.2-0.20260721212935-faa1696b477d |
| ♾️ | github.com/gorilla/websocket | 1.5.1 |
1.5.3 |
    |
|||
| Removed vulnerabilities (1): |
|||
| ➖ | golang.org/x/net | 0.40.0 |
|
   |
|||
| Removed vulnerabilities (10): |
|||
| ♾️ | golang.org/x/sys | 0.33.0 |
0.46.0 |
  |
|||
| Removed vulnerabilities (1): |
|||
| ♾️ | golang.org/x/term | 0.32.0 |
0.44.0 |
| ♾️ | stdlib | 1.26.3 |
1.26.5 |
   |
      ?labelColor=lightgrey){kind=icon} |
||
| Removed vulnerabilities (5): |
|||
Changes for packages of type npm (6 changes)
| Package | Versiongluwa/creditcoin3:latest |
Versiongluwa/creditcoin3:latest |
|
|---|---|---|---|
| ♾️ | @types/node | 26.1.2 |
22.7.5 |
| ♾️ | node-gyp | 13.0.1 |
13.0.2 |
| ♾️ | picomatch | 4.0.5 |
4.0.7 |
| ♾️ | undici | 8.10.0 |
8.10.2 |
| ♾️ | undici-types | 8.3.0 |
6.21.0 |
| ♾️ | ws | 8.21.2 |
8.21.3 |
DylanVerstraete
marked this pull request as draft
September 10, 2026 07:24
…wReceipts
Adds `BlockFetchMode` to the eth `Client`. `Json` is the existing
eth_getBlockByNumber(full) + eth_getBlockReceipts pair. `RawRlp` pulls the
node's stored RLP through `debug_getRawBlock` and `debug_getRawReceipts`,
decodes it with alloy consensus types and rebuilds the rpc-shaped block and
receipts (recovered sender, cumulative-gas delta, positional log metadata),
so `OrderedBlock::try_from_fetched_block` and the V1 leaf encoding run
unchanged on both paths. The header hash is sliced from the raw block RLP
rather than re-encoded, so it does not depend on alloy knowing every header
field a chain uses. Decoding and ECDSA recovery run on the blocking pool.
Intended for a historical sweep against a node we operate: payloads are
several times smaller and the node does no JSON marshalling. Public providers
generally do not expose the debug namespace, so `json` stays the default and
the mode is explicit.
Archiver: `--fetch-mode {json,raw-rlp}` / `FETCH_MODE`, applied to every
fetch client including the reconnect path; logged at startup.
Tests: unit tests build a two-transaction block (legacy transfer + 1559
creation with logs) and check the raw path against hand-built rpc types,
header-hash slicing, and error classification; an ignored integration test
(`ETH_RAW_RLP_TEST_URL`) compares both modes on a live node. Verified against
`reth node --dev`: 21 blocks, identical hashes/roots/senders/gas, and the
archiver binary in both modes stored identical roots over the same range.
The default `available CPUs - (max_fetch_tasks + 1)` sizing starves the compute pool when several shards share a box or run under a cgroup CPU quota (Rust's available_parallelism honours the quota). Allow it to be set explicitly via `--max-compute-threads` / `MAX_COMPUTE_THREADS`.
… independent RPC Lets us prove archived BSC roots match Chainstack (or any other ETH RPC) using the same simple_merkle_tree path, and keeps local sled/report artifacts out of git. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
jakerumbles
force-pushed
the
feat/eth-raw-rlp-fetch
branch
from
September 15, 2026 21:37
d82bbae to
5f7c1ad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
For the BSC historical sweep (plan: archiver → checkpoint ladder → attestors) the bottleneck is how fast we can pull blocks + receipts out of our own node. The JSON path makes geth build a JSON object per transaction and receipt and ships payloads of up to ~1 MB per BSC block.
debug_getRawBlock+debug_getRawReceiptsreturn the RLP the node already stores: several times smaller, no marshalling on the node.What
eth::ClientgainsBlockFetchMode { Json (default), RawRlp }andwith_fetch_mode. InRawRlpthe client fetches both raw payloads from the same provider, decodes them with alloy consensus types and rebuilds the rpc-shapedBlock+Vec<TransactionReceipt>:recover_signer), like the node does for JSONgas_used= delta of the cumulative counters; status, logs, bloom straight from the stored receiptEverything then flows into the unchanged
OrderedBlock::try_from_fetched_block(tx-root and receipt-root checks) and the V1 leaf encoding, so roots are byte-identical between modes. Decoding + ECDSA recovery run on the blocking pool. Public providers don't exposedebug_*, sojsonstays the default and the mode is explicit.Archiver:
--fetch-mode {json,raw-rlp}/FETCH_MODE, applied to every fetch client incl. the reconnect path;--max-compute-threads/MAX_COMPUTE_THREADSto size the merkle pool explicitly (thecores − (fetch_tasks+1)default starves it under cgroup quotas or multi-shard boxes).Attestor and proof-gen are untouched (default
Json).Verification
raw_rlp_tests): two-tx block (legacy transfer + 1559 contract creation with 2 logs), header whose roots match → raw path equals hand-built rpc types (sender, gas deltas, contract address, log indices, bloom) and yields the same merkle root; header-hash slicing equalshash_slow(); error classification (receipt count mismatch, wrong number, garbage block, garbage receipt); mode parsing.common/eth/tests/raw_rlp_vs_json.rs(ETH_RAW_RLP_TEST_URL): run againstreth node --devseeded with legacy/1559/transfer/create txs, blocks 75..=95: identical hash, item count, root, senders and gas_used on all 21 blocks (10 with txs).--fetch-mode raw-rlpandjsonover the same range on that node: 26 stored roots identical (10 non-empty).cargo clippy --workspace --all-targets -- -D warnings(excl. node crate),cargo fmt,taplo,cargo machete: clean.