perf(reth): stream authenticated MPT chunks to avoid guest OOM

[hero78119]

Problem

Large Reth blocks can exhaust the zkVM guest heap during post-execution state-root computation. The previous input format kept the full authenticated state trie, updated storage tries, and post-update witnesses live in one long-lived arena. This makes peak memory scale with the complete modified state and prevents larger shard caps from being used reliably on a 24 GB prover GPU.

Design Rationale

The state update is partitioned at authenticated MPT boundaries instead of weakening state validation or retaining the full mutable trie.

The host records modified accounts and storage-slot counts, then builds a weighted digest frontier around only the affected subtrees. Frontier boundaries are limited by node count, encoded bytes, and update weight. Each extracted subtree carries its absolute nibble prefix and expected root; inline child references conservatively remain in their hash-addressed parent because they cannot be represented safely by a digest-only frontier slot.

The guest authenticates each chunk against the parent frontier, verifies that account and storage witnesses belong to that chunk, applies the sorted bundle updates, rejects duplicate or unused witnesses, and replaces the frontier slot with the updated digest. Each chunk and its storage tries use a short-lived arena and are dropped before the next chunk is decoded. The final block state root is still checked through the existing execution path, so memory lifetime changes without changing the state transition being proven.

REVM state maps receive bounded capacity hints to reduce repeated guest allocations. The bounds remain growth hints rather than correctness limits. The Reth/REVM upgrade is handled through the narrow CapacityBlockExecutor seam so the upstream execution order, post-execution validation, bundle retention, and block-access-list handling remain explicit.

The trade-off is a versioned host/guest witness format and additional host-side frontier construction. This is preferred to global preallocation or an unauthenticated streaming format because chunk memory is bounded while every replacement remains tied to the parent state root.

Change Highlights

• crates/mpt: adds weighted frontier construction, nibble-key subtree operations, authenticated frontier replacement, and coverage for multi-chunk, empty-slot, and wrong-digest cases.
• crates/executor/client: introduces witness format version 2, streams post-state updates by authenticated chunk, allocates chunk/state arenas lazily, and preallocates bounded REVM state maps.
• crates/host-bench: emits the digest frontier and ordered chunk witnesses; adds native execution and ShardRAM-analysis paths for cycle and memory diagnosis.
• Workspace dependencies: updates Reth to v2.3.0, REVM to v40.0.3, Alloy to v2.x-compatible versions, and gkr-backend to v1.0.0-alpha.35.
• Benchmark/server configuration: uses a validated default CENO_MAX_CELL_PER_SHARD=4500000000 and keeps the value externally overridable.

Benchmark / Performance Impact

The primary result is successful proof generation for blocks that exercised the previous OOM path. Blocks 23817600, 25580200, 25586000, and 25586200 completed on a 24 GB RTX 4090 with the 4.5B default shard cap.

The timing comparison below is default-to-default rather than an isolated implementation microbenchmark: the base used CENO_MAX_CELL_PER_SHARD=1245708288, while this PR used the new 4500000000 default. The configuration difference is part of this PR and must be considered when interpreting the result.

Operation

Operation	base (s)	this PR (s)	Improve (base -> this PR)
E2E proof, block 23817600	70.500	67.100	4.8%
total_create_proof	67.676	64.064	5.3%

Layer

Layer	base (s)	this PR (s)	Improve (base -> this PR)
emulator	6.590	6.500	1.4%
app prove	54.400	50.700	6.8%
recursion	8.780	9.160	-4.3%
root verify	0.028	0.028	0.0%

Benchmark command(s):

CENO_MAX_CELL_PER_SHARD=<1245708288|4500000000> \
CENO_GPU_ENABLE_WITGEN=0 \
CENO_GPU_CACHE_LEVEL=1 \
CENO_GPU_JAGGED_RESHAPE_LOG_HEIGHT=23 \
RUST_MIN_STACK=536870912 \
cargo run --release --features "jemalloc,gpu,aot,parallel" \
  --bin ceno-reth-benchmark-bin -- \
  --mode prove-stark \
  --block-number 23817600 \
  --rpc-url "$RPC_URL_1" \
  --output-dir output \
  --cache-dir rpc-cache \
  --chain-id 1

Environment: self-hosted x64 Linux runner, 24 GB RTX 4090, Rust nightly-2025-11-20 (rustc 1.93.0-nightly). Base commit: a70dd56a; measured PR commit: 396c3986.

Raw data:

• base: block 23817600 summary
• this PR: block 23817600 summary
• OOM regression coverage: block 25580200, block 25586000, block 25586200

Testing

RUSTUP_TOOLCHAIN=nightly-2025-11-20 RUSTFLAGS='-Dwarnings' cargo make clippy
RUSTUP_TOOLCHAIN=nightly-2025-11-20 cargo test -p openvm-client-executor
RUSTUP_TOOLCHAIN=nightly-2025-11-20 cargo test -p openvm-mpt --features host test_frontier
bash -n server/prove_block.sh

• Workspace Clippy, both dependency-deny checks, and formatting completed successfully.
• Executor tests passed, including bounded capacity hints and zero eager allocation for the streaming-state arena.
• Both authenticated-frontier regression tests passed.
• run-benchmark-v2.yml parsed successfully and the linked end-to-end runs produced root proofs.

Risks and Rollout

• Witness format version 2 requires matching host and guest binaries. Existing cached block inputs remain usable because Ceno hints are regenerated by the benchmark, but serialized hint streams are not backward compatible.
• Frontier construction and replacement add correctness-sensitive MPT logic. Root, prefix, membership, duplicate, unused-witness, and hash-addressability checks fail closed before accepting an updated digest.
• The 4.5B shard cap is validated for a 24 GB RTX 4090 and remains configurable for other GPUs.
• The Reth/REVM dependency upgrade changes upstream execution APIs. Existing consensus validation, final state-root checks, unit tests, and end-to-end proof generation provide the rollout guardrails.
• Rollback consists of reverting this PR and restoring the previous shard-cap default; no persistent data migration is required.

Follow-ups (optional)

• Record peak guest heap and GPU memory as first-class benchmark metrics for controlled before/after comparisons.
• Extend the large-block regression set across additional state-access distributions and GPU capacities.

Copilot Reviewer Directive (keep this section)

When Copilot reviews this PR, apply .github/copilot-instructions.md strictly.