Description
With op-reth v2.4.0 and v2 proofs history enabled, a clean SIGTERM can leave the persisted proofs window behind the execution-chain tip. On restart the gap is permanent: ExEx notifications for the missing blocks are not replayed, proofs init is a no-op for an initialized store and debug_proofsSyncStatus.latest remains below eth_blockNumber. Rebuilding the proofs store from the chain DB repairs it.
Reproduction
- Initialize v2 proofs history and run op-reth with
--proofs-history while an op-node sequencer produces blocks.
- Observe a small live lag, for example execution tip
3642 and persisted proof latest 3640.
- Send SIGTERM and wait for op-reth to exit cleanly.
- Restart against the same execution DB and proofs DB.
- The execution tip remains
3642, while debug_proofsSyncStatus.latest remains 3640 indefinitely. Running op-reth proofs init does not advance it because the store is already initialized.
This was reproduced with the official op-reth:v2.4.0 image. The same shutdown path is still present on current develop.
Suspected cause
OpProofsExEx::handle_notification emits ExExEvent::FinishedHeight after the collector action returns, but that action only buffers the block; persistence happens asynchronously and below-threshold blocks can remain in memory. The engine shutdown path drains only an already in-flight save and does not schedule the remaining memory buffer for persistence. The ExEx WAL therefore considers those notifications finished even though their proof state was never persisted.
Relevant code:
rust/op-reth/crates/exex/src/lib.rs: FinishedHeight is emitted after queueing collector work.rust/op-reth/crates/trie/src/engine/runner.rs: shutdown calls only drain_persistence().rust/op-reth/crates/trie/src/engine/state.rs: drain_persistence() waits for an existing save but does not start one for buffered blocks.
Expected behavior
A clean shutdown should persist all buffered proof-history blocks before the collector exits and before a completed ExEx height can become unrecoverable. A regression test should cover shutdown with fewer buffered blocks than the persistence threshold and verify that the restarted proof window reaches the execution tip.
Description
With
op-reth v2.4.0and v2 proofs history enabled, a clean SIGTERM can leave the persisted proofs window behind the execution-chain tip. On restart the gap is permanent: ExEx notifications for the missing blocks are not replayed,proofs initis a no-op for an initialized store anddebug_proofsSyncStatus.latestremains beloweth_blockNumber. Rebuilding the proofs store from the chain DB repairs it.Reproduction
--proofs-historywhile an op-node sequencer produces blocks.3642and persisted proof latest3640.3642, whiledebug_proofsSyncStatus.latestremains3640indefinitely. Runningop-reth proofs initdoes not advance it because the store is already initialized.This was reproduced with the official
op-reth:v2.4.0image. The same shutdown path is still present on currentdevelop.Suspected cause
OpProofsExEx::handle_notificationemitsExExEvent::FinishedHeightafter the collector action returns, but that action only buffers the block; persistence happens asynchronously and below-threshold blocks can remain in memory. The engine shutdown path drains only an already in-flight save and does not schedule the remaining memory buffer for persistence. The ExEx WAL therefore considers those notifications finished even though their proof state was never persisted.Relevant code:
rust/op-reth/crates/exex/src/lib.rs:FinishedHeightis emitted after queueing collector work.rust/op-reth/crates/trie/src/engine/runner.rs: shutdown calls onlydrain_persistence().rust/op-reth/crates/trie/src/engine/state.rs:drain_persistence()waits for an existing save but does not start one for buffered blocks.Expected behavior
A clean shutdown should persist all buffered proof-history blocks before the collector exits and before a completed ExEx height can become unrecoverable. A regression test should cover shutdown with fewer buffered blocks than the persistence threshold and verify that the restarted proof window reaches the execution tip.