Recover stale checkpoint restarts

[tdenisenko]

Summary

• Refresh stale persisted consensus state from the checkpoint sync endpoint during startup instead of failing after a long pause.
• Archive the old consensus state while preserving existing EL/log storage and known peer data.
• Bridge the missing EL gap from the saved execution head to the refreshed CL checkpoint by validating the parent-header chain to the checkpoint anchor, then fetching and ingesting the missing bodies, receipts, and logs.
• Pipeline checkpoint-gap body/receipt fetches with bounded lookahead so long-offline forward catch-up can request the next chunks while earlier chunks are being validated and written.
• Record checkpoint-gap progress once per contiguous batch so the dashboard no longer reports impossible multi-billion logs/sec rates after a batch write.

Root Cause

The client previously trusted the persisted cl/consensus_state.json on restart. If the machine was offline long enough, that state and the saved local EL head could fall outside the recent checkpoint window, so startup refused to continue even though the existing verified EL/log data was still usable.

The first recovery path also classified a fresh future-only CL anchor as a fatal reorg because it was ahead of the persisted recent-header window. After adding the bridge, the first batch-oriented implementation updated progress once per block immediately after writing an entire chunk, which compressed rate samples into microseconds and produced impossible dashboard logs/sec values.

Implementation Notes

• Startup can resolve a fresh checkpoint when persisted CL state or local EL progress is stale.
• Stale consensus state is archived rather than deleted.
• Future-only consensus anchors are treated as restart gaps, not fatal reorgs.
• The gap bridge validates headers up to the terminal CL execution anchor before ingesting logs, preserving the trust model.
• The gap bridge uses existing peer-manager body/receipt planning and validation primitives, with bounded pipeline depth based on serving peers.
• ProgressTracker now supports batched forward progress updates for contiguous forward chunks.

Validation

• cargo test -p logex-node archived_consensus_state
• cargo test -p logex-node restart_guard
• cargo test -p logex-sync locate_consensus_reorg
• cargo test -p logex-sync checkpoint_gap_pipeline_depth
• cargo test -p logex-sync forward_batch_progress_records_one_live_rate_sample
• cargo clippy -p logex-node -p logex-sync --all-targets -- -D warnings
• git diff --check
• Remote Mac mini native run on the saved full-sync data directory: storage integrity check passed, stale checkpoint recovery resumed, and fixed dashboard metrics reported sane logs/sec values rather than billion-rate artifacts.

Notes

Normal historical reverse sync and live tip-following paths are unchanged. The new pipeline is scoped to long-offline checkpoint-gap recovery.