Read only block headers in the -reindex scan (#916)

[bkeroack]

Fixes #916.

A flat-file -reindex plans its chain from the block headers alone. Its phase 1 scan still read every blk*.dat whole (std::fs::read) to get them. On mainnet that is the whole chain, about 700 GB, before the first block connects. Since #915 a continued reindex paid that again on every start.

Change

• FlatFileManager::for_each_block_header walks the same records as for_each_block, and ends a file on the same zero-padding and length checks. It reads each record's 8-byte framing and up to 80 header bytes, then seeks to the next record.
• Reads are 4 KiB: a block larger than that costs one read, and smaller blocks share reads. Each read is de-obfuscated at its file offset, so an xor.dat blocks dir works as before.
• The reindex scan uses it and logs the bytes_read. The plan comes from the same records as before. check_resumable_prefix compares the stored entries with the scan's map in memory, so a continued reindex reads no more than a fresh one.
• One behaviour change: a read error partway through a file now ends that file there, with a warning. Before, a failed fs::read dropped the whole file without a word.

Why headers only, not skipping the files below the stored tip

The issue offered both. Skipping the early files still means checking each stored entry against them. The issue proposes doing that by reading the header at each recorded position, which is the same one small read per block the header scan makes. Skipping would also need:

• a fallback for out-of-order blocks and more-work branches in early files;
• a fingerprint of the files kept in the rebuild marker.

The header scan makes the same reads without any of that. Its plan matches a full scan's by construction, and it speeds up a fresh reindex too.

Not measured: the issue's caveat that on a hard drive, one seek per large block may cost about as much as reading the files sequentially.

Tests

• the_header_scan_yields_the_records_the_full_scan_does builds a plaintext dir and an XOR-obfuscated one. Each holds records that straddle the 4 KiB reads, records shorter than a header, raw zero padding, and a record that runs past the end of its file. The header scan must yield the same headers, file numbers and positions as for_each_block, with at most one read per record plus one per file.
• a_reindex_scan_reads_the_block_headers_not_the_blocks uses 12 blocks with 32 KB of unspendable output each. A fresh reindex's scan and a continued reindex's scan must each read less than a fifth of the block files. The continued run must end on the uninterrupted run's UTXO set.
• Perturbations, each failing the named test:
  • scanning with for_each_block in the replay fails the reindex test;
  • dropping the de-obfuscation of the scan's reads fails the equivalence test;
  • so does dropping the padding check;
  • so does dropping the length check.

Release notes

• A CHANGELOG.md bullet.
• A section in docs/release-notes/0.6.1-pre.md.
• One sentence in the manual's IBD chapter.

Verified locally

• cargo clippy --all-targets --all-features --locked -- -D warnings
• cargo test --test e2e --no-run --locked --features e2e
• the node lib suite: 2150 passed
• the satd bin tests: 307 passed
• the regtest reindex and SIGTERM tests: 16 passed

This targets release/0.6 for 0.6.1. Once it merges, it needs a forward-port to master. The forward-port of #914 and #915 is open separately as #926.

🤖 Generated with Claude Code