Continue an interrupted -reindex from the block files on any start (#906)

[bkeroack]

Fixes #906. Stacked on #914 (#907): merge #914 first, then retarget this PR to release/0.6.

The problem

0.6.0 refused every start after an interrupted full -reindex except one with --reindex, and that start wiped the database and replayed from genesis. A mainnet replay takes days, so every stop, reboot or settings change during one threw the work away. A package that passes --reindex once, as StartOS's migration from Bitcoin Core does, was left with a node that would not start.

The change

You chose this behaviour: any start that finds an unfinished full reindex continues it, whatever the flags.

How it continues. resume_reindex_from_flat_files scans the block files and plans the chain exactly as before. Before connecting anything, check_resumable_prefix checks every block the interrupted run stored against the scan:

• the stored tip is the planned block at its height;
• each entry at or below it is Valid at its planned height, with the header the files hold, at the file position the scan found;
• each entry's chainwork is the sum of its ancestors' header work.

What the continued replay reads from those entries (parent links, chainwork, the timestamps behind median time past) is then backed by the block files, as in a replay from genesis. It then connects the rest. Nothing is downloaded and no block file is written. The check costs one index read per stored block.

When it can't continue. The rebuild marker gains resumable. It is set once the wipe has finished and genesis is indexed and flushed durably, which is just before the scan. Three cases run the reindex again from genesis instead, still on any start:

• a marker without the flag: one 0.6.0 wrote, or one written before its wipe had finished;
• a marker at another chainstate schema;
• a stored chain that fails the check. satd exits 1 with ReindexCannotResume, having changed nothing, and clears resumable; the next start runs from genesis.

This differs from Core in one case. Core also continues an interrupted reindex on a plain start, but -reindex passed again makes Core wipe and start over. Here it continues: you chose that so a package that keeps passing --reindex, or an operator following 0.6.0's "restart with --reindex" message, doesn't lose the replay. To start over, remove the chainstate directory.

Smaller pieces:

• A fresh reindex_from_flat_files now refuses a chainstate past genesis.
• A continued run that had already reached -stopatheight connects nothing more and indexes no side chain.
• An interrupted -reindex-chainstate is unchanged: its flags restart it from genesis (reindex: support resuming an interrupted -reindex-chainstate #508).

Tests

• Unit, node:
  • a stopped reindex continues onto the uninterrupted run's UTXO set;
  • a stored entry at the wrong position fails ReindexCannotResume and connects nothing;
  • a fresh reindex refuses a stored chain;
  • a continued run past -stopatheight connects nothing;
  • a 0.6.0 marker reads as not resumable.
• Unit, satd: the decision matrix across flag combinations. A resumable marker at the current schema continues; a 0.6.0 marker or one at another schema restarts.
• Integration (regtest):
  • a_killed_full_reindex_continues_from_the_block_files: SIGKILL after the durable flush at height 1000.
  • a_full_reindex_killed_before_it_connects_anything_continues: SIGKILL during the scan. If the kill lands after the first flush (a stalled test thread), it retries on a fresh copy, up to three times.
  • In both, a plain start must log that it continues, clear nothing, write no block file (names and sizes compared), and end on the fixture's UTXO set.
  • a_full_reindex_left_by_0_6_0_runs_again_on_any_start replaces the 0.6.0 refusal test.
  • Stop a startup block replay cleanly on SIGTERM, also as PID 1 (#907) #914's SIGTERM tests now finish with a plain start that continues.
• Perturbations:
  • Routing ContinueFullReindex to a restart fails the kill test.
  • Dropping the position comparison fails the unit refusal test.
  • The prefix is skipped twice over: the path is sliced, and its headers are dropped from the scan's map for memory. Undoing both fails the continue test with BadPrevBlock; either alone still skips it.
  • Dropping the durable flush before resumable is set does not fail a kill test. Genesis reaches RocksDB through the cache's pass-through with the WAL on, which a kill doesn't lose, so the flush only adds power-cut safety. The test notes say so.

Local gates

• Both clippy runs are clean.
• cargo test --all --locked: 442 of 453 regtest tests pass. The 11 failures are the address-index backfill tests, whose disk guard wants 80 GiB free (this machine has 76 GB on /tmp).
• After rebasing onto release/0.6 (Correct the streaming limit docs and stale SDK and proto comments #898, Report a refused watch add in-band (WatchAddRejected) #899): clippy is clean; the targeted unit tests and the 28 reindex, rebuild and shutdown integration tests pass.

Since opening

• Rebased onto Stop a startup block replay cleanly on SIGTERM, also as PID 1 (#907) #914's block abort. The resume's stop path keeps the resume_from offset in its progress count.
• The first CI run failed only test_esplora_address_txs_mempool_visibility, which also failed locally under load and passes alone. It is unrelated to this change.

Not done

• A continued reindex reads every block file in full again before it connects anything: about 700 GB on mainnet, more than an hour on a hard drive. Filed as A continued -reindex reads every block file in full before it connects anything #916, to fix after these two.

• No integration test drives the ReindexCannotResume exit path end to end. That needs a stored chain the files contradict, which the regtest harness can't produce without writing RocksDB rows by hand. The check itself is unit-tested, and the exit path is a few lines in main.rs.

• Reindex throughput (Start9's other point) is untouched.

🤖 Generated with Claude Code