spv: parallel genesis header download over checkpoint segments by xanimo · Pull Request #378 · dogecoinfoundation/libdogecoin

xanimo · 2026-08-02T19:56:23Z

Initial header sync requests headers serially from a single peer, so a genesis
sync is bounded by one peer's latency for the whole chain.

This splits the range into checkpoint-delimited segments and downloads them
concurrently across connected peers, then stitches them in order. Segments are
validated against the checkpoint that bounds them, so a peer that lies about a
segment is caught at the boundary rather than after the fact.

To make the segmentation useful the checkpoint table is densified from 33 to 87
entries at 100k-block intervals. Sparse checkpoints mean long unbounded
segments, which is exactly where a malicious peer has room to work.

Serial sync remains the fallback when fewer peers are connected than segments.

Fill all gaps larger than 100k blocks in the mainnet checkpoint array using block hashes queried from local Dogecoin Core 1.14.99 via RPC. Previous latest checkpoint: height 5,400,000 (Sep 2024). New latest checkpoint: height 6,100,000 (Jan 2026). New entries cover heights 100k–6.1M at 100k intervals, reducing the maximum headers-sync range on a fresh -p (checkpoint) sync from ~840k blocks to ~140k blocks. (cherry picked from commit efd12f5)

Header sync from genesis is serial: one peer, 2000 headers per round trip, several million headers. The chainparams checkpoint array already partitions the chain into intervals whose endpoints are known-good block hashes, so those intervals can be fetched independently and stitched back together in order. Segment i covers the open-closed height range (checkpoint[i-1], checkpoint[i]], with segment 0 anchored at the genesis hash. Each is assigned to a different peer, buffered as raw 80-byte headers, and flushed into the primary headers DB strictly in ascending segment order so the on-disk chain is never written out of sequence. On mainnet the current array yields 86 segments. Because every segment is bounded by a checkpoint at both ends, the chain is validated by construction, so the flush path sets two new headers DB flags: skip_pow, which skips the per-header scrypt verification that the checkpoint anchors already guarantee, and batch_write, which suppresses the per-record fdatasync in favour of a single commit at the end. Both default off and are only set for the duration of a segment flush. Enabled with -H/--genesis_headers; without it nothing changes, since every hook is guarded on par_hdr->active. dogecoin_net_spv_request_headers returns early while a parallel sync is running so the serial path cannot interleave, handshake completion assigns a segment instead of issuing a getheaders, and incoming headers are routed to the owning segment. par_hdr_skip_auxpow advances past the AUXPoW chain data that follows the standard header for merge-mined blocks. It deliberately does not call check_auxpow: the segment endpoints are checkpoints, so the work is already accounted for, and running scrypt over millions of headers is the cost this feature exists to avoid. Independent of compact block filters -- the implementation makes no reference to filter state. 78/78, and -H yields 86 segments with the expected height ranges (0: 1..100000, 1: 100001..104679).

Wires the BIP157 message flow into the SPV client: getcfcheckpt to pin the peer's checkpoints, getcfheaders to build and validate the filter header chain, getcfilters to fetch filters, GCS matching against watched scripts, and a getdata for the blocks that matched. Filter headers are persisted through the cfheaders/cfilters DBs added in the previous layer, so a restart resumes from the stored tip rather than re-downloading. Cached filters are rescanned at startup, which covers scripts registered after the filters were stored. BIP37 and BIP157 are mutually exclusive here, by design and for privacy: a bloom filter tells the peer which scripts are being watched, which is the disclosure compact filters exist to avoid. Compact filters are on by default and dogecoin_spv_client_filterload fails closed while they are, so a BIP37 consumer opts out explicitly via dogecoin_spv_enable_compact_filters(client, false). The test asserts both halves of that. headersdb gains a block-hash-by-height lookup with a resume cursor. cfheaders batches request ascending heights, and rescanning the whole file per lookup is O(N^2) over a 6.2M-record file; resuming from the last match keeps it linear. Deliberately not included, to keep this reviewable and because each belongs elsewhere: - parallel cfilter/cfheaders download. The state fields live in the compact_filter layer; the logic lands in its own PR. Every path here takes the sequential branch. - the par_hdr parallel genesis header download, which is dogecoinfoundation#378. - skip_pow and the stored-hash-on-load shortcut. Both were in the prototype branch, both weaken what header loading verifies, and neither is needed by filter sync. They belong with the bulk-load work where their trust assumptions can be reviewed as a unit. Header loading here is unchanged from 0.1.5-dev. Report the height range actually covered on completion. cf_scan_start_height was only set on one of the two scan-start paths, and the log falls back to 1 when it is unset, so a run that covered the checkpoint tail reported "scanned heights 1..tip" -- overstating coverage in exactly the way that hides a gap. Verified against a local 1.14.99 node serving BIP157 with the corrected cfilter wire order, run with an isolated HOME so the real filter cache was untouched: 1442 cfilters parsed off the wire, full getcfcheckpt/getcfheaders/getcfilters round trip, zero validation errors or misbehaviour, and the completion line now reports 6314395..6315836 rather than 1..6315836. That run does not exercise GCS matching: -a is not wired to the filter state until the CLI layer, so no scripts were watched. The matching block is byte-identical to the prototype's sequential arm, which was separately confirmed to match at a predicted height over the wire, and the unit tests cover GCS matching directly -- but this layer's own end-to-end run proves the wire path, not the match path. WITH_NET=ON with -DBUILD_SHARED_LIBS=1: 81/81. WITH_NET=OFF: 74/74.

Connects the BIP157 client to the command line, which is what makes the feature reachable by a user rather than only by the library API. -a/--address registers the address as a watched script with the filter state, so cfilters are matched against it -e/--no_cfilters disables compact filters, the documented opt-out for a consumer that wants BIP37 instead -o logs filter header checkpoints during sync --cfheaders_path override cfheaders.dat location --cfilters_path override cfilters.dat location wallet gains dogecoin_wallet_add_watchonly_addr so a watched address is persisted without a private key. It follows the existing add-address path exactly, including that dogecoin_address_to_pubkey_hash returns a pointer into a static buffer and is not freed by its existing caller either. Deliberately excluded, because each is the seam with another feature and not the CLI's to own: --cf_from_genesis, --genesis_headers, --filter_hash_db, --cf_workers --genesis_headers is the clearest case. It calls dogecoin_spv_client_enable_genesis_headers, which initialises par_hdr *and* resets cf_start_height and clears the CF databases -- one function spanning parallel header download and compact filters. Exposing it here would pull dogecoinfoundation#378 into this PR. The peer-list replication that gave each parallel worker its own TCP connection goes with them. End to end against a local 1.14.99 node serving BIP157, run with an isolated HOME so the real filter cache was untouched: [bip157] watching address: DJoQKLWJtM251NhzsR7DmbgT88TwqBbPX7 [bip157] MATCH at height 6315000 [bip157] all filters processed: scanned heights 6314418..6315859, 1 matched blocks [bip157] requesting 1 matched full blocks across 1 peers [bip157] processing 2 txs from matched block height=6315000 1442 cfilters parsed off the wire, zero validation errors. The address was taken from block 6315000 via getrawtransaction before the run, so the matched height was predicted rather than read back afterwards. This closes the gap left open in the spv layer, where -a was not yet wired to the filter state and so GCS matching was never exercised end to end. WITH_NET=ON with -DBUILD_SHARED_LIBS=1: 81/81. WITH_NET=OFF: 74/74.