Current behavior
The destination side of a predicate move funnels all streamed data through the destination group's Raft, one chunk at a time. batchAndProposeKeyValues (worker/predicate_move.go) accumulates 32MB of KVs, then calls proposeAndWait and blocks until that chunk is replicated to quorum, written to the Raft WAL, and applied on the leader before the next chunk is proposed. The apply path (populateKeyValues -> posting.TxnWriter) opens one badger transaction per KV, so a 32MB chunk of small posting lists commits tens of thousands of individual managed transactions.
Net effect: every byte moved is written roughly 6x on the destination group (Raft WAL plus badger, times 3 replicas), and throughput is bounded by one quorum round trip per 32MB chunk. The report in #9784 works out to about 0.35 MB/s on-disk (20GB in roughly 16 hours).
Proposed change
Two contained optimizations in the receive path:
- Pipeline the proposals. After the initial
CleanPredicate barrier, chunks are order-independent: badger's stream framework partitions key ranges so keys are disjoint, and every KV is written at the same version (MoveTs). Keep a bounded number of proposals in flight (4-8) and wait for all of them before acking EOF.
- Batch the writes. All KVs in a move share the same commit timestamp, so
populateKeyValues can write them through shared transactions or a write batch instead of one CommitAt per KV.
Plausibly a 4-8x improvement for changes confined to batchAndProposeKeyValues and populateKeyValues.
Context
Surfaced while fixing #9784 (size-aware move timeout). That fix stops large moves from being cancelled but does not make them faster.
Jira: DGR-308 (https://istari.atlassian.net/browse/DGR-308)
Current behavior
The destination side of a predicate move funnels all streamed data through the destination group's Raft, one chunk at a time.
batchAndProposeKeyValues(worker/predicate_move.go) accumulates 32MB of KVs, then callsproposeAndWaitand blocks until that chunk is replicated to quorum, written to the Raft WAL, and applied on the leader before the next chunk is proposed. The apply path (populateKeyValues->posting.TxnWriter) opens one badger transaction per KV, so a 32MB chunk of small posting lists commits tens of thousands of individual managed transactions.Net effect: every byte moved is written roughly 6x on the destination group (Raft WAL plus badger, times 3 replicas), and throughput is bounded by one quorum round trip per 32MB chunk. The report in #9784 works out to about 0.35 MB/s on-disk (20GB in roughly 16 hours).
Proposed change
Two contained optimizations in the receive path:
CleanPredicatebarrier, chunks are order-independent: badger's stream framework partitions key ranges so keys are disjoint, and every KV is written at the same version (MoveTs). Keep a bounded number of proposals in flight (4-8) and wait for all of them before acking EOF.populateKeyValuescan write them through shared transactions or a write batch instead of oneCommitAtper KV.Plausibly a 4-8x improvement for changes confined to
batchAndProposeKeyValuesandpopulateKeyValues.Context
Surfaced while fixing #9784 (size-aware move timeout). That fix stops large moves from being cancelled but does not make them faster.
Jira: DGR-308 (https://istari.atlassian.net/browse/DGR-308)