fix: reduce overlapping import costs

[zxch3n]

Summary

Reduce repeated-history import costs without narrowing peer-id reuse detection (#1118):

• Compare byte-identical known blocks before decoding them; dirty in-memory blocks shadow stale KV bytes.
• Compare cold Text-insert blocks directly from encoded deps/container IDs/positions/UTF-8 text, preserving exact mismatch IDs across merged histories. Other op kinds and short imported changes keep the semantic fallback. Reader errors also fall back without recording parse failures; a content mismatch is used only after the whole block passes eligibility checks.
• Decode unmatched snapshot history in a temporary arena, take owned frontier changes without cloning, then move only the checked new suffix into the document arena. Parent links are registered on converted ops.
• Detached imports only open rollback for MovableList Move/Set validation, including pending ops that a Text/scalar update unlocks. Read the detached flag once so rollback preflight and execution select the same mode even during concurrent attach/detach. The batch retains its txn lock and outer rollback scope.
• Add regression coverage for accepted merged cold Text overlaps (including Unicode and a check that the known block remains unparsed), peer-id conflicts, and fallback for encoded op-length/change-boundary differences accepted by the ordinary parser. Include a patch changeset, context updates, the verifier probe and an interleaving runner. The new cold range reader rejects integer overflows as errors.

Rationale

The previous overlap check parsed and retained every cold local block. Snapshot decoding also allocated incoming known text/list values in the document arena and cloned changes that trimming dropped. This change keeps the full comparison scope: it skips only byte-proven equal blocks and otherwise checks the same op content/dependencies. Existing limits (delta sync without the conflicting prefix, no-new-op duplicates, shallow history) are unchanged and documented in context/import-peer-id-reuse.md.

The cold comparison runs only when an imported change reaches the end of the local block. Short imported changes over a non-identical cold block still parse it, so the remaining roughly 2.2x overlap_snap cost at FAT=1 is expected. The cold reader is an optimization, not a second validity rule: only the ordinary parser may declare local history unparsable.

Under #[cfg(test)], the override at the end of preflight_import_changes forces rollback only for applies_to_dag && !detached; detached tests continue to exercise the production rollback decision.

Measurements

Measured on macOS arm64, rustc 1.96.0 (ac68faa20 2026-05-25), release builds,
CARGO_BUILD_JOBS=4. Baselines: 1.16.3 ad5b2a6d4546473d9f4a96412d2ae6808c8403ec,
main c00c9fa501f8d32f68d6255eacb7035a67fb6ab6, fixed = optimization commit
0c8253e1b966d475a3fb5376a9cdb769ea0d0b40. The table retains the initial
PR measurements; the review follow-up reran correctness checks, not this matrix.
The verifier directory was read only; its probe and NOSEED/SEEDSYNC patch were
copied here. Each cell has 3 interleaved old/main/fixed process runs, each reporting
3 timed rounds after a warmup. Tables show medians of process medians.
The machine is shared; recorded one-minute load ranged 3.48-4.37.
Use ratios, not an individual wall time.

The first five scenarios use FAT=64 REPEATS=4. Other scenarios use FAT=1;
text_stream uses NOSEED=1 (no concurrent seed). snap_mem has no warmup:
its time column is the median of each process's four growing-snapshot imports,
then the median over 3 processes. Its first-import latency is separately covered
by snap_plus.

Scenario	Changes	1.16.3 ms	main ms	fixed ms	fixed/main	fixed/1.16.3
overlap_snap	8,000	1.757	3.095	2.328	0.75x	1.32x
overlap_snap	32,000	7.377	12.803	9.504	0.74x	1.29x
overlap_snap	64,000	15.462	26.566	19.440	0.73x	1.26x
overlap_partial	8,000	1.568	2.249	1.852	0.82x	1.18x
overlap_partial	32,000	6.546	9.233	7.676	0.83x	1.17x
overlap_partial	64,000	13.401	19.139	15.429	0.81x	1.15x
overlap_mem	8,000	0.387	0.397	0.393	0.99x	1.02x
overlap_mem	32,000	1.549	1.567	1.567	1.00x	1.01x
overlap_mem	64,000	3.107	3.125	3.082	0.99x	0.99x
snap_plus	8,000	2.474	3.925	1.597	0.41x	0.65x
snap_plus	32,000	10.213	16.615	6.475	0.39x	0.63x
snap_plus	64,000	20.615	34.627	13.397	0.39x	0.65x
snap_mem	8,000	1.148	1.919	0.203	0.11x	0.18x
snap_mem	32,000	4.825	7.888	0.722	0.09x	0.15x
snap_mem	64,000	9.572	15.917	1.581	0.10x	0.17x
detached	8,000	6.865	8.897	6.954	0.78x	1.01x
detached	32,000	28.110	35.118	28.423	0.81x	1.01x
detached	64,000	57.046	72.767	57.281	0.79x	1.00x
mlist_batch	8,000	7.460	8.909	8.993	1.01x	1.21x
mlist_batch	32,000	29.524	37.508	37.358	1.00x	1.27x
mlist_batch	64,000	79.806	96.798	96.028	0.99x	1.20x
text_stream	8,000	18.200	20.952	20.529	0.98x	1.13x
text_stream	32,000	73.324	85.664	82.378	0.96x	1.12x
text_stream	64,000	149.703	165.534	166.158	1.00x	1.11x
batch	8,000	29.818	29.709	29.698	1.00x	1.00x
batch	32,000	124.071	127.269	125.134	0.98x	1.01x
batch	64,000	257.137	255.777	255.857	1.00x	1.00x

Retained allocator memory

Process-wide malloc_zone_statistics().size_in_use, in KiB. snap_plus is the
one-import delta; snap_mem is the delta from the loaded base to import 4.
These statistics have allocator growth steps and include state/history caches;
the exact arena regression test verifies that no incoming known prefix survives.

Scenario	Changes	1.16.3 KiB	main KiB	fixed KiB	fixed/main
snap_plus	8,000	2,164	4,598	1,604	0.35x
snap_plus	32,000	8,682	19,192	6,452	0.34x
snap_plus	64,000	21,282	41,202	16,827	0.41x
snap_mem	8,000	5,382	10,839	1,605	0.15x
snap_mem	32,000	21,547	43,384	6,455	0.15x
snap_mem	64,000	47,007	91,137	16,828	0.18x

Fixed snap_mem retained-heap deltas from its loaded base after imports 1/2/3/4
(median of 3 processes, KiB):

Changes	Import 1	Import 2	Import 3	Import 4
8,000	1,605	1,605	1,605	1,605
32,000	6,455	6,455	6,455	6,455
64,000	16,826	16,827	16,828	16,828

FAT=1 controls for the original regression shape

Same release/interleaving/round/repeat settings; these reproduce the originally
reported 9 ms and 15 ms regressions without large inserted strings.

Scenario	Changes	1.16.3 ms	main ms	fixed ms	fixed/main	fixed/1.16.3
overlap_snap	8,000	0.266	1.071	0.594	0.55x	2.23x
overlap_snap	32,000	1.056	4.438	2.360	0.53x	2.23x
overlap_snap	64,000	2.170	9.080	4.828	0.53x	2.22x
overlap_partial	8,000	0.247	0.673	0.445	0.66x	1.80x
overlap_partial	32,000	1.043	2.725	1.667	0.61x	1.60x
overlap_partial	64,000	2.181	5.479	3.372	0.62x	1.55x
snap_plus	8,000	0.801	1.745	0.361	0.21x	0.45x
snap_plus	32,000	3.274	7.571	1.137	0.15x	0.35x
snap_plus	64,000	6.492	15.340	2.236	0.15x	0.34x

R1 retains some comparison cost above 1.16.3, which did not check known history.
Movable-list element validation is unchanged: mlist_batch retains its roughly
1.2x cost relative to 1.16.3. Text streaming and batch controls are comparable to
main. Snapshot decoding is faster than both baselines and repeated imports plateau.

Reproduce with separate binaries named old, main, fixed:

export CARGO_BUILD_JOBS=4
cargo build -p examples --release --example import_scaling_stress
# Copy each revision's binary to $BENCH_BIN_DIR/{old,main,fixed}.
python3 crates/examples/import_overlap_measure.py \
  --bin-dir "$BENCH_BIN_DIR" --output-dir /tmp/import-overlap-fat64
python3 crates/examples/import_overlap_measure.py \
  --bin-dir "$BENCH_BIN_DIR" --output-dir /tmp/import-overlap-fat1 \
  --fat 1 --scenarios overlap_snap overlap_partial snap_plus

Samples: FAT=64 matrix,
FAT=1 controls.
The runner also saves full probe output and load readings to raw.jsonl.

Validation

Re-run after the review fixes, with CARGO_BUILD_JOBS=4:

• cargo test -p loro --test import_reused_peer_id: 11 passed.
• cargo test -p loro-internal --lib cold_text: 3 passed (the new positive/fallback tests and the cold dependency-conflict test).
• cargo test -p loro-internal --lib import_atomicity: 18 passed.
• cargo test -p loro-internal --lib known_history: 2 passed.
• cargo test -p loro-internal --lib oplog::change_store: 23 passed, including this PR's identical-block, snapshot-arena, range-overflow and cold-block tests.
• cargo test -p loro-internal --lib: 415 passed, 4 ignored.
• pnpm test: 1754 passed, 37 skipped across 102 nextest binaries; 62 doctests passed (57 + 1 + 4).
• pnpm check: passed (cargo clippy --all-features -- -Dwarnings).
• pnpm test-loom: 9 passed, 0 failed (LOOM_MAX_PREEMPTIONS=2, release, RUSTFLAGS='--cfg loom').
• git diff --check: passed.

The initial release probe check also passed: 15 PASS + CHECK_OK on old/main/fixed; old still accepts the deliberate peer-id conflict, main/fixed reject it as expected.

No API or binary-format change. No broad fuzz/browser matrix was run. The verifier worktree was not modified.

[github-actions]

WASM Size Report

• Original size: 3332.91 KB
• Gzipped size: 1110.96 KB
• Brotli size: 778.29 KB