fix(loro.js): converge plain-text edits with Rust, add a Rust differential suite

[zxch3n]

Summary

Adds a randomized differential suite that runs the same Text edits in loro.js and in Rust (loro-crdt, nodejs WASM) and compares every step, and fixes the plain-text divergences it found. Rich text (marks) is covered by the follow-up #1137; this PR runs the suite without marks.

Breaking (major changeset, loro.js 0.2.1 → 1.0.0): loro.js reads all data with Rust's semantics and no version marker (decision: option A). Documents edited only with loro.js 0.2 that have the concurrent-insert pattern below read differently, and cursors encoded by 0.2 with side 1, or at the end of the text, resolve one character earlier (a cursor reports its target's own offset, as in Rust; other cursors are unchanged, whatever the characters). A document loaded from a 0.2 snapshot keeps its 0.2 text only while it holds that state: history imported after the snapshot, and its own shallow-snapshot export, read like Rust. README "Upgrading from 0.2" gives measured readings and migration paths (reading with 1.0 right after importing a 0.2 snapshot is only valid when the snapshot is the final version); the decision is recorded in context/loro-js-rust-differential.md.

The suite (loro-js/tests/richtext-differential.test.ts)

• Seeded multi-peer scenarios (tests/support/richtext-differential.ts): insert/delete at UTF-16 and UTF-8 boundaries (BMP, CJK, astral, multi-scalar emoji); sync by binary or JSON updates, within a runtime or crossed, so each runtime reads the other's positions; checkout to two versions in a row (commit versions or the version right after a random op) and back; full and shallow snapshots from either runtime imported into both; cursors; revertTo; undo/redo. Marks, unmarks, and all four expand modes are generated when enabled (fix(loro.js): keep rich-text style anchors in the text sequence #1137).
• After each step: normalized delta with attributes, UTF-16/Unicode/UTF-8 lengths, cursor positions, loro.js events replayed against loro.js state, and every inserted scalar and mark in the JSON history with its entity position.
• Failing seeds shrink to a replayable action list (shrinkActions).
• Skips without a WASM build. pnpm test-loro-js (new, added to test-all after release-wasm) requires the build, so CI now runs the loro.js tests too.
• Design and knobs: context/loro-js-rust-differential.md. New loro-js/AGENTS.md points agents at it and at the interop fixture pipeline (closing the CONTEXT-GAPS entry).

Fixes (all plain text)

Divergence	Fix
Concurrent insert placed after unrelated concurrent text: after the last sibling, indexedFugueInsertion jumped to the origin-right bound, but that bound is only the next causally included element, so an ancestor's later concurrent subtree can sit before it. Peers 1/2/3: b, 9, é between them, concurrent a after b → Rust béa9, loro.js bé9a.	#1139's fix, merged into this branch: after the last child, binary-search where originLeft's subtree ends (descent test that skips implicit runs). This PR first walked the sibling subtree's ID runs; #1139 is faster on long runs and real traces, slower on deep explicit-link chains (measurements in the PR comment and context/loro-js-performance.md).
Imported deletes applied by recorded start_id. Rust's tracker deletes by position; its WASM build can record a wrong start_id around astral text (#1135), e.g. revertTo([]) over 👍é🏽.	LoroText._deleteTargets resolves by position in the op's causal view; the recorded IDs are only a fallback. Local deletes skip the lookup.
checkout/diff/revertTo restored a character twice when two concurrent deletes removed it (packed spans hand out a new element view per lookup, so a Set did not dedupe).	Collect transition elements by ID.
Text cursors: end/empty cursors carried an ID; getCursorPos added the target length for side 1; deleted targets used the next element; the encoded origin was UTF-16.	Rust semantics throughout (#textCursorPos); encoded cursors now match Rust byte for byte.
Op shape: loro.js did not merge consecutive inserts in a transaction and wrote multi-run deletes first run first.	Merge inserts and write delete runs like Rust, so these calls use the same counters (op IDs for undo and mid-change frontiers). The ops are not identical to Rust's in every case (see known gaps).
Checkout across scattered edits was O(operations × elements): #canTransitionRecords walked the whole sequence once per insert op (698 ms at 8k characters with 2k edits on main).	Check each container once with all its runs, reading IDs without building element views (4.2 ms; text-scattered-edits-checkout).

Rationale

The previous interop checks were fixed fixtures. The suite found a convergence bug in plain concurrent text (the Fugue one) that fixtures had not, and three Rust bugs (#1135). Resolving deletes by position is what makes loro.js agree with Rust on documents Rust has already written with mismatched IDs.

Validation

• pnpm --dir loro-js check on the merge with main (fix(loro.js): correct checkout after a snapshot import #1126–fix(loro.js): drop containers deleted before the shallow root, keep tree node metadata #1128, fix: resolve a container's parent from the change that created it #1159): 24 files, 368 tests incl. fix(loro.js): correct checkout after a snapshot import #1126's snapshot-checkout suite and fix(loro.js): stop a concurrent insert at the end of its origin's subtree #1139's fugue-index.test.ts; format, lint, typecheck, build. tests/text-interop.test.ts (regression tests per fix, incl. a Rust-written fixture with a mismatched start_id) fails on the previous main; tests/legacy-data.test.ts pins how loro.js and loro-crdt read fixtures written by the published loro.js@0.2.1.
• Differential suite without marks against a WASM build of main ce6498b5: 300 seeds × 120 steps (13001–13150, 14001–14150) with per-op two-version checkouts, plus 600+ earlier seeds of 80–160 steps (with snapshotCheckout off, below). After the second review: 200 seeds × 120 steps with the second checkout version made independent of the first (84431ba; it used to repeat the first version for most history lengths).
• B4 (bench:b4, 259,778 actions), alternating with main at 1-minute load 6–9 (September 29): apply 233–235 vs 238–240 ms; update export 9 vs 72–73 ms (274,574 vs 1,153,540 B) because consecutive inserts merge into one op as in Rust; update import 109 vs 165–184 ms; snapshot 214,928 vs 309,780 B; heap after apply 91.3 vs 107.0 MB.
• bench:complexity 1000,2000,4000,8000: no entry grows with size, including the new text-concurrent-insert-after-long-run (0.15–0.21 ms from 64k to 512k characters; the review head took 2.3 → 12.0 ms) and text-scattered-edits-checkout (1.0 → 4.2 ms; main 13 → 698 ms). Median at 8k vs main: 0.91 (0.97 over entries above 1 ms).

Not included / known gaps

• snapshotCheckout is off by default here (fix(loro.js): keep rich-text style anchors in the text sequence #1137 turns full-snapshot checkout on). Checkout into the retained range of a shallow snapshot that loro.js exported diverges, in plain text too: loro.js uses the requested frontiers as the root, while Rust moves it back to a critical version (#calculateShallowStart). The suite samples only a few versions per export (15 of 80 seeds); an exhaustive check in the second review found most multi-peer histories affected (8,484/17,746 non-critical-root versions on main, 10,980/17,746 here; 0 with a critical root). It is pre-existing, and fix(loro.js): root shallow snapshots at the latest critical version #1144 roots shallow snapshots at the latest critical version.
• Op shape (N4): loro.js merges adjacent deletes in one transaction over a narrower range than Rust (#mergeTrailingTextDeletes), so the same calls can write more delete ops. State and IDs of inserted text are the same; fix(loro.js): record absent-key map deletes and zero counter increments like loro-crdt #1145 works on local op parity.
• diff() from a checked-out version can throw duplicate sequence id on a document imported from a snapshot (it replays elements the state already holds). It is pre-existing on main (e.g. with a List); fix(loro.js): keep rich-text style anchors in the text sequence #1137 fixes it together with the anchor case.
• Undo parity: loro.js's UndoManager does not transform against remote edits like Rust's; the suite runs undo in loro.js and checks that Rust reads the resulting ops identically (undoParity compares both, off by default).

Merge order / rebase plan

• Merge order: fix(loro.js): stop a concurrent insert at the end of its origin's subtree #1139 → this PR → fix(loro.js): keep rich-text style anchors in the text sequence #1137. This branch contains fix(loro.js): stop a concurrent insert at the end of its origin's subtree #1139's commit, so its diff shrinks once fix(loro.js): stop a concurrent insert at the end of its origin's subtree #1139 lands.
• Base: main, merged in at ce6498b5 (fix(loro.js): correct checkout after a snapshot import #1126–fix(loro.js): drop containers deleted before the shallow root, keep tree node metadata #1128 and fix: resolve a container's parent from the change that created it #1159). The only conflict was the SequenceElementSet that fix(loro.js): correct checkout after a snapshot import #1126 and this PR both added; main's copy is kept. fix(loro.js): include reattached child containers' state in diff() and revertTo() #1131 still conflicts textually (resolved in an earlier trial merge).
• fix(loro.js): keep rich-text style anchors in the text sequence #1137 is stacked on this branch.

🤖 Generated with Claude Code

[github-actions]

WASM Size Report

• Original size: 3277.59 KB
• Gzipped size: 1089.33 KB
• Brotli size: 762.83 KB

[zxch3n]

Review follow-up (head b1e4bade, merged with main including #1135; the differential reference is now a nodejs WASM build of main 27982a5).

P2 Fugue subtree scan: fugueSubtreeEnd no longer visits every scalar after the last sibling. An element right after its ID predecessor always has that predecessor as origin-left (the op was placed right after its origin-left in its causal view, the predecessor is in that view, and only concurrent elements can sit between an element and its origin-left), so only the first element of each physical ID run can end the subtree. SequenceIndex.nextPhysicalIdRunStart finds run starts in expected O(log n) from the treap's existing idRunCount aggregates, so the walk costs O((runs in the subtree + 1) log n). The removed forEachPhysicalFrom has no other users.

• Review probe (warm origin index, one concurrent character after a 512k run), 1-minute load 5–25, interleaved: main 0.20–0.45 ms, previous head 8.9–14.8 ms, now 0.23–0.32 ms.
• New bench:complexity entry text-concurrent-insert-after-long-run (run of 64 × size characters): at 64k/128k/256k/512k characters, previous head 2.39/3.72/7.61/12.79 ms, now 0.21/0.21/0.21/0.15 ms (main 0.16–0.37 ms).
• Tests: randomized nextPhysicalIdRunStart vs. a brute-force reference (tests/indexes.test.ts, fails when the right subtree gets a wrong predecessor), and a Rust-compared scenario where the sibling subtree has several ID runs from two peers (tests/text-interop.test.ts).

Compatibility (B2), decided as option A: loro.js reads all data with Rust's semantics and no version marker.

• Changeset → major, with the concurrent-order and cursor changes marked breaking.
• README "Upgrading from 0.2" gives measured readings of data from the published loro.js@0.2.1: updates/JSON read béa9; full and shallow snapshots keep bé9a as current state, but their history reads like Rust. Snapshot-loaded and update-loaded replicas stay different after new edits (béQ9aZ vs béQa9Z). Encoded 0.2 cursors move (ab😀: 4 → 2).
• Migration paths: rebuild from the final state, or import old updates everywhere.
• Decision and reasons: context/loro-js-rust-differential.md. tests/legacy-data.test.ts pins the readings of 0.2.1 fixtures in loro.js and loro-crdt.

Differential suite: checkout now visits two versions in a row, each a commit version or the version right after a random op (mid-change/mid-op versions). 150 plain-text seeds × 120 steps pass against main's WASM. The stale notes about pre-#1135 Rust bugs are updated.

Validation: pnpm check (21 files / 301 tests); B4 and bench:complexity interleaved with main and the previous head: B4 apply main 239–265 ms, previous 266–414 ms, now 240–313 ms (loaded runs); update export 71–76 → 9–17 ms (1,153,540 → 274,574 B); snapshot 309,780 → 214,928 B; heap after apply 107.0 → 91.2 MB. bench:complexity at 8k: median now/previous 0.87, and no entry grows with size.

[zxch3n]

Follow-up: this PR now uses #1139's Fugue fix (head 67361973 merges #1139's 829db4fe).

#1139 and the first version of this PR fixed the same misordering: after the last sibling, the insert jumped to the origin-right bound. The two computed the same boundary in different ways:

• Here: walk the physical ID runs of the last sibling's subtree.
• fix(loro.js): stop a concurrent insert at the end of its origin's subtree #1139: binary-search where originLeft's subtree ends, using a descent test that skips implicit runs.

I compared them on a concurrent insert that must stop at the end of a sibling subtree, with a later concurrent root element (Node 22, 1-minute load 8–17). They give identical text in every case (hash-equal on the B4 trace); main misorders.

Sibling subtree shape	run walk (previous head)	#1139 binary search
One typed run, 16k	0.68 ms	0.52 ms
B4 trace (260k actions)	11.1 ms	7.3 ms
16k random-position inserts	6.1 ms	5.3 ms
Deep chain of explicit links (two positions typed alternately), 4k / 16k / 64k	1.1 / 4.2 / 15.5 ms	2.1 / 10.7 / 60.6 ms

#1139 is faster on long runs and on the real trace, and originLeft = undefined skips the search entirely. It's slower on deep explicit chains: each of its O(log n) probes walks the chain. That shape is synthetic, so I kept #1139's implementation and dropped fugueSubtreeEnd / SequenceIndex.nextPhysicalIdRunStart (and its unit test). The trade-off is recorded in context/loro-js-performance.md. tests/text-interop.test.ts keeps the Rust-compared multi-run case, and bench:complexity keeps text-concurrent-insert-after-long-run.

Merge order: #1139 → this PR → #1137. If #1139 lands first, this PR's diff loses those lines. The changeset keeps the ordering change as breaking (major; reading 0.2 data, option A) and points to #1139; #1139's own changeset is a patch.

Also still open: #1126 and this PR both add an identical SequenceElementSet to document.ts; whichever lands second keeps one.

Validation:

• pnpm check: 21 files, 304 tests (includes fix(loro.js): stop a concurrent insert at the end of its origin's subtree #1139's fugue-index.test.ts).
• 150 plain-text differential seeds × 120 steps with per-op checkouts, against the main WASM build.

Benchmark rerun (main / #1136 review head b58a8103 / now), interleaved (Node 22, fresh processes, order reversed in round 2; waited 120 s for 1-minute load < 8, round 1 ran at 7.4–8.1 and round 2 at 10–17)

	main	review head b58a8103	now c14f3164
Concurrent char after a 512k run (review probe)	0.29–0.30 ms	9.7–10.7 ms	0.28–0.31 ms
text-concurrent-insert-after-long-run, 64k → 512k	0.17–0.30 ms	2.30 → 11.97 ms	0.18–0.30 ms
B4 apply	246–255 ms	254–258 ms	248–252 ms
B4 update export	72–76 ms, 1,153,540 B	9–10 ms, 274,574 B	9–10 ms, 274,574 B
B4 update import	262–270 ms	116–120 ms	116–120 ms
B4 snapshot export / import	114–121 / 126–127 ms, 309,780 B	45–47 / 92 ms, 214,928 B	44–47 / 89 ms, 214,928 B
Heap after B4 apply	106.9 MB	91.2 MB	91.2 MB

bench:complexity at 8k: median now/review head 0.94, now/main 1.00; no entry grows with size. context/loro-js-performance.md records these numbers and the trade-off above.

[zxch3n]

Merged the latest main (ce6498b5: #1126–#1128 and #1159). Head is now 981c12e6.

Integration

• The only conflict was SequenceElementSet, which fix(loro.js): correct checkout after a snapshot import #1126 and this PR each added with the same behavior. I kept main's copy (fix(loro.js): correct checkout after a snapshot import #1126) and deleted this PR's.

Checkout performance fix (new)
While measuring first checkouts, I found that #canTransitionRecords walked the whole sequence once per insert op. That makes a checkout across scattered edits O(ops × elements). It's already on main, for plain Text and List too.

• Now each container is checked once with all its runs (containsIdRuns), and IDs are read without building element views.
• New bench:complexity entry text-scattered-edits-checkout:
  • main: 13 → 39 → 153 → 698 ms at 1k–8k;
  • now: 1.0 → 1.3 → 2.7 → 4.2 ms.

0.2 snapshots after #1126:

• A checkout keeps the 0.2 state (fix(loro.js): correct checkout after a snapshot import #1126's unreplayable fallback). Older versions it shows are approximate.
• A fork or shallow export replays the history, giving Rust's reading. Rust's own fork keeps the state.
• README "Upgrading from 0.2" is updated. tests/legacy-data.test.ts pins the fork difference.

Validation

• pnpm check: 24 files, 368 tests.
• 300 plain-text differential seeds × 120 steps (13001–13150, 14001–14150) with per-op checkouts, against a WASM build of main ce6498b5.
• With snapshotCheckout on, 15 of 80 seeds still fail. All are "shallow snapshot from js: checkout", the known shallow-export gap.

Performance (Node 22, alternating main / this PR / #1137, 1-minute load 6–9):

	main	this PR
B4 apply	238–240 ms	233–235 ms
B4 update export / import	72–73 / 165–184 ms	9 / 109 ms
B4 update / snapshot bytes	1,153,540 / 309,780 B	274,574 / 214,928 B
Heap after B4 apply	107.0 MB	91.3 MB
text-scattered-edits-checkout, 8k	698 ms	4.2 ms
Concurrent char after a 512k run	0.36–0.43 ms	0.22–0.24 ms

bench:complexity at 8k: median this PR / main 0.91 (0.97 over entries above 1 ms). No entry grows with size.

The per-round numbers are in context/loro-js-performance.md.

[zxch3n]

Second review, #1136 part.

• D1/D2: fixed in eabcddf.
  • Cursors: side-1 and end-of-text cursors encoded by 0.2 resolve one character earlier; the others are unchanged, whatever the characters.
  • Migration path 1 is only valid when the 0.2 snapshot is the final version, and a shallow-snapshot export switches the document to Rust's reading.
  • The README, changeset, context, and this PR description are aligned.
• Harness, fixed in 84431ba. The second checkout version came from imul(roll, 0x9e3779b1), whose low bits repeat the roll's, so it usually revisited the first version. It now uses the high bits. pnpm --dir loro-js check with the Rust reference: 368 tests. The differential suite passes 200 seeds × 120 steps.
• N3 and N4 are in known gaps with the corrected numbers, linking fix(loro.js): root shallow snapshots at the latest critical version #1144 and fix(loro.js): record absent-key map deletes and zero counter increments like loro-crdt #1145.
• The pre-existing diff() duplicate sequence id on snapshot-imported documents is listed as fixed by fix(loro.js): keep rich-text style anchors in the text sequence #1137.

🤖 Generated with Claude Code