From b58a81031f44941291c89f9a3945b3344fc8b317 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 03:42:48 +0800 Subject: [PATCH 01/25] fix(loro.js): converge plain-text edits with Rust A seeded differential suite now runs the same Text edits, syncs, checkouts, snapshots, cursors, undo, and revertTo in loro.js and loro-crdt (nodejs WASM) and compares every step; it skips when no WASM build exists. It found these divergences, fixed here: Fugue placement after a sibling subtree, deletes applied by recorded ID instead of position, doubly deleted text restored twice by checkout events, and cursor semantics. Ops are also written in Rust's shape (merged inserts, last delete run first). See context/loro-js-rust-differential.md and context/loro-js-performance.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- .changeset/loro-js-text-interop.md | 11 + AGENTS.md | 2 + context/CONTEXT-GAPS.md | 1 - context/loro-js-performance.md | 13 +- context/loro-js-rust-differential.md | 79 ++ loro-js/AGENTS.md | 30 + loro-js/CLAUDE.md | 1 + loro-js/README.md | 3 + loro-js/src/runtime/containers.ts | 119 ++- loro-js/src/runtime/document.ts | 142 ++- loro-js/src/runtime/sequence-index.ts | 47 + .../fixtures/rust/text-delete-positions.blob | Bin 0 -> 108 bytes .../fixtures/rust/text-delete-positions.json | 49 + loro-js/tests/richtext-differential.test.ts | 22 + .../tests/support/richtext-differential.ts | 981 ++++++++++++++++++ loro-js/tests/support/rust-reference.ts | 27 + loro-js/tests/text-interop.test.ts | 170 +++ package.json | 3 +- 18 files changed, 1661 insertions(+), 39 deletions(-) create mode 100644 .changeset/loro-js-text-interop.md create mode 100644 context/loro-js-rust-differential.md create mode 100644 loro-js/AGENTS.md create mode 120000 loro-js/CLAUDE.md create mode 100644 loro-js/tests/fixtures/rust/text-delete-positions.blob create mode 100644 loro-js/tests/fixtures/rust/text-delete-positions.json create mode 100644 loro-js/tests/richtext-differential.test.ts create mode 100644 loro-js/tests/support/richtext-differential.ts create mode 100644 loro-js/tests/support/rust-reference.ts create mode 100644 loro-js/tests/text-interop.test.ts diff --git a/.changeset/loro-js-text-interop.md b/.changeset/loro-js-text-interop.md new file mode 100644 index 000000000..ec51a6bc3 --- /dev/null +++ b/.changeset/loro-js-text-interop.md @@ -0,0 +1,11 @@ +--- +"loro.js": patch +--- + +Text edits now converge with Rust (`loro-crdt`) in cases a randomized comparison with the Rust WASM build found: + +- A concurrent insert after a sibling's subtree was placed after later, unrelated concurrent text. Replicas that merged such edits in different orders could end with different text. +- An imported delete is now applied by its position, as Rust does, instead of by its recorded `start_id`. Rust's WASM build records a wrong `start_id` when the deleted text spans astral characters (for example, emoji) inserted in separate edits, so loro.js deleted different characters than Rust. +- `checkout`, `diff`, and `revertTo` no longer restore a character twice when two concurrent deletes removed it. +- Text cursors follow Rust: a cursor at the end or in empty text has no target, `getCursorPos` returns the target's own offset for every side, and a deleted target reports the length of the text before it with side `-1`. The encoded cursor matches Rust's bytes. +- Consecutive inserts in one transaction are merged into one op, and a delete that spans several ID runs writes the last run first, as Rust does. diff --git a/AGENTS.md b/AGENTS.md index cb00f0dd4..beb65f99b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,6 +31,8 @@ Loro is a Rust CRDT workspace with JS/WASM packaging and a MoonBit codec. [skills/loro/SKILL.md](skills/loro/SKILL.md). - Pure TypeScript runtime indexes, complexity contracts, benchmarks, and remaining gaps: [context/loro-js-performance.md](context/loro-js-performance.md). +- loro.js vs Rust randomized Text differential tests and the Rust behaviors they + work around: [context/loro-js-rust-differential.md](context/loro-js-rust-differential.md). - WASM panic/OOM reporting channels, `__wbindgen_start` glue invariant, and trap-testing recipes: [context/wasm-error-reporting.md](context/wasm-error-reporting.md). diff --git a/context/CONTEXT-GAPS.md b/context/CONTEXT-GAPS.md index debfca641..9b92d88f0 100644 --- a/context/CONTEXT-GAPS.md +++ b/context/CONTEXT-GAPS.md @@ -5,4 +5,3 @@ fix the docs in that change. Format: `YYYY-MM-DD | | | why it was hard | suggested home` -2026-09-21 | How do I test that loro.js output imports into Rust/loro-crdt? | `loro-js/scripts/rewrite-rust-fixture.mjs` (`pnpm -C loro-js fixtures:rewrite`) writes `loro-js/tests/fixtures/rust/*.ts.blob`, consumed by `crates/loro/tests/loro_js_interop.rs`; loro.js state encoders live in `loro-js/src/runtime/document.ts` (`#containerState`) and must follow `docs/encoding-container-states.md` (tree sibling order: loro-dev/loro#1088) | root AGENTS only links loro.js performance context; the interop path is mentioned only in `loro-js/README.md` | a `loro-js/AGENTS.md` or a loro.js interop context article diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 896482166..38a37474a 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -1,6 +1,6 @@ # loro-js Performance Architecture -Verified against code 2026-07-22. +Verified against code 2026-09-28. The pure TypeScript runtime lives in `loro-js/src/runtime`. Its performance target is the asymptotic behavior of the Rust runtime, while accepting a larger @@ -103,7 +103,16 @@ JavaScript constant factor. keep their single-child edge implicit, and `SequenceIndex` can skip an entire future ID run while finding the next causally included element. Ordinary local edits keep the smaller unindexed path. MovableList continues to use the scan - because moves break the origin-tree physical preorder. + because moves break the origin-tree physical preorder. After the last sibling + the new element goes after that sibling's subtree (`fugueSubtreeEnd`), not at + the origin-right bound: when that bound is only the next causally included + element, later concurrent elements of an ancestor's subtree can sit before it. + The walk stops at the first element whose origin-left precedes the sibling. +- An imported Text delete is resolved by its position in the op's causal view, + like Rust's tracker (`LoroText._deleteTargets`): O(log n + runs) through + `visibleIdRuns` or the cached causal view. The recorded `start_id` is only a + fallback, because Rust's WASM build can record one that is off by the UTF-16 + length of astral text. Local deletes skip the lookup. - Merging adjacent changes appends only the new operations and key-table entries to the retained record. The cached operation length, peer end, frontier set, operation indexes, and subscriber update slice are updated incrementally, so diff --git a/context/loro-js-rust-differential.md b/context/loro-js-rust-differential.md new file mode 100644 index 000000000..898e19ce4 --- /dev/null +++ b/context/loro-js-rust-differential.md @@ -0,0 +1,79 @@ +# loro-js Differential Tests Against Rust + +Verified against code 2026-09-28. + +`loro-js/tests/richtext-differential.test.ts` runs random multi-peer Text +scenarios in loro.js and in Rust (`loro-crdt`, nodejs WASM build) side by side +and compares them after every step. It is the check for "does loro.js mean the +same thing as Rust" questions about Text positions, deletes, concurrency, +checkout, snapshots, cursors, and events. + +## Running it + +1. Build the reference: `pnpm -C crates/loro-wasm build-dev` (or + `build-release`). The tests load `crates/loro-wasm/nodejs/index.js`, or the + file named by `LORO_WASM_NODEJS`. Only the nodejs target is needed. +2. `pnpm --dir loro-js test` runs the suite; it is skipped when no build exists. + Set `LORO_REQUIRE_WASM_REFERENCE=1` to fail instead of skipping. +3. Scale with `LORO_DIFF_SEEDS` (default 40), `LORO_DIFF_STEPS` (default 80), + and `LORO_DIFF_FIRST_SEED`. 40 × 80 takes a few seconds. + +CI runs the loro.js suite with the reference required (`pnpm test-loro-js`, part +of `test-all` after `release-wasm`), so a Rust change that alters Text behavior +also shows up here. + +## What a scenario does + +`tests/support/richtext-differential.ts` generates plain-data actions from a +seed (`generateActions`), so a failure replays exactly. Each peer exists once +per runtime with the same peer ID. Actions: + +- `insert`/`delete` at UTF-16 or UTF-8 boundaries (BMP, CJK, astral, and + multi-scalar emoji), `mark`/`unmark` with before/after/both/none styles; +- `sync` by binary or JSON updates, within a runtime or crossed (Rust imports + the loro.js peer's updates and loro.js imports Rust's), so each side reads + the other's positions; +- `checkout` to a recorded version and back; +- full and shallow snapshot export from either runtime, imported into both; +- cursor creation, then cursor resolution after every later step; +- `revertTo` and undo/redo. + +After each step it compares the delta (normalized: merged runs, `null` +attributes dropped), UTF-16/Unicode/UTF-8 lengths, cursor positions, and the +atoms of the JSON history (every inserted scalar and mark with its entity +position; Rust groups atoms into ops depending on its encoding state). loro.js +events must rebuild the loro.js state; Rust events are not checked (see below). + +`revertTo` must produce the same state on forks of both runtimes, but its ops +may differ, so the replica runs it in one runtime and the other imports the +change. Undo/redo run in loro.js and Rust imports the result: loro.js's +`UndoManager` does not transform against remote edits like Rust's, so exact undo +parity is only an option (`undoParity`). + +## Shrinking a failure + +`shrinkActions(rust, actions, options)` removes actions while the scenario +still fails with the same first-line message (numbers ignored). A throwaway +Vitest file that calls `generateActions`, `runActions`, and `shrinkActions` +and prints the result is the quickest loop; paste the shrunk action list into a +regression test. + +## Rust behaviors the harness works around + +The Rust reference has bugs of its own in UTF-16 (WASM) builds: + +- A subscribed transaction that merges deletes of astral text can panic with + "Op/hint length mismatch" (`change_to_diff` in + `crates/loro-internal/src/txn.rs`), even after unsubscribing. The harness + subscribes only to loro.js documents. +- `get_text_entity_ranges` advanced the recorded delete `start_id` by the UTF-16 + length of astral text, so Rust can write a delete whose `start_id` names the + wrong elements. Rust applies deletes by position, and so does loro.js + (`LoroText._deleteTargets`). Rust still resolves a deleted cursor target + through the recorded IDs, so the harness skips cursors Rust cannot resolve. +- Rust's shallow import falls back to the recorded delete IDs when they disagree + with the positions, which can diverge from its own full-history import. The + harness compares loro.js with the source document instead. + +Snapshot-import checkout (`snapshotCheckout`) is off by default until loro.js +can check out a document imported from a snapshot (loro-dev/loro#1126). diff --git a/loro-js/AGENTS.md b/loro-js/AGENTS.md new file mode 100644 index 000000000..cd5b3661c --- /dev/null +++ b/loro-js/AGENTS.md @@ -0,0 +1,30 @@ +# loro.js + +Pure TypeScript implementation of Loro (`loro-js/src`). It must read and write +the same binary and JSON data as Rust (`loro-crdt`) and give the same results. + +## Context + +- Index structures, complexity contracts, and benchmarks: + [context/loro-js-performance.md](../context/loro-js-performance.md). +- Randomized Text comparison with Rust (build the nodejs WASM package first): + [context/loro-js-rust-differential.md](../context/loro-js-rust-differential.md). +- Encoding formats: [context/internal-encoding.md](../context/internal-encoding.md) + and `docs/encoding*.md`. + +## Interop fixtures + +- Rust output read by loro.js: `tests/fixtures/rust/*.blob` and `*.json`. +- loro.js output read by Rust: `pnpm fixtures:rewrite` runs + `scripts/rewrite-rust-fixture.mjs`, which writes `tests/fixtures/rust/*.ts.blob`; + `crates/loro/tests/loro_js_interop.rs` imports them. Container state encoders + live in `src/runtime/document.ts` (`#containerState`) and must follow + `docs/encoding-container-states.md`. + +## Commands + +From `loro-js/`: `pnpm test`, `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, +`pnpm build`, or `pnpm check` for all of them. From the root, `pnpm test-loro-js` +runs the tests and fails if the Rust WASM reference is missing (CI runs it after +`release-wasm`). Benchmarks: `pnpm bench:b4`, +`pnpm bench:complexity -- 1000,2000,4000,8000`. diff --git a/loro-js/CLAUDE.md b/loro-js/CLAUDE.md new file mode 120000 index 000000000..47dc3e3d8 --- /dev/null +++ b/loro-js/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/loro-js/README.md b/loro-js/README.md index 5c22ea5c4..6e037ad0e 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -94,6 +94,9 @@ The following paths are checked against the Rust implementation: - JSON update import/export and the postcard Awareness/EphemeralStore protocols. - Rust-produced fixtures imported by TypeScript and TypeScript-produced fixtures imported by Rust. +- Randomized multi-peer text edits run side by side with the `loro-crdt` WASM build + (`tests/richtext-differential.test.ts`): positions, deletes, concurrency, checkout, + snapshots, cursors, events, and `revertTo`. This is not yet a claim of complete behavioral equivalence with every `loro-crdt` edge case. Important current limits are: diff --git a/loro-js/src/runtime/containers.ts b/loro-js/src/runtime/containers.ts index 9b3979b91..caa37a646 100644 --- a/loro-js/src/runtime/containers.ts +++ b/loro-js/src/runtime/containers.ts @@ -1722,19 +1722,25 @@ export class LoroText extends LoroContainer { return this._sequence.metricOffsetAtVisibleIndex(unicodeIndex, to); } + /** + * Matches Rust: an empty text or a position at the end yields an ID-less + * cursor (left side when empty, right side at the end); otherwise the cursor + * names the scalar that starts at the UTF-16 position. The encoded origin is + * Rust's too: the Unicode position, or the UTF-16 length at the end. + */ getCursor(pos: number, side: Side = 0): Cursor | undefined { if (!Number.isSafeInteger(pos) || pos < 0) return undefined; - if (pos >= this.length) { - return new Cursor( - this.id, - this._sequence.atVisible(this._sequence.visibleLength - 1)?.id, - 1, - this.length, - ); - } + const length = this.length; + if (length === 0) return new Cursor(this.id, undefined, side === 0 ? -1 : side, 0); + if (pos >= length) return new Cursor(this.id, undefined, 1, length); const unicodePosition = this.convertPos(pos, "utf16", "unicode"); if (unicodePosition === undefined) return undefined; - return new Cursor(this.id, this._sequence.atVisible(unicodePosition)!.id, side, pos); + return new Cursor( + this.id, + this._sequence.atVisible(unicodePosition)!.id, + side, + unicodePosition, + ); } getEditorOf(pos: number): string | undefined { @@ -1878,6 +1884,64 @@ export class LoroText extends LoroContainer { this._sequence.deleteIdSpan(startId, length, deletedBy); } + /** + * Resolves the elements a delete operation removes. Rust's tracker deletes the + * visible elements at `position` in the operation's causal view and uses + * `startId` only to name them, so the position wins when the two disagree + * (Rust's WASM build can record a `startId` that is off by the UTF-16 length + * of astral characters). The recorded IDs are used only when the position + * range does not fit the causal view. + */ + _deleteTargets( + position: number, + length: number, + startId: CodecId, + causalVersion: CausalVersion, + ): SequenceIdRun[] { + const recorded = [{ start: startId, length }]; + const current = this._sequence.isFullyIncluded(causalVersion); + if (current && length === 1) { + const element = this._sequence.atVisible(position); + if ( + element === undefined || + (element.id.peer === startId.peer && element.id.counter === startId.counter) + ) { + return recorded; + } + return [{ start: element.id, length: 1 }]; + } + const runs = current + ? this._sequence.visibleIdRuns(position, position + length) + : this._sequence.causalView(causalVersion).idRuns(position, position + length); + let resolved = 0; + for (const run of runs) resolved += run.length; + return resolved === length ? runs : recorded; + } + + /** + * Deletes resolved target runs. Operation counters follow Rust: a forward + * delete's k-th counter removes the k-th target, a reversed delete's k-th + * counter removes the k-th target from the right. + */ + _deleteTargetRuns( + runs: readonly SequenceIdRun[], + reversed: boolean, + deletedBy: CodecId, + ): void { + const total = runs.reduce((sum, run) => sum + run.length, 0); + let offset = 0; + for (const run of runs) { + const counter = reversed + ? deletedBy.counter + total - offset - run.length + : deletedBy.counter + offset; + this._deleteIdSpan(run.start, reversed ? -run.length : run.length, { + peer: deletedBy.peer, + counter, + }); + offset += run.length; + } + } + _applyMark( start: number, end: number, @@ -3054,7 +3118,9 @@ function indexedFugueInsertion( } if (!scanning) { - insertIndex = candidates[childIndex + 1]?.index ?? originRightIndex; + insertIndex = + candidates[childIndex + 1]?.index ?? + fugueSubtreeEnd(sequence, candidates[childIndex]!.index, originRightIndex); } } @@ -3066,6 +3132,39 @@ function indexedFugueInsertion( }; } +/** + * Returns the physical index right after the Fugue subtree rooted at + * `rootIndex`, bounded by `limit`. Elements between the last sibling and the + * origin-right bound need not descend from that sibling: when the bound is only + * the next causally included element, later concurrent elements that belong to + * an ancestor's subtree can sit before it. An element is a descendant exactly + * when its origin-left lies inside the scanned subtree. + */ +function fugueSubtreeEnd( + sequence: SequenceIndex, + rootIndex: number, + limit: number, +): number { + let end = rootIndex + 1; + if (end >= limit) return limit; + let previous = sequence.atPhysicalRaw(rootIndex)!.id; + sequence.forEachPhysicalFrom(end, (element, index) => { + if (index >= limit) return false; + const left = element.originLeft; + if (left === undefined) return false; + if (left.peer !== previous.peer || left.counter !== previous.counter) { + const leftElement = sequence.findByIdRaw(left); + const leftIndex = + leftElement === undefined ? undefined : sequence.physicalIndexOf(leftElement); + if (leftIndex === undefined || leftIndex < rootIndex) return false; + } + previous = element.id; + end = index + 1; + return undefined; + }); + return end; +} + function getFugueOriginIndex( sequence: SequenceIndex, ): FugueOriginIndex { diff --git a/loro-js/src/runtime/document.ts b/loro-js/src/runtime/document.ts index 57f9a8a4d..a28d1d17f 100644 --- a/loro-js/src/runtime/document.ts +++ b/loro-js/src/runtime/document.ts @@ -1951,6 +1951,7 @@ export class LoroDoc = Record = Record true)) return undefined; + const offset = text._sequence.visibleMetricOffsetOf(target, "utf16") ?? 0; + const update = text.getCursor(offset, -1); + return update === undefined ? { offset, side: -1 } : { offset, side: -1, update }; + } + _subscribeContainer( container: LoroContainer, listener: (event: LoroEventBatch) => void, @@ -2086,7 +2117,14 @@ export class LoroDoc = Record = Record = Record = Record = Record ({ position: start, length: end - start })); } + #textDeleteTargets( + text: LoroText, + content: Extract, + causalVersion: CausalVersion, + ): SequenceIdRun[] { + const length = Number(content.length); + // A reversed (backspace) span stores its rightmost position. + return text._deleteTargets( + length < 0 ? content.position + length + 1 : content.position, + Math.abs(length), + content.startId, + causalVersion, + ); + } + #recordedEventDiffs( recording: EventRecording, includeMapTombstones = false, @@ -3132,12 +3179,18 @@ export class LoroDoc = Record= 0; index -= 1) { + const run = runs[index]!; + end -= run.length; this.#appendAndApply( container, { type: "text-delete", - position, + position: end, length: BigInt(run.length), startId: run.start, }, @@ -3288,6 +3341,7 @@ export class LoroDoc = Record = Record = Record>(); + // Keyed by ID: packed text spans hand out a fresh element view per lookup, + // so an element deleted by two concurrent ops must not be collected twice. + const sequences = new Map(); const bulkSequenceRemovals = new Map(); const bulkSequenceRestorations = new Map(); const textAttributeRuns = new Map>(); @@ -3875,10 +3933,10 @@ export class LoroDoc = Record>(); const treeSubjects = new Map>(); const counters = new Map(); - const sequenceElements = (container: LoroList | LoroText): Set => { + const sequenceElements = (container: LoroList | LoroText): SequenceElementSet => { let elements = sequences.get(container); if (elements === undefined) { - elements = new Set(); + elements = new SequenceElementSet(); sequences.set(container, elements); } return elements; @@ -6533,6 +6591,26 @@ function appendToTrailingListInsert( operation: DecodedOperation, ): boolean { const content = operation.content; + if (content.type === "text-insert") { + // Rust merges a text insert that continues the previous one in the change. + const previousIndex = operations.length - 1; + const previous = operations[previousIndex]; + if ( + previous === undefined || + previous.content.type !== "text-insert" || + !containerIdsEqual(previous.container, operation.container) || + previous.counter + previous.length !== operation.counter || + previous.content.position + previous.length !== content.position + ) { + return false; + } + operations[previousIndex] = { + ...previous, + length: previous.length + operation.length, + content: { ...previous.content, value: previous.content.value + content.value }, + }; + return true; + } if (content.type !== "list-insert" && content.type !== "movable-list-insert") { return false; } @@ -8128,6 +8206,20 @@ function eventValuesEqual(left: unknown, right: unknown): boolean { return false; } +/** A set of sequence elements keyed by ID rather than by element identity. */ +class SequenceElementSet implements Iterable { + readonly #elements = new Map(); + + add(element: SequenceElement): void { + const key = idKey(element.id); + if (!this.#elements.has(key)) this.#elements.set(key, element); + } + + [Symbol.iterator](): Iterator { + return this.#elements.values(); + } +} + function subtractSequenceIdRuns( runs: readonly SequenceIdRun[], removedRuns: readonly SequenceIdRun[], diff --git a/loro-js/src/runtime/sequence-index.ts b/loro-js/src/runtime/sequence-index.ts index df09a73eb..9d7d92290 100644 --- a/loro-js/src/runtime/sequence-index.ts +++ b/loro-js/src/runtime/sequence-index.ts @@ -423,6 +423,53 @@ export class SequenceIndex { }); } + /** Visits elements in physical order from `start` until `visit` returns false. */ + forEachPhysicalFrom( + start: number, + visit: (element: T, index: number) => boolean | void, + ): void { + if (!Number.isSafeInteger(start) || start < 0 || start >= this.allLength) return; + const stack: SequenceNode[] = []; + let node = this.#root; + let base = 0; + let current: SequenceNode | undefined; + let offset = 0; + while (node !== undefined) { + const leftCount = allCount(node.left); + if (start < base + leftCount) { + stack.push(node); + node = node.left; + continue; + } + const ownStart = base + leftCount; + if (start < ownStart + nodeLength(node)) { + current = node; + offset = start - ownStart; + break; + } + base = ownStart + nodeLength(node); + node = node.right; + } + let index = start; + while (current !== undefined) { + for (; offset < nodeLength(current); offset += 1) { + if (visit(nodeElement(current, offset), index) === false) return; + index += 1; + } + let next = current.right; + if (next === undefined) { + current = stack.pop(); + } else { + while (next.left !== undefined) { + stack.push(next); + next = next.left; + } + current = next; + } + offset = 0; + } + } + findNextIncludedPhysical( start: number, version: ReadonlyMap, diff --git a/loro-js/tests/fixtures/rust/text-delete-positions.blob b/loro-js/tests/fixtures/rust/text-delete-positions.blob new file mode 100644 index 0000000000000000000000000000000000000000..4fd66bff8494a9d53b33918791f9fa399049c598 GIT binary patch literal 108 zcmd1FFUn^?0zN%!Wf@pP8Q2-v83h>Gp)!n&UEs literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/rust/text-delete-positions.json b/loro-js/tests/fixtures/rust/text-delete-positions.json new file mode 100644 index 000000000..1e952d15e --- /dev/null +++ b/loro-js/tests/fixtures/rust/text-delete-positions.json @@ -0,0 +1,49 @@ +{ + "text": "b", + "updates": { + "schema_version": 1, + "start_version": {}, + "peers": [ + "7" + ], + "changes": [ + { + "id": "0@0", + "timestamp": 0, + "deps": [], + "lamport": 0, + "msg": null, + "ops": [ + { + "container": "cid:root-text:Text", + "content": { + "type": "insert", + "pos": 0, + "text": "a😀b" + }, + "counter": 0 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "insert", + "pos": 2, + "text": "x" + }, + "counter": 3 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "delete", + "pos": 0, + "len": 3, + "start_id": "0@0" + }, + "counter": 4 + } + ] + } + ] + } +} diff --git a/loro-js/tests/richtext-differential.test.ts b/loro-js/tests/richtext-differential.test.ts new file mode 100644 index 000000000..3c84c83c2 --- /dev/null +++ b/loro-js/tests/richtext-differential.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, test } from "vitest"; + +import { generateActions, runActions } from "./support/richtext-differential"; +import { loadRustReference } from "./support/rust-reference"; + +const rust = loadRustReference(); +const seeds = Number(process.env.LORO_DIFF_SEEDS ?? 40); +const steps = Number(process.env.LORO_DIFF_STEPS ?? 80); +const firstSeed = Number(process.env.LORO_DIFF_FIRST_SEED ?? 1); +// Marks need loro.js to keep Rust's style anchors in the text sequence. +const options = { marks: false }; + +describe.skipIf(rust === undefined)("text differential against Rust", () => { + test.each(Array.from({ length: seeds }, (_, index) => firstSeed + index))( + "random concurrent text edits converge with Rust (seed %i)", + (seed) => { + expect(runActions(rust!, generateActions(seed, steps, options), options)).toBe( + undefined, + ); + }, + ); +}); diff --git a/loro-js/tests/support/richtext-differential.ts b/loro-js/tests/support/richtext-differential.ts new file mode 100644 index 000000000..47d0a75dd --- /dev/null +++ b/loro-js/tests/support/richtext-differential.ts @@ -0,0 +1,981 @@ +import * as Js from "../../src/index"; +import type { Delta, Frontiers, LoroEventBatch, Value } from "../../src/index"; +import type { RustReference } from "./rust-reference"; + +/** + * Differential rich-text scenarios between loro.js and the Rust implementation + * (`loro-crdt`). Every peer exists once per runtime; each action runs the same + * public API call on both copies and then compares text, delta, attributes, + * event-reconstructed state, and cursors. Syncs randomly move binary or JSON + * updates within a runtime or across runtimes, so positions written by one + * implementation are interpreted by the other. + * + * Actions are plain data with unresolved positions, so a failing sequence can be + * replayed, shrunk (`shrinkActions`), and pasted into a regression test. + */ + +type Runtime = "rust" | "js"; +type Doc = Js.LoroDoc; +type Text = Js.LoroText; +type UndoManager = Js.UndoManager; +type Cursor = Js.Cursor; + +export const STYLE_CONFIG = { + bold: { expand: "after" }, + link: { expand: "none" }, + em: { expand: "before" }, + hl: { expand: "both" }, +} as const; +const STYLE_KEYS = Object.keys(STYLE_CONFIG); +const STYLE_VALUES: Value[] = [true, false, 1, 2, "x", "y", { a: 1 }]; +const TEXTS = [ + "a", + "b", + "cd", + "xyz", + "é", + "中文", + "😀", + "a😀b", + "👍🏽", + "\n", + "ab\ncd", + "𝒳y", +]; + +export type Action = + | { + readonly type: "insert"; + readonly peer: number; + readonly pos: number; + readonly text: string; + readonly utf8?: boolean; + } + | { + readonly type: "delete"; + readonly peer: number; + readonly pos: number; + readonly len: number; + readonly utf8?: boolean; + } + | { + readonly type: "mark"; + readonly peer: number; + readonly pos: number; + readonly len: number; + readonly key: string; + readonly value: Value; + } + | { + readonly type: "unmark"; + readonly peer: number; + readonly pos: number; + readonly len: number; + readonly key: string; + } + | { readonly type: "commit"; readonly peer: number } + | { + readonly type: "sync"; + readonly from: number; + readonly to: number; + readonly json: boolean; + readonly cross: boolean; + } + | { readonly type: "checkout"; readonly peer: number; readonly version: number } + | { readonly type: "undo" | "redo"; readonly peer: number } + | { + readonly type: "revert"; + readonly peer: number; + readonly version: number; + readonly inRust: boolean; + } + | { + readonly type: "snapshot"; + readonly peer: number; + readonly fromRust: boolean; + readonly version: number; + } + | { + readonly type: "shallow"; + readonly peer: number; + readonly fromRust: boolean; + readonly version: number; + } + | { + readonly type: "cursor"; + readonly peer: number; + readonly pos: number; + readonly side: -1 | 0 | 1; + }; + +export interface ScenarioOptions { + readonly peers?: number; + readonly undo?: boolean; + readonly crossRuntimeSync?: boolean; + readonly snapshots?: boolean; + readonly snapshotCheckout?: boolean; + readonly checkout?: boolean; + readonly revert?: boolean; + readonly cursors?: boolean; + readonly events?: boolean; + readonly marks?: boolean; + readonly compareOps?: boolean; + /** + * Run undo/redo in both runtimes and compare, instead of running it in + * loro.js and importing the result into Rust. + */ + readonly undoParity?: boolean; +} + +const DEFAULT_OPTIONS: Required = { + peers: 3, + undo: true, + crossRuntimeSync: true, + snapshots: true, + // Checking out a document imported from a snapshot needs loro-dev/loro#1126. + snapshotCheckout: false, + checkout: true, + revert: true, + cursors: true, + events: true, + marks: true, + compareOps: true, + undoParity: false, +}; + +export class DifferentialFailure extends Error {} + +class Random { + #state: number; + + constructor(seed: number) { + this.#state = (seed * 2_654_435_761 + 0x9e37_79b9) >>> 0 || 1; + } + + next(): number { + let value = this.#state; + value ^= value << 13; + value ^= value >>> 17; + value ^= value << 5; + this.#state = value >>> 0; + return this.#state; + } + + int(bound: number): number { + return bound <= 0 ? 0 : this.next() % bound; + } + + pick(values: readonly T[]): T { + return values[this.int(values.length)]!; + } +} + +/** Generates a replayable action list. Positions are resolved at run time. */ +export function generateActions( + seed: number, + steps: number, + options: ScenarioOptions = {}, +): Action[] { + const config = { ...DEFAULT_OPTIONS, ...options }; + const random = new Random(seed); + const actions: Action[] = []; + const peerIds = Array.from({ length: config.peers }, (_, index) => index + 1); + const commitMaybe = (peer: number): void => { + if (random.int(5) !== 0) actions.push({ type: "commit", peer }); + }; + for (let step = 0; step < steps; step += 1) { + const peer = random.pick(peerIds); + const roll = random.int(100); + const pos = random.next(); + const len = 1 + random.int(6); + if (roll < 32) { + const text = random.int(4) === 0 ? `${step % 10}` : random.pick(TEXTS); + actions.push({ type: "insert", peer, pos, text, utf8: roll >= 26 }); + commitMaybe(peer); + } else if (roll < 48) { + actions.push({ type: "delete", peer, pos, len, utf8: roll >= 44 }); + commitMaybe(peer); + } else if (roll < 62 && config.marks) { + const key = random.pick(STYLE_KEYS); + const value = random.int(8) === 0 ? null : random.pick(STYLE_VALUES); + actions.push({ type: "mark", peer, pos, len, key, value }); + commitMaybe(peer); + } else if (roll < 68 && config.marks) { + actions.push({ type: "unmark", peer, pos, len, key: random.pick(STYLE_KEYS) }); + commitMaybe(peer); + } else if (roll < 80) { + const to = random.pick(peerIds.filter((id) => id !== peer)); + actions.push({ + type: "sync", + from: peer, + to, + json: random.int(2) === 0, + cross: config.crossRuntimeSync && random.int(2) === 0, + }); + } else if (roll < 84 && config.checkout) { + actions.push({ type: "checkout", peer, version: random.next() }); + } else if (roll < 88 && config.undo) { + actions.push({ type: random.int(3) === 0 ? "redo" : "undo", peer }); + } else if (roll < 90 && config.revert) { + actions.push({ + type: "revert", + peer, + version: random.next(), + inRust: random.int(2) === 0, + }); + } else if (roll < 93 && config.snapshots) { + actions.push({ + type: "snapshot", + peer, + fromRust: random.int(2) === 0, + version: random.next(), + }); + } else if (roll < 95 && config.snapshots) { + actions.push({ + type: "shallow", + peer, + fromRust: random.int(2) === 0, + version: random.next(), + }); + } else if (roll < 98 && config.cursors) { + actions.push({ type: "cursor", peer, pos, side: random.pick([-1, 0, 1] as const) }); + } else { + actions.push({ type: "commit", peer }); + } + } + return actions; +} + +interface Replica { + readonly runtime: Runtime; + readonly doc: Doc; + readonly shadow: EventShadow; + readonly undo: UndoManager | undefined; +} + +interface EventShadow { + delta: Delta[]; +} + +interface Peer { + readonly id: number; + readonly rust: Replica; + readonly js: Replica; + readonly versions: Frontiers[]; + readonly cursors: { rust: Cursor; js: Cursor }[]; +} + +export class RichtextScenario { + readonly #rust: typeof Js; + readonly #options: Required; + readonly #peers: Peer[] = []; + readonly #log: string[] = []; + + constructor(rust: RustReference, options: ScenarioOptions = {}) { + this.#rust = rust as unknown as typeof Js; + this.#options = { ...DEFAULT_OPTIONS, ...options }; + for (let index = 0; index < this.#options.peers; index += 1) { + const id = index + 1; + this.#peers.push({ + id, + rust: this.#replica("rust", id), + js: this.#replica("js", id), + versions: [], + cursors: [], + }); + } + } + + /** The replicas of a peer, for diagnostics. */ + replicas(peer: number): { rust: Doc; js: Doc } { + const selected = this.#peer(peer); + return { rust: selected.rust.doc, js: selected.js.doc }; + } + + run(actions: readonly Action[]): void { + for (const action of actions) this.#apply(action); + this.#syncAll(); + this.#compareAll("final sync"); + } + + #replica(runtime: Runtime, peer: number): Replica { + const module = runtime === "rust" ? this.#rust : Js; + const doc = new module.LoroDoc(); + doc.setPeerId(peer); + doc.configTextStyle(STYLE_CONFIG); + doc.getText("t"); + const shadow: EventShadow = { delta: [] }; + // Only loro.js events are checked: they must reproduce loro.js state, which + // is compared with Rust state. Rust itself panics ("Op/hint length + // mismatch", `txn.rs`) when a subscribed transaction merges deletes whose + // UTF-16 and Unicode lengths differ, even after unsubscribing. + if (this.#options.events && runtime === "js") { + doc.subscribe((batch: LoroEventBatch) => { + for (const event of batch.events) { + if (event.diff.type !== "text") continue; + shadow.delta = applyDelta(shadow.delta, event.diff.diff as Delta[]); + } + }); + } + const undo = + this.#options.undo && (runtime === "js" || this.#options.undoParity) + ? new module.UndoManager(doc, { mergeInterval: 0, maxUndoSteps: 100 }) + : undefined; + return { runtime, doc, shadow, undo }; + } + + #peer(id: number): Peer { + return this.#peers[(id - 1) % this.#peers.length]!; + } + + #apply(action: Action): void { + switch (action.type) { + case "insert": { + const peer = this.#peer(action.peer); + const position = pickBoundary(this.#currentText(peer), action.pos, action.utf8); + const text = action.text; + if (action.utf8) { + this.#both(peer, `insertUtf8(${position}, ${JSON.stringify(text)})`, (r) => + this.#text(r).insertUtf8(position, text), + ); + } else { + this.#both(peer, `insert(${position}, ${JSON.stringify(text)})`, (r) => + this.#text(r).insert(position, text), + ); + } + return; + } + case "delete": { + const peer = this.#peer(action.peer); + const range = pickRange( + this.#currentText(peer), + action.pos, + action.len, + action.utf8, + ); + if (range === undefined) return; + const [start, end] = range; + if (action.utf8) { + this.#both(peer, `deleteUtf8(${start}, ${end - start})`, (r) => + this.#text(r).deleteUtf8(start, end - start), + ); + } else { + this.#both(peer, `delete(${start}, ${end - start})`, (r) => + this.#text(r).delete(start, end - start), + ); + } + return; + } + case "mark": { + const peer = this.#peer(action.peer); + const range = pickRange(this.#currentText(peer), action.pos, action.len); + if (range === undefined) return; + const [start, end] = range; + const { key, value } = action; + this.#both( + peer, + `mark(${start}..${end}, ${key}, ${JSON.stringify(value)})`, + (r) => this.#text(r).mark({ start, end }, key, value), + ); + return; + } + case "unmark": { + const peer = this.#peer(action.peer); + const range = pickRange(this.#currentText(peer), action.pos, action.len); + if (range === undefined) return; + const [start, end] = range; + this.#both(peer, `unmark(${start}..${end}, ${action.key})`, (r) => + this.#text(r).unmark({ start, end }, action.key), + ); + return; + } + case "commit": + this.#commit(this.#peer(action.peer)); + return; + case "sync": + this.#sync(action); + return; + case "checkout": { + const peer = this.#peer(action.peer); + if (peer.versions.length === 0) return; + this.#commit(peer); + const version = peer.versions[action.version % peer.versions.length]!; + this.#both(peer, `checkout(${JSON.stringify(version)})`, (r) => + r.doc.checkout(version), + ); + this.#both(peer, "checkoutToLatest", (r) => r.doc.checkoutToLatest()); + return; + } + case "undo": + case "redo": { + // loro.js's undo is a separate implementation that does not produce + // Rust's ops (it does not transform against remote edits, for example). + // Run it in loro.js and check that Rust reads the resulting ops the same. + const peer = this.#peer(action.peer); + this.#commit(peer); + if (this.#options.undoParity) { + this.#both(peer, action.type, (replica) => { + if (action.type === "redo") replica.undo?.redo(); + else replica.undo?.undo(); + }); + this.#commit(peer); + return; + } + this.#oneSided(peer, peer.js, action.type, () => { + if (action.type === "redo") peer.js.undo?.redo(); + else peer.js.undo?.undo(); + }); + return; + } + case "revert": { + const peer = this.#peer(action.peer); + if (peer.versions.length === 0) return; + this.#commit(peer); + const version = peer.versions[action.version % peer.versions.length]!; + const label = `revertTo(${JSON.stringify(version)})`; + // The resulting state must match, but the ops need not: Rust applies its + // diff delta in document order, including where inserted text sits + // relative to deleted text, while loro.js composes its diff first. + const rustFork = peer.rust.doc.fork(); + const jsFork = peer.js.doc.fork(); + rustFork.revertTo(version); + jsFork.revertTo(version); + this.#expectDelta( + jsFork, + normalizeDelta(rustFork.getText("t").toDelta()), + `${label} on forks`, + ); + const actor = action.inRust ? peer.rust : peer.js; + this.#oneSided(peer, actor, label, () => actor.doc.revertTo(version)); + return; + } + case "snapshot": + this.#snapshotRoundTrip(action); + return; + case "shallow": + this.#shallowRoundTrip(action); + return; + case "cursor": + this.#createCursor(action); + return; + } + } + + #text(replica: Replica): Text { + return replica.doc.getText("t"); + } + + #currentText(peer: Peer): string { + return this.#text(peer.rust).toString(); + } + + #both(peer: Peer, label: string, action: (replica: Replica) => void): void { + this.#log.push(`peer ${peer.id}: ${label}`); + const errors: { runtime: Runtime; error: unknown }[] = []; + for (const replica of [peer.rust, peer.js]) { + try { + action(replica); + } catch (error) { + errors.push({ runtime: replica.runtime, error }); + } + } + if (errors.length === 1) { + this.#fail( + `${label}: only ${errors[0]!.runtime} threw: ${String(errors[0]!.error)}`, + ); + } + this.#compare(peer, label); + } + + /** + * Runs an edit on one replica only, then imports its change into the other + * replica of the same peer, so both keep identical histories. + */ + #oneSided(peer: Peer, actor: Replica, label: string, edit: () => void): void { + this.#log.push(`peer ${peer.id}: ${label} in ${actor.runtime}`); + try { + edit(); + actor.doc.commit(); + } catch (error) { + this.#fail(`${label} in ${actor.runtime} threw: ${String(error)}`); + } + const other = actor === peer.rust ? peer.js : peer.rust; + other.doc.import(actor.doc.export({ mode: "update" })); + this.#compare(peer, `${label} in ${actor.runtime}`); + peer.versions.push(peer.rust.doc.frontiers()); + } + + #commit(peer: Peer): void { + this.#both(peer, "commit", (replica) => replica.doc.commit()); + const version = peer.rust.doc.frontiers(); + const previous = peer.versions.at(-1); + if (previous === undefined || JSON.stringify(previous) !== JSON.stringify(version)) { + peer.versions.push(version); + } + } + + #sync(action: Extract): void { + const from = this.#peer(action.from); + const to = this.#peer(action.to); + if (from === to) return; + this.#commit(from); + this.#commit(to); + const cross = action.cross && this.#options.crossRuntimeSync; + this.#log.push( + `sync ${from.id} -> ${to.id} (${action.json ? "json" : "binary"}${cross ? ", cross" : ""})`, + ); + const move = (source: Replica, target: Replica): void => { + if (action.json) { + target.doc.importJsonUpdates(JSON.stringify(source.doc.exportJsonUpdates())); + } else { + target.doc.import(source.doc.export({ mode: "update" })); + } + }; + move(cross ? from.js : from.rust, to.rust); + move(cross ? from.rust : from.js, to.js); + this.#compare(to, `after sync ${from.id} -> ${to.id}`); + to.versions.push(to.rust.doc.frontiers()); + } + + #syncAll(): void { + this.#log.push("sync all"); + for (const from of this.#peers) { + for (const to of this.#peers) { + if (from === to) continue; + this.#commit(from); + const cross = this.#options.crossRuntimeSync; + to.rust.doc.import((cross ? from.js : from.rust).doc.export({ mode: "update" })); + to.js.doc.import((cross ? from.rust : from.js).doc.export({ mode: "update" })); + } + } + } + + #snapshotRoundTrip(action: Extract): void { + const peer = this.#peer(action.peer); + this.#commit(peer); + const source = action.fromRust ? peer.rust : peer.js; + const bytes = source.doc.export({ mode: "snapshot" }); + this.#log.push(`peer ${peer.id}: snapshot from ${source.runtime}`); + const version = + peer.versions.length === 0 + ? undefined + : peer.versions[action.version % peer.versions.length]!; + this.#compareImported(peer, bytes, `snapshot from ${source.runtime}`, version); + } + + #shallowRoundTrip(action: Extract): void { + const peer = this.#peer(action.peer); + if (peer.versions.length === 0) return; + this.#commit(peer); + const source = action.fromRust ? peer.rust : peer.js; + const index = action.version % peer.versions.length; + const frontiers = peer.versions[index]!; + const bytes = source.doc.export({ mode: "shallow-snapshot", frontiers }); + this.#log.push( + `peer ${peer.id}: shallow snapshot from ${source.runtime} at ${JSON.stringify(frontiers)}`, + ); + const later = peer.versions.slice(index); + this.#compareImported( + peer, + bytes, + `shallow snapshot from ${source.runtime}`, + later[action.version % later.length], + ); + } + + /** + * Imports `bytes` into fresh loro.js and Rust documents. loro.js must match the + * Rust import of the same bytes; both must match the source unless Rust + * disagrees with itself (a Rust shallow document can apply a delete whose + * recorded IDs and positions disagree differently from a full one). + */ + #compareImported( + peer: Peer, + bytes: Uint8Array, + label: string, + version: Frontiers | undefined, + ): void { + const js = new Js.LoroDoc(); + js.import(bytes); + const rust = new this.#rust.LoroDoc(); + rust.import(bytes); + const reference = normalizeDelta(rust.getText("t").toDelta()); + const source = normalizeDelta(this.#text(peer.rust).toDelta()); + if (JSON.stringify(reference) !== JSON.stringify(source)) { + // Rust's shallow import falls back to the recorded delete IDs when they + // disagree with the positions, which diverges from its own full-history + // result. loro.js keeps the full-history (position) result. + this.#log.push(" (the Rust import differs from its source)"); + } + this.#expectDelta(js, source, `${label}: loro.js import vs source`); + if (!this.#options.snapshotCheckout || version === undefined) return; + const errors: string[] = []; + for (const doc of [rust, js]) { + try { + doc.checkout(version); + } catch (error) { + errors.push(String(error)); + } + } + if (errors.length === 1) { + this.#fail( + `${label}: checkout ${JSON.stringify(version)} threw only in one runtime`, + ); + } + if (errors.length === 0) { + this.#expectDelta( + js, + normalizeDelta(rust.getText("t").toDelta()), + `${label}: checkout ${JSON.stringify(version)}`, + ); + } + } + + #createCursor(action: Extract): void { + const peer = this.#peer(action.peer); + const position = pickBoundary(this.#currentText(peer), action.pos); + const side = action.side; + const rust = this.#text(peer.rust).getCursor(position, side); + const js = this.#text(peer.js).getCursor(position, side); + this.#log.push(`peer ${peer.id}: cursor(${position}, ${side})`); + if ((rust === undefined) !== (js === undefined)) { + this.#fail(`cursor(${position}, ${side}) existence differs`); + } + if (rust === undefined || js === undefined) return; + if ( + JSON.stringify(rust.pos()) !== JSON.stringify(js.pos()) || + rust.side() !== js.side() + ) { + this.#fail( + `cursor(${position}, ${side}) differs: rust ${JSON.stringify(rust.pos())}/${rust.side()} js ${JSON.stringify(js.pos())}/${js.side()}`, + ); + } + peer.cursors.push({ rust, js }); + } + + #compareCursors(peer: Peer, label: string): void { + // Rust resolves a deleted target against the oplog head but converts the + // position with the checked-out state, so detached results are not compared. + if (peer.rust.doc.getPendingTxnLength() > 0 || peer.rust.doc.isDetached()) return; + for (const cursor of peer.cursors) { + const rust = peer.rust.doc.getCursorPos(cursor.rust); + const js = peer.js.doc.getCursorPos(cursor.js); + if ( + rust === undefined && + js !== undefined && + !recordedAsDeleted(peer.rust.doc, cursor.rust) + ) { + // Rust finds a deleted target through the recorded delete IDs, which its + // WASM build can get wrong for astral text; loro.js resolves deletes by + // position and still knows where the target was. + continue; + } + if (rust?.offset !== js?.offset || rust?.side !== js?.side) { + this.#fail( + `${label}: cursor ${JSON.stringify(cursor.rust.pos())}/${cursor.rust.side()} position differs: rust ${JSON.stringify(rust && { offset: rust.offset, side: rust.side })} js ${JSON.stringify(js && { offset: js.offset, side: js.side })}`, + ); + } + } + } + + #expectDelta(doc: Doc, expected: unknown, label: string): void { + const actual = normalizeDelta(doc.getText("t").toDelta()); + if (JSON.stringify(actual) !== JSON.stringify(expected)) { + this.#fail( + `${label}: delta ${JSON.stringify(actual)} != expected ${JSON.stringify(expected)}`, + ); + } + } + + #compare(peer: Peer, label: string): void { + const rustText = this.#text(peer.rust); + const jsText = this.#text(peer.js); + if (jsText.toDelta().some((item) => "insert" in item && item.insert.length === 0)) { + this.#fail(`${label}: peer ${peer.id} loro.js delta has an empty insert`); + } + const rustDelta = normalizeDelta(rustText.toDelta()); + const jsDelta = normalizeDelta(jsText.toDelta()); + if (JSON.stringify(rustDelta) !== JSON.stringify(jsDelta)) { + this.#fail( + `${label}: peer ${peer.id} delta differs\n rust ${JSON.stringify(rustDelta)}\n js ${JSON.stringify(jsDelta)}`, + ); + } + if (rustText.length !== jsText.length) { + this.#fail(`${label}: length ${rustText.length} != ${jsText.length}`); + } + for (const unit of ["unicode", "utf8"] as const) { + const rustLength = rustText.convertPos(rustText.length, "utf16", unit); + const jsLength = jsText.convertPos(jsText.length, "utf16", unit); + if (rustLength !== jsLength) { + this.#fail(`${label}: ${unit} length ${rustLength} != ${jsLength}`); + } + } + if (this.#options.events) { + for (const replica of [peer.js]) { + if (replica.doc.getPendingTxnLength() > 0) continue; + const events = normalizeDelta(replica.shadow.delta); + const state = normalizeDelta(this.#text(replica).toDelta()); + if (JSON.stringify(events) !== JSON.stringify(state)) { + this.#fail( + `${label}: ${replica.runtime} events do not reproduce the state\n events ${JSON.stringify(events)}\n state ${JSON.stringify(state)}`, + ); + } + } + } + if (this.#options.cursors) this.#compareCursors(peer, label); + if (this.#options.compareOps) this.#compareOps(peer, label); + } + + #compareOps(peer: Peer, label: string): void { + const rust = atomicOps(peer.rust.doc); + const js = atomicOps(peer.js.doc); + const index = rust.findIndex((atom, position) => atom !== js[position]); + if (index >= 0 || rust.length !== js.length) { + const at = index >= 0 ? index : Math.min(rust.length, js.length); + this.#fail( + `${label}: peer ${peer.id} ops differ at atom ${at}\n rust ${rust[at]}\n js ${js[at]}`, + ); + } + } + + #compareAll(label: string): void { + const expected = normalizeDelta(this.#text(this.#peers[0]!.rust).toDelta()); + for (const peer of this.#peers) { + this.#compare(peer, label); + this.#expectDelta(peer.js.doc, expected, `${label}: converge peer ${peer.id}`); + } + } + + #fail(message: string): never { + throw new DifferentialFailure( + `${message}\nsteps:\n ${this.#log.slice(-40).join("\n ")}`, + ); + } +} + +/** + * The JSON history as atomic ops sorted by ID: one entry per inserted scalar + * with its entity position, plus every other non-delete op. Rust merges adjacent + * ops of merged changes depending on internal encoding state, so this form + * ignores how atoms are grouped into ops. + */ +function atomicOps(doc: Doc): string[] { + const json = doc.exportJsonUpdates(); + const peers = json.peers; + const peerOf = (value: string): string => + peers == null ? value : peers[Number(value)]!; + const expand = (value: string): [number, string] => { + const [counter, peer] = value.split("@"); + return [Number(counter), peerOf(peer!)]; + }; + const atoms: { peer: string; counter: number; text: string }[] = []; + for (const change of json.changes) { + const [, peer] = expand(change.id); + for (const op of change.ops) { + const content = op.content as Record; + const push = (offset: number, text: string): void => { + atoms.push({ peer, counter: op.counter + offset, text }); + }; + if (content.type === "insert" && typeof content.text === "string") { + // One atom per Unicode scalar, each with its entity position. + let index = 0; + for (const char of content.text) { + push(index, `insert ${Number(content.pos) + index} ${JSON.stringify(char)}`); + index += 1; + } + } else if (content.type === "delete") { + // Deletes are compared through state: Rust's WASM build can record + // delete IDs that disagree with the positions it applies (astral text). + continue; + } else { + push(0, JSON.stringify(content)); + } + } + } + return atoms + .sort((left, right) => + left.peer < right.peer + ? -1 + : left.peer > right.peer + ? 1 + : left.counter - right.counter, + ) + .map((atom) => `${atom.counter}@${atom.peer} ${atom.text}`); +} + +/** Whether some recorded delete op names the cursor's target, as Rust requires. */ +function recordedAsDeleted(doc: Doc, cursor: Cursor): boolean { + const id = cursor.pos(); + if (id === undefined) return true; + const json = doc.exportJsonUpdates(); + const peers = json.peers; + for (const change of json.changes) { + for (const op of change.ops) { + const content = op.content as { type: string; len?: number; start_id?: string }; + if (content.type !== "delete" || content.start_id === undefined) continue; + const [counter, peerIndex] = content.start_id.split("@"); + const peer = peers == null ? peerIndex : peers[Number(peerIndex)]; + const start = Number(counter); + const length = Math.abs(content.len ?? 0); + if (peer === id.peer && id.counter >= start && id.counter < start + length) { + return true; + } + } + } + return false; +} + +/** Runs one scenario and returns the failure message, if any. */ +export function runActions( + rust: RustReference, + actions: readonly Action[], + options: ScenarioOptions = {}, +): string | undefined { + try { + new RichtextScenario(rust, options).run(actions); + return undefined; + } catch (error) { + return error instanceof DifferentialFailure + ? error.message + : error instanceof Error + ? (error.stack ?? error.message) + : String(error); + } +} + +/** + * Greedily removes actions while the scenario still fails the same way (same + * first failure line after replacing numbers), so shrinking does not wander + * into an unrelated divergence. + */ +export function shrinkActions( + rust: RustReference, + actions: readonly Action[], + options: ScenarioOptions = {}, +): Action[] { + let current = [...actions]; + const original = runActions(rust, current, options); + if (original === undefined) return current; + const kind = failureKind(original); + const stillFails = (candidate: readonly Action[]): boolean => { + const failure = runActions(rust, candidate, options); + return failure !== undefined && failureKind(failure) === kind; + }; + for (let chunk = Math.max(1, current.length >> 1); chunk >= 1; chunk >>= 1) { + let index = 0; + while (index < current.length) { + const candidate = [...current.slice(0, index), ...current.slice(index + chunk)]; + if (stillFails(candidate)) current = candidate; + else index += chunk; + } + } + return current; +} + +function failureKind(failure: string): string { + const line = failure.split("\n")[0]!; + const colon = line.indexOf(": "); + return (colon < 0 ? line : line.slice(colon + 2)).replace(/[0-9]+/g, "N").slice(0, 40); +} + +function boundaries(value: string, utf8 = false): number[] { + const output = [0]; + let offset = 0; + for (const scalar of value) { + offset += utf8 ? new TextEncoder().encode(scalar).length : scalar.length; + output.push(offset); + } + return output; +} + +function pickBoundary(value: string, raw: number, utf8 = false): number { + const all = boundaries(value, utf8); + return all[raw % all.length]!; +} + +function pickRange( + value: string, + raw: number, + length: number, + utf8 = false, +): [number, number] | undefined { + const all = boundaries(value, utf8); + if (all.length < 2) return undefined; + const start = raw % (all.length - 1); + const end = Math.min(all.length - 1, start + length); + return [all[start]!, all[end]!]; +} + +export function normalizeDelta(delta: readonly Delta[]): Delta[] { + const output: { insert: string; attributes?: Record }[] = []; + for (const item of delta) { + if (!("insert" in item) || item.insert.length === 0) continue; + const attributes = normalizeAttributes(item.attributes); + const previous = output.at(-1); + if ( + previous !== undefined && + JSON.stringify(previous.attributes) === JSON.stringify(attributes) + ) { + previous.insert += item.insert; + } else { + output.push( + attributes === undefined + ? { insert: item.insert } + : { insert: item.insert, attributes }, + ); + } + } + return output; +} + +function normalizeAttributes( + attributes: Readonly> | undefined, +): Record | undefined { + if (attributes === undefined) return undefined; + const entries = Object.entries(attributes) + .filter(([, value]) => value !== null && value !== undefined) + .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)); + return entries.length === 0 ? undefined : Object.fromEntries(entries); +} + +/** Applies a Quill-style text delta; `null` attribute values remove the key. */ +export function applyDelta( + document: readonly Delta[], + change: readonly Delta[], +): Delta[] { + const chars: { value: string; attributes: Record }[] = []; + for (const item of document) { + if (!("insert" in item)) continue; + for (let offset = 0; offset < item.insert.length; offset += 1) { + chars.push({ value: item.insert[offset]!, attributes: { ...item.attributes } }); + } + } + let index = 0; + for (const item of change) { + if ("insert" in item) { + const attributes = normalizeAttributes(item.attributes) ?? {}; + for (let offset = 0; offset < item.insert.length; offset += 1) { + chars.splice(index, 0, { + value: item.insert[offset]!, + attributes: { ...attributes }, + }); + index += 1; + } + } else if ("delete" in item) { + chars.splice(index, item.delete); + } else { + for (let offset = 0; offset < item.retain; offset += 1) { + const target = chars[index + offset]; + if (target === undefined) break; + for (const [key, value] of Object.entries(item.attributes ?? {})) { + if (value === null || value === undefined) delete target.attributes[key]; + else target.attributes[key] = value; + } + } + index += item.retain; + } + } + return normalizeDelta( + chars.map((char) => ({ insert: char.value, attributes: char.attributes })), + ); +} diff --git a/loro-js/tests/support/rust-reference.ts b/loro-js/tests/support/rust-reference.ts new file mode 100644 index 000000000..b3eef5ee7 --- /dev/null +++ b/loro-js/tests/support/rust-reference.ts @@ -0,0 +1,27 @@ +import { existsSync } from "node:fs"; +import { createRequire } from "node:module"; +import { fileURLToPath } from "node:url"; + +/** + * Loads the Rust implementation (`loro-crdt`, nodejs WASM target) as a + * differential-testing reference. Build it with + * `pnpm -C crates/loro-wasm build-dev` (or `build-release`), or point + * `LORO_WASM_NODEJS` at another build's `nodejs/index.js`. + * + * Returns `undefined` when no build is available so the differential suites can + * skip instead of failing on machines without a WASM toolchain. + */ +export function loadRustReference(): RustReference | undefined { + const path = + process.env.LORO_WASM_NODEJS ?? + fileURLToPath(new URL("../../../crates/loro-wasm/nodejs/index.js", import.meta.url)); + if (!existsSync(path)) { + if (process.env.LORO_REQUIRE_WASM_REFERENCE === "1") { + throw new Error(`Rust WASM reference build is missing: ${path}`); + } + return undefined; + } + return createRequire(import.meta.url)(path) as RustReference; +} + +export type RustReference = typeof import("../../src/index"); diff --git a/loro-js/tests/text-interop.test.ts b/loro-js/tests/text-interop.test.ts new file mode 100644 index 000000000..944b7f5f9 --- /dev/null +++ b/loro-js/tests/text-interop.test.ts @@ -0,0 +1,170 @@ +import { readFileSync } from "node:fs"; + +import { describe, expect, test } from "vitest"; + +import { LoroDoc, type Delta, type JsonSchema } from "../src/index"; +import { applyDelta, normalizeDelta } from "./support/richtext-differential"; + +const fixture = (name: string): Uint8Array => + new Uint8Array(readFileSync(new URL(`./fixtures/rust/${name}`, import.meta.url))); + +function doc(peer: number): LoroDoc { + const created = new LoroDoc(); + created.setPeerId(peer); + return created; +} + +function sync(from: LoroDoc, to: LoroDoc): void { + to.import(from.export({ mode: "update" })); +} + +describe("text interoperability with Rust", () => { + test("applies a remote delete by its position, like Rust's tracker", () => { + // Rust's WASM build recorded `start_id` 0@7 for a delete of a, 😀, x: it + // advanced the ID by the UTF-16 length of 😀, so the recorded IDs name b + // instead of x. Rust applies the delete by position and keeps "b". + const { text, updates } = JSON.parse( + readFileSync( + new URL("./fixtures/rust/text-delete-positions.json", import.meta.url), + "utf8", + ), + ) as { text: string; updates: JsonSchema }; + expect(text).toBe("b"); + + const fromJson = new LoroDoc(); + fromJson.importJsonUpdates(updates); + expect(fromJson.getText("text").toString()).toBe("b"); + + const fromBinary = new LoroDoc(); + fromBinary.import(fixture("text-delete-positions.blob")); + expect(fromBinary.getText("text").toString()).toBe("b"); + }); + + test("orders a concurrent insert before a later sibling subtree", () => { + const peer1 = doc(1); + const peer2 = doc(2); + const peer3 = doc(3); + peer2.getText("t").insert(0, "9"); + peer2.commit(); + peer1.getText("t").insert(0, "b"); + peer1.commit(); + sync(peer1, peer2); + expect(peer2.getText("t").toString()).toBe("b9"); + peer2.getText("t").insert(1, "é"); + peer2.commit(); + sync(peer1, peer3); + peer3.getText("t").insert(1, "a"); + peer3.commit(); + + // "a" follows b and é (b's subtree) but precedes 9, which is not in it. + for (const target of [peer1, peer2, peer3]) { + for (const source of [peer1, peer2, peer3]) { + if (source !== target) sync(source, target); + } + expect(target.getText("t").toString()).toBe("béa9"); + } + }); + + test("writes multi-run deletes last run first, like Rust", () => { + const owner = doc(1); + const text = owner.getText("t"); + text.insert(0, "abcd"); + text.insert(2, "X"); + owner.commit(); + text.delete(1, 3); // b, X, c: three ID runs + owner.commit(); + const ops = owner + .exportJsonUpdates() + .changes.at(-1)! + .ops.map((op) => op.content); + // Output of Rust (loro-crdt) for the same edits; peer 1 is index 0. + expect(ops.slice(-3)).toEqual([ + { type: "delete", pos: 3, len: 1, start_id: "2@0" }, + { type: "delete", pos: 2, len: 1, start_id: "4@0" }, + { type: "delete", pos: 1, len: 1, start_id: "1@0" }, + ]); + expect(text.toString()).toBe("ad"); + }); + + test("merges consecutive text inserts in a transaction, like Rust", () => { + const owner = doc(1); + const text = owner.getText("t"); + text.insert(0, "中文"); + owner.commit(); + text.insert(1, "😀"); + text.insert(3, "ab"); + owner.commit(); + const ops = owner + .exportJsonUpdates() + .changes.flatMap((change) => change.ops.map((op) => op.content)); + expect(ops).toEqual([ + { type: "insert", pos: 0, text: "中文" }, + { type: "insert", pos: 1, text: "😀ab" }, + ]); + }); + + test("checkout events restore an element deleted twice only once", () => { + const peer1 = doc(1); + const peer2 = doc(2); + peer2.getText("t").insert(0, "中文"); + peer2.commit(); + const base = peer2.frontiers(); + sync(peer2, peer1); + peer2.getText("t").delete(0, 2); + peer2.commit(); + peer1.getText("t").delete(1, 1); + peer1.commit(); + sync(peer2, peer1); + peer1.getText("t").insert(0, "xyz"); + peer1.commit(); + + let shadow: Delta[] = peer1.getText("t").toDelta(); + peer1.subscribe((batch) => { + for (const event of batch.events) { + if (event.diff.type === "text") { + shadow = applyDelta(shadow, event.diff.diff as Delta[]); + } + } + }); + peer1.checkout(base); + expect(peer1.getText("t").toString()).toBe("中文"); + expect(normalizeDelta(shadow)).toEqual(normalizeDelta(peer1.getText("t").toDelta())); + peer1.checkoutToLatest(); + expect(normalizeDelta(shadow)).toEqual(normalizeDelta(peer1.getText("t").toDelta())); + }); + + test("text cursors follow Rust's position semantics", () => { + const owner = doc(1); + const text = owner.getText("t"); + const empty = text.getCursor(0, 0)!; + expect(empty.pos()).toBeUndefined(); + expect(empty.side()).toBe(-1); + + text.insert(0, "a😀bc"); + owner.commit(); + const end = text.getCursor(text.length, -1)!; + expect(end.pos()).toBeUndefined(); + expect(end.side()).toBe(1); + expect(owner.getCursorPos(end)).toEqual({ offset: 5, side: 1 }); + expect(text.getCursor(2, 0)).toBeUndefined(); + + // The encoded cursor matches Rust byte for byte: its origin is the Unicode + // position (2), not the UTF-16 position (3). + expect([...text.getCursor(3, 1)!.encode()]).toEqual([1, 1, 4, 0, 1, 116, 0, 2, 2]); + expect([...text.getCursor(5, 1)!.encode()]).toEqual([0, 0, 1, 116, 0, 2, 5]); + + // A visible target reports its own offset for every side. + const right = text.getCursor(3, 1)!; + expect(right.pos()).toEqual({ peer: "1", counter: 2 }); + expect(owner.getCursorPos(right)).toEqual({ offset: 3, side: 1 }); + + // A deleted target reports the length before it, on the left side. + text.delete(1, 3); + owner.commit(); + const deleted = owner.getCursorPos(right)!; + expect(deleted.offset).toBe(1); + expect(deleted.side).toBe(-1); + expect(deleted.update?.pos()).toEqual({ peer: "1", counter: 3 }); + expect(deleted.update?.side()).toBe(-1); + }); +}); diff --git a/package.json b/package.json index f7aae33fe..cd851ff4a 100644 --- a/package.json +++ b/package.json @@ -8,9 +8,10 @@ "check-all": "cargo hack check --each-feature", "build": "cargo build", "test": "cargo nextest run --features=test_utils,jsonpath --no-fail-fast && cargo test --doc", - "test-all": "pnpm test && pnpm test-fractional-index && pnpm release-wasm && pnpm test-loom", + "test-all": "pnpm test && pnpm test-fractional-index && pnpm release-wasm && pnpm test-loro-js && pnpm test-loom", "test-wasm": "cd crates/loro-wasm && pnpm i && pnpm build-dev", "test-fractional-index": "pnpm --filter @loro-dev/fractional-index install --frozen-lockfile && pnpm --filter @loro-dev/fractional-index check", + "test-loro-js": "pnpm --filter loro.js install --frozen-lockfile && LORO_REQUIRE_WASM_REFERENCE=1 pnpm --filter loro.js test", "test-loom": "LOOM_MAX_PREEMPTIONS=2 RUSTFLAGS='--cfg loom' cargo test -p loro --test multi_thread_test --release", "test-loom-reproduce": "LOOM_MAX_PREEMPTIONS=2 LOOM_LOG=info LOOM_LOCATION=1 LOOM_CHECKPOINT_INTERVAL=1 LOOM_CHECKPOINT_FILE=loom_test.json RUSTFLAGS='--cfg loom' cargo test -p loro --test multi_thread_test --release", "watch-loom": "RUSTFLAGS='--cfg loom' bacon", From 8b67bda30e914fa342d61a5f6885c39f38f49b99 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 04:13:40 +0800 Subject: [PATCH 02/25] fix(loro.js): keep rich-text style anchors in the text sequence Rust writes every Text op position as an entity index that counts style anchors. loro.js kept styles beside the text, so once a document had a mark, each runtime applied the other's later edits at different places (#1133). Anchors are now zero-width elements of the Text sequence: remote marks insert them with Fugue, local edits follow Rust's insert/delete/mark position rules, styles cover the elements between their anchors, and snapshots, checkout, events, and undo use them. The differential suite now runs with marks. See context/loro-js-richtext-anchors.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- .changeset/loro-js-richtext-anchors.md | 15 + AGENTS.md | 2 + context/loro-js-performance.md | 37 +- context/loro-js-richtext-anchors.md | 135 +++ crates/loro/tests/loro_js_interop.rs | 87 ++ loro-js/AGENTS.md | 3 + loro-js/README.md | 18 +- loro-js/scripts/rewrite-rust-fixture.mjs | 62 ++ .../scripts/write-rust-richtext-fixtures.cjs | 81 ++ loro-js/src/runtime/containers.ts | 849 ++++++++++++++---- loro-js/src/runtime/document.ts | 353 ++++---- loro-js/src/runtime/sequence-index.ts | 224 +++++ loro-js/src/runtime/text-style-index.ts | 207 +++-- .../tests/fixtures/rust/richtext-base.ts.blob | Bin 0 -> 150 bytes .../fixtures/rust/richtext-checkout-mark.blob | Bin 0 -> 104 bytes .../fixtures/rust/richtext-checkout-mark.json | 80 ++ .../rust/richtext-checkout-mark.snapshot.blob | Bin 0 -> 278 bytes .../tests/fixtures/rust/richtext-left.ts.blob | Bin 0 -> 124 bytes .../rust/richtext-mark-positions.blob | Bin 0 -> 118 bytes .../rust/richtext-mark-positions.json | 79 ++ .../richtext-mark-positions.snapshot.blob | Bin 0 -> 296 bytes .../fixtures/rust/richtext-right.ts.blob | Bin 0 -> 138 bytes .../fixtures/rust/richtext-snapshot.ts.blob | Bin 0 -> 598 bytes .../fixtures/rust/richtext-unmark-emoji.blob | Bin 0 -> 123 bytes .../fixtures/rust/richtext-unmark-emoji.json | 92 ++ .../rust/richtext-unmark-emoji.snapshot.blob | Bin 0 -> 309 bytes .../fixtures/rust/richtext.expected.json | 63 ++ loro-js/tests/fixtures/rust/shallow.ts.blob | Bin 414 -> 369 bytes loro-js/tests/indexes.test.ts | 82 +- loro-js/tests/richtext-anchors.test.ts | 297 ++++++ loro-js/tests/richtext-differential.test.ts | 14 +- 31 files changed, 2341 insertions(+), 439 deletions(-) create mode 100644 .changeset/loro-js-richtext-anchors.md create mode 100644 context/loro-js-richtext-anchors.md create mode 100644 loro-js/scripts/write-rust-richtext-fixtures.cjs create mode 100644 loro-js/tests/fixtures/rust/richtext-base.ts.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-checkout-mark.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-checkout-mark.json create mode 100644 loro-js/tests/fixtures/rust/richtext-checkout-mark.snapshot.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-left.ts.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-mark-positions.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-mark-positions.json create mode 100644 loro-js/tests/fixtures/rust/richtext-mark-positions.snapshot.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-right.ts.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-snapshot.ts.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-unmark-emoji.blob create mode 100644 loro-js/tests/fixtures/rust/richtext-unmark-emoji.json create mode 100644 loro-js/tests/fixtures/rust/richtext-unmark-emoji.snapshot.blob create mode 100644 loro-js/tests/fixtures/rust/richtext.expected.json create mode 100644 loro-js/tests/richtext-anchors.test.ts diff --git a/.changeset/loro-js-richtext-anchors.md b/.changeset/loro-js-richtext-anchors.md new file mode 100644 index 000000000..5e27ad710 --- /dev/null +++ b/.changeset/loro-js-richtext-anchors.md @@ -0,0 +1,15 @@ +--- +"loro.js": minor +--- + +Rich text now uses Rust's position model. A mark's start and end anchors are elements of the text sequence, as in Rust (`loro-crdt`), so every insert, delete, and mark position loro.js writes or reads counts them. Before, loro.js wrote and read positions without the anchors, so as soon as a document had a mark, loro.js and Rust applied each other's later edits at different places (for example, after marking `ab` in `abcd`, inserting `X` at 3 and deleting 1 gave `acXd` in Rust but `acdX` when loro.js imported Rust's update, and `bXcd` when Rust imported loro.js's). + +Behavior now follows Rust: + +- Text inserted where a mark starts or ends goes inside or outside the mark according to the mark's expand setting, as Rust decides it; concurrent edits at a mark boundary converge with Rust. +- `mark` and `unmark` skip a mark that would change nothing, and throw when `start >= end`. +- `applyDelta` drops a style that inserted text would inherit when the insert does not list it, like Rust's `apply_delta`. +- Undo and redo restore the attributes a mark replaced. +- Snapshots write and read the anchors like Rust, so checking out an older version of a styled text gives the same result in both runtimes. + +Documents that loro.js edited after a mark with an earlier version contain positions that only loro.js read that way; Rust has always read them with the anchors, and loro.js now does too. diff --git a/AGENTS.md b/AGENTS.md index beb65f99b..623ecaa9a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,6 +33,8 @@ Loro is a Rust CRDT workspace with JS/WASM packaging and a MoonBit codec. [context/loro-js-performance.md](context/loro-js-performance.md). - loro.js vs Rust randomized Text differential tests and the Rust behaviors they work around: [context/loro-js-rust-differential.md](context/loro-js-rust-differential.md). +- loro.js rich-text style anchors, entity positions, and Rust's insert/delete/mark + position rules: [context/loro-js-richtext-anchors.md](context/loro-js-richtext-anchors.md). - WASM panic/OOM reporting channels, `__wbindgen_start` glue invariant, and trap-testing recipes: [context/wasm-error-reporting.md](context/wasm-error-reporting.md). diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 38a37474a..8b41bb5b5 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -40,12 +40,19 @@ JavaScript constant factor. - `ordered-index.ts` is the ordered rank index used for map keys and tree children. Insert/delete/rank lookup are expected O(log n), while ordered iteration is O(n). +- Rich-text style anchors are zero-width elements of the Text sequence, as in + Rust ([loro-js-richtext-anchors.md](loro-js-richtext-anchors.md)). Each + subtree also counts its elements without UTF-16 width, so Unicode and entity + positions convert in O(log n); a sequence without anchors takes the old fast + paths. - `text-style-index.ts` stores style histories in disjoint operation-ID ranges, - separately from scalar Text elements. Applying, unapplying, or checking a - style range is expected O(log style-runs + affected style-runs). Full-range - marks and their subscribed checkout events no longer write or inspect every - character. Delta and snapshot output reuse a run-local style resolver so - their work remains linear in returned text and style runs. + separately from scalar Text elements. A style covers the elements physically + between its anchors, so applying one is expected O(log n + ID runs in its + range). Checking or undoing a style range is expected O(log style-runs + + affected style-runs). Full-range marks and their subscribed checkout events + no longer write or inspect every character. Delta and snapshot output reuse a + run-local style resolver so their work remains linear in returned text and + style runs. - `LoroDoc` maintains per-peer change arrays, end counters, operation counts, current frontiers, sorted-history cache, and per-change dependency-version caches. Latest version/frontier lookup is O(peer/frontier count), and @@ -208,7 +215,10 @@ heap and 322.1 MB RSS. The original array implementation was estimated at 30–50 minutes. Prefix measurements from 20k through the full trace scale approximately linearly. The matching Rust Criterion benchmark has a 47.711 ms point estimate on the same -machine, so TypeScript is about 7.4x slower in absolute time. B4 leaves 182,315 +machine, so TypeScript is about 7.4x slower in absolute time. Merging +consecutive inserts of a transaction into one op, as Rust does, shrank the B4 +update export from 1,153,540 to 274,574 bytes, and merging contiguous text in +the Text state shrank the snapshot from 309,780 to 206,553 bytes. B4 leaves 182,315 scalar objects but packs them into 13,613 TypeScript treap nodes. The same run measured snapshot export at 162.4 ms, update export at 129.3 ms, snapshot import at 161.5 ms, and @@ -268,6 +278,21 @@ isolated repeated probe. Retreating/restoring that full-range mark takes about now scale with ID/style runs and emitted formatting ranges rather than the 64k characters. +With style anchors in the sequence (September 28, Node 22, best of 7 on a +loaded machine, same process for both revisions): the 64k full-range mark takes +0.07–0.08 ms (0.06–0.08 before), its retreat/restore 0.06–0.07 ms (about 0.01 +before), and the subscribed restore 0.06 ms. Typing 1,000 characters inside the +bold range takes 3.1–3.3 ms (1.7–2.1 before): each insert intersects the style +memberships of its two physical neighbors. Marking the same range again and +again nests anchors: the n-th mark's range contains the n-1 earlier start +anchors as separate ID runs, so applying it and moving the version across it +are O(n), as in Rust, whose `StyleRangeMap` has one segment per anchor there. +`text-repeated-mark-tail-{retreat,restore}` therefore grows with its size +(0.5/1.1/2.2/6.9 ms at 1k/2k/4k/8k, versus 0.16–0.29 ms before); the Rust WASM +build takes 7.1/19/362 ms to retreat 1k/4k/16k such marks, and building the 16k +history takes 402 s in loro.js and 542 s in Rust. Every other +`bench:complexity` entry stays flat from 1k to 8k. + A subscribed forward checkout that combines a full-range delete and mark takes 0.41 ms at 1k characters and 0.16 ms at 8k after warmup. Historical mark positions are converted directly to causal ID runs, and removed ID runs are diff --git a/context/loro-js-richtext-anchors.md b/context/loro-js-richtext-anchors.md new file mode 100644 index 000000000..c9e26be7b --- /dev/null +++ b/context/loro-js-richtext-anchors.md @@ -0,0 +1,135 @@ +# loro-js Rich-Text Anchors + +Verified against code 2026-09-28. + +loro.js keeps rich-text style anchors in the text sequence, as Rust does, so a +Text op position means the same thing in both runtimes. This article covers the +model, the position rules, and how the indexes keep the +[complexity contract](loro-js-performance.md). + +## Why anchors are sequence elements + +Rust writes a mark as two ops: `StyleStart { start, end, key, value, info }` at +counter `c` and `StyleEnd` at `c + 1`. Each inserts a zero-width anchor into the +Fugue sequence (`RichtextDiffCalculator::apply_crdt_op_to_tracker` in +`crates/loro-internal/src/diff_calc.rs`): the start anchor at entity position +`start`, then the end anchor at `end + 1`, in a causal view that now contains the +start anchor. Every Text op position (`insert.pos`, `delete.pos`, +`mark.start/end`, binary and JSON alike) is an *entity index*: Unicode scalars +plus visible anchors, in the op's causal view (`container/richtext.rs`). + +Converting positions only when importing or exporting cannot be exact. A +concurrent insert can have an anchor as its Fugue origin, and whether text is +inside a style depends on its Fugue order relative to the anchors. So loro.js +stores the anchors in `LoroText._sequence`. + +## Data model + +- An anchor is a `TextElement` with `value: ""` and `anchor: { style, isEnd }` + (`loro-js/src/runtime/containers.ts`). The two anchors of a mark share one + `TextStyle`: `startId`, `lamport`, `info`, `value`, `key`, and the op's `end`. + The start anchor has the `StyleStart` ID and lamport, the end anchor the next + counter and lamport. +- Anchors use single-element storage. `TextSequenceSpan` never holds one; + `LoroText.compact` keeps 32-element chunks with anchors as element arrays. +- `SequenceIndex` counts the elements with no UTF-16 width in each node + (`ownZeroWidth`, `ownVisibleZeroWidth`, `visibleZeroWidth`, `allZeroWidth`). + Lists never have one. A sequence without anchors pays one addition per + recompute. +- Styles stay in `TextStyleIndex`, keyed by element ID, but membership now comes + from the anchors: a style covers every element physically between its start + and end anchors, both included. That includes hidden elements and elements + inserted there later, and it never changes once the elements exist. A version + only filters which styles apply. +- `LoroText` keeps each peer's start-anchor counters in a sorted array + (`#anchorStarts`), so an ID run can be split at anchors without looking up its + elements. + +## Positions + +| Unit | How loro.js converts it | +| --- | --- | +| UTF-16 / UTF-8 | The existing prefix metrics. Anchors have width 0, so every entity index in a run of anchors maps to the same offset. | +| Unicode | `entity - zeroWidthBeforeVisibleIndex(entity)`; back with `visibleIndexOfWidthElement(unicode)`. O(log n). | +| Entity | The visible index of `SequenceIndex`. Op positions use it. | + +Local edits follow Rust's `TextHandler`: + +- **Insert** (`LoroText.#insertPosition`, Rust `find_best_insert_pos`): start + right after the scalar that ends at the UTF-16 position (or at 0), then skip + the anchors there until the first one that must stay after the new text: a + start anchor of an unmark (`null` or `false` value), a start anchor of a style + that does not expand before, or an end anchor of a style that expands after. + Rust's insert-position cache used to bypass this rule at text-leaf boundaries + (fixed in loro-dev/loro#1135; + `insert_position_near_style_anchors_does_not_depend_on_cursor_cache` in + `crates/loro/tests/contracts/text_richtext_advanced.rs`). +- **Delete** (`LoroText._deleteRuns`, Rust `get_text_entity_ranges`): the text + from the scalar containing the first UTF-16 unit to the one containing the + last, as runs that are consecutive in both entity position and ID. Anchors are + never deleted; they split runs. Ops are written last run first, so each + recorded position is still valid when applied. +- **Mark** (`LoroText.mark`): both ends use the insert rule. Like Rust's + `mark_with_txn`, the mark is skipped when every entity position in the range + already resolves the key to the value, or when it is an unmark and no position + has the key. `start >= end` throws, as in Rust. Detached texts insert anchors + with Rust's `TextStyleInfoFlag::BOLD` info. +- **applyDelta** follows Rust's `apply_delta`: inserted text without an + attribute explicitly drops each style it would inherit (the styles shared by + both visible neighbors), and all marks run after the whole delta. + +Remote ops: + +- Text inserts and anchors use the same Fugue insertion at the entity position + in the op's causal view. +- Deletes are resolved by position in the causal view, like Rust's tracker, not + by `start_id` (`LoroText._deleteTargets`). +- A new element takes every style shared by its physical neighbors, which is + exactly the set of styles whose anchors enclose it (`#inheritGapStyles`). A + new end anchor applies its style to the physical range between its anchors + (`LoroText._applyStyleEnd`). + +## Styles, versions, events, and undo + +- The value of a key is the covering style with the greatest `(lamport, peer)`; + `null` means the attribute is absent (Rust `StyleValue::get`). +- A style applies once its end anchor is included in a version: + `startId.counter + 1 < version` (`latestIncluded` in `text-style-index.ts`). +- A mark's event is emitted with its end anchor and covers the text whose + resolved value changed (`LoroText._styleChangeRuns`). +- Checkout toggles anchors like inserted text and reports style changes through + `TextStyleIndex.transitions` over the style's member runs. +- Undoing a mark gives back, wherever it changed the value, the value from just + before its op (`LoroText._undoStyle`). That matches Rust for local histories; + loro.js undo still does not transform against remote edits the way Rust's + `UndoManager` does. + +## Snapshots + +`LoroDoc.#containerState` writes Rust's text state layout (`encode_snapshot_fast` +in `crates/loro-internal/src/state/richtext_state.rs`): text spans and anchors in +document order. A start anchor is a zero-length span followed by its mark entry; +an end anchor is a span of length -1 at the start counter + 1 that keeps the +start's lamport offset. Contiguous text is merged into one span. +`#hydrateContainerState` rebuilds the same elements and each style's range +(`LoroText._appendElements`). + +## Complexity + +- Without anchors, every conversion short-circuits on + `visibleZeroWidthLength === 0`, and delete runs use `visibleIdRuns` directly. +- Unicode conversions, the insert rule, delete runs, and applying a style are + O(log n + anchors at the position + returned runs). Applying a style to a + contiguous range is O(log n + ID runs), as before. +- Inserting inside styled text intersects two style histories per insert, which + is proportional to the number of styles covering that position. + +## Testing + +- `loro-js/tests/richtext-anchors.test.ts` compares loro.js with Rust fixtures + (`loro-js/scripts/write-rust-richtext-fixtures.cjs`): the JSON ops loro.js + writes, imports of Rust updates and snapshots, and checkouts. +- `crates/loro/tests/loro_js_interop.rs` imports loro.js rich-text updates and + snapshots written by `loro-js/scripts/rewrite-rust-fixture.mjs`. +- The randomized Rust differential suite is described in + [loro-js-rust-differential.md](loro-js-rust-differential.md). diff --git a/crates/loro/tests/loro_js_interop.rs b/crates/loro/tests/loro_js_interop.rs index 1aea9892a..60fb124b3 100644 --- a/crates/loro/tests/loro_js_interop.rs +++ b/crates/loro/tests/loro_js_interop.rs @@ -39,6 +39,16 @@ const TS_LEGACY_TREE_MOVE_SNAPSHOT: &[u8] = include_bytes!("../../../loro-js/tests/fixtures/rust/legacy-tree-move-snapshot.ts.blob"); const TS_LEGACY_TREE_MOVE_SHALLOW_SNAPSHOT: &[u8] = include_bytes!("../../../loro-js/tests/fixtures/rust/legacy-tree-move-shallow.ts.blob"); +const TS_RICHTEXT_BASE: &[u8] = + include_bytes!("../../../loro-js/tests/fixtures/rust/richtext-base.ts.blob"); +const TS_RICHTEXT_LEFT: &[u8] = + include_bytes!("../../../loro-js/tests/fixtures/rust/richtext-left.ts.blob"); +const TS_RICHTEXT_RIGHT: &[u8] = + include_bytes!("../../../loro-js/tests/fixtures/rust/richtext-right.ts.blob"); +const TS_RICHTEXT_SNAPSHOT: &[u8] = + include_bytes!("../../../loro-js/tests/fixtures/rust/richtext-snapshot.ts.blob"); +const TS_RICHTEXT_EXPECTED_JSON: &[u8] = + include_bytes!("../../../loro-js/tests/fixtures/rust/richtext.expected.json"); const TS_CURSOR: &[u8] = include_bytes!("../../../loro-js/tests/fixtures/rust/cursor.ts.blob"); const TS_AWARENESS: &[u8] = include_bytes!("../../../loro-js/tests/fixtures/rust/awareness.ts.blob"); @@ -157,6 +167,83 @@ fn imports_typescript_shallow_snapshot_and_checks_out_its_root() { ); } +/// A Quill delta with null attributes dropped and equal neighbors merged, so +/// deltas compare by content rather than by how each runtime splits them. +fn normalized_delta(value: serde_json::Value) -> serde_json::Value { + let mut output: Vec = Vec::new(); + for item in value.as_array().expect("a delta is an array") { + let insert = item["insert"].as_str().expect("a text insert").to_string(); + let attributes: serde_json::Map = item + .get("attributes") + .and_then(|attributes| attributes.as_object()) + .map(|attributes| { + attributes + .iter() + .filter(|(_, value)| !value.is_null()) + .map(|(key, value)| (key.clone(), value.clone())) + .collect() + }) + .unwrap_or_default(); + if let Some(previous) = output.last_mut() { + let previous_attributes = previous + .get("attributes") + .cloned() + .unwrap_or(serde_json::Value::Object(Default::default())); + if previous_attributes == serde_json::Value::Object(attributes.clone()) { + let merged = format!("{}{}", previous["insert"].as_str().unwrap(), insert); + previous["insert"] = merged.into(); + continue; + } + } + let mut entry = serde_json::json!({ "insert": insert }); + entry["attributes"] = serde_json::Value::Object(attributes); + output.push(entry); + } + serde_json::Value::Array(output) +} + +fn richtext_delta(doc: &LoroDoc) -> serde_json::Value { + normalized_delta(doc.get_text("text").get_richtext_value().to_json_value()) +} + +#[test] +fn imports_typescript_rich_text_with_anchor_positions() { + let expected: serde_json::Value = + serde_json::from_slice(TS_RICHTEXT_EXPECTED_JSON).expect("valid rich-text fixture"); + let expected_delta = normalized_delta(expected["delta"].clone()); + + let doc = LoroDoc::new(); + for update in [TS_RICHTEXT_BASE, TS_RICHTEXT_LEFT, TS_RICHTEXT_RIGHT] { + doc.import(update) + .expect("Rust should import TypeScript rich-text updates"); + } + assert_eq!(richtext_delta(&doc), expected_delta); + + let snapshot = LoroDoc::new(); + snapshot + .import(TS_RICHTEXT_SNAPSHOT) + .expect("Rust should import a TypeScript rich-text snapshot"); + assert_eq!(richtext_delta(&snapshot), expected_delta); + let frontiers: Vec = expected["base"]["frontiers"] + .as_array() + .unwrap() + .iter() + .map(|id| { + ID::new( + id["peer"].as_str().unwrap().parse().unwrap(), + id["counter"].as_i64().unwrap() as i32, + ) + }) + .collect(); + snapshot + .checkout(&frontiers.into()) + .expect("the snapshot history should check out"); + assert_eq!( + richtext_delta(&snapshot), + normalized_delta(expected["base"]["delta"].clone()) + ); +} + #[test] fn decodes_typescript_cursor_with_rust() { let cursor = Cursor::decode(TS_CURSOR).expect("Rust should decode a TypeScript cursor"); diff --git a/loro-js/AGENTS.md b/loro-js/AGENTS.md index cd5b3661c..b74257f0b 100644 --- a/loro-js/AGENTS.md +++ b/loro-js/AGENTS.md @@ -9,6 +9,9 @@ the same binary and JSON data as Rust (`loro-crdt`) and give the same results. [context/loro-js-performance.md](../context/loro-js-performance.md). - Randomized Text comparison with Rust (build the nodejs WASM package first): [context/loro-js-rust-differential.md](../context/loro-js-rust-differential.md). +- Rich text: style anchors live in the Text sequence and op positions count + them. Read [context/loro-js-richtext-anchors.md](../context/loro-js-richtext-anchors.md) + before changing Text positions, marks, or Text snapshot state. - Encoding formats: [context/internal-encoding.md](../context/internal-encoding.md) and `docs/encoding*.md`. diff --git a/loro-js/README.md b/loro-js/README.md index 6e037ad0e..7ddd5f286 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -94,9 +94,11 @@ The following paths are checked against the Rust implementation: - JSON update import/export and the postcard Awareness/EphemeralStore protocols. - Rust-produced fixtures imported by TypeScript and TypeScript-produced fixtures imported by Rust. -- Randomized multi-peer text edits run side by side with the `loro-crdt` WASM build - (`tests/richtext-differential.test.ts`): positions, deletes, concurrency, checkout, - snapshots, cursors, events, and `revertTo`. +- Randomized multi-peer text and rich-text edits run side by side with the + `loro-crdt` WASM build (`tests/richtext-differential.test.ts`): positions, deletes, + marks with every expand mode, concurrency, checkout, snapshots, cursors, events, + and `revertTo`. Style anchors are text-sequence elements, as in Rust, so every op + position counts them. This is not yet a claim of complete behavioral equivalence with every `loro-crdt` edge case. Important current limits are: @@ -109,11 +111,11 @@ edge case. Important current limits are: importing into a new document or using `LoroDoc.fromSnapshot()`. - `diff()` preserves parent-before-child ordering, but independent containers at the same depth can appear in a different order than Rust's internal hash-map iteration. -- `UndoManager` uses ID-span-based semantic undo for sequence edits and common - map/tree/counter changes. Style-only rich-text edits and movable-list move/set undo - still use simplified behavior. -- The hardest overlapping rich-text-anchor and concurrent movable-list cases still use - simplified metadata compared with the Rust implementation. +- `UndoManager` uses ID-span-based semantic undo for sequence edits, marks, and + common map/tree/counter changes. It does not transform an undo against later remote + edits like Rust's, and movable-list move/set undo still uses simplified behavior. +- The hardest concurrent movable-list cases still use simplified metadata compared + with the Rust implementation. - `subscribeJsonpath()` deliberately uses broad invalidation, so its callback can have false positives. diff --git a/loro-js/scripts/rewrite-rust-fixture.mjs b/loro-js/scripts/rewrite-rust-fixture.mjs index aa2aab24b..616c0820d 100644 --- a/loro-js/scripts/rewrite-rust-fixture.mjs +++ b/loro-js/scripts/rewrite-rust-fixture.mjs @@ -221,6 +221,68 @@ writeFileSync( cursorDoc.getText("text").getCursor(1).encode(), ); +// Rich text: concurrent edits at style boundaries after marks and unmarks. Op +// positions count style anchors, so Rust must read them the same way. +const richStyles = { + bold: { expand: "after" }, + link: { expand: "none" }, + em: { expand: "before" }, + hl: { expand: "both" }, +}; +const richLeft = new LoroDoc(); +richLeft.setPeerId(21); +richLeft.configTextStyle(richStyles); +richLeft.getText("text").insert(0, "Hello 😀 World"); +richLeft.getText("text").mark({ start: 0, end: 5 }, "bold", true); +richLeft.getText("text").mark({ start: 6, end: 8 }, "link", "https://loro.dev"); +richLeft.commit(); +const richVersion = richLeft.frontiers(); +const richBase = richLeft.export({ mode: "update" }); +const richRight = new LoroDoc(); +richRight.setPeerId(22); +richRight.configTextStyle(richStyles); +richRight.import(richBase); +richLeft.getText("text").insert(5, "!"); +richLeft.getText("text").unmark({ start: 1, end: 3 }, "bold"); +richLeft.getText("text").mark({ start: 3, end: 10 }, "hl", "yellow"); +richLeft.commit(); +richRight.getText("text").insert(0, ">"); +richRight.getText("text").insert(6, "?"); +richRight.getText("text").delete(7, 3); +richRight.getText("text").mark({ start: 2, end: 4 }, "em", true); +richRight.commit(); +const richLeftUpdate = richLeft.export({ + mode: "update", + from: richRight.oplogVersion(), +}); +const richRightUpdate = richRight.export({ + mode: "update", + from: richLeft.oplogVersion(), +}); +richLeft.import(richRightUpdate); +richRight.import(richLeftUpdate); +if ( + JSON.stringify(richLeft.getText("text").toDelta()) !== + JSON.stringify(richRight.getText("text").toDelta()) +) { + throw new Error("TypeScript rich-text replicas did not converge"); +} +writeFileSync(fixtureUrl("richtext-base.ts.blob"), richBase); +writeFileSync(fixtureUrl("richtext-left.ts.blob"), richLeftUpdate); +writeFileSync(fixtureUrl("richtext-right.ts.blob"), richRightUpdate); +writeFileSync( + fixtureUrl("richtext-snapshot.ts.blob"), + richLeft.export({ mode: "snapshot" }), +); +const richLatest = richLeft.getText("text").toDelta(); +richLeft.checkout(richVersion); +const richBaseDelta = richLeft.getText("text").toDelta(); +richLeft.checkoutToLatest(); +writeFileSync( + fixtureUrl("richtext.expected.json"), + `${JSON.stringify({ delta: richLatest, base: { frontiers: richVersion, delta: richBaseDelta } }, null, 2)}\n`, +); + const awareness = new AwarenessWasm("123", 30_000); awareness.setLocalState({ status: "typing", position: 3 }); writeFileSync(fixtureUrl("awareness.ts.blob"), awareness.encodeAll()); diff --git a/loro-js/scripts/write-rust-richtext-fixtures.cjs b/loro-js/scripts/write-rust-richtext-fixtures.cjs new file mode 100644 index 000000000..4dee4eb8a --- /dev/null +++ b/loro-js/scripts/write-rust-richtext-fixtures.cjs @@ -0,0 +1,81 @@ +/* eslint-disable no-console */ +// Writes the Rust (loro-crdt) side of the rich-text interop fixtures used by +// tests/richtext-anchors.test.ts. Build the nodejs WASM package first +// (`pnpm -C crates/loro-wasm build-dev`) or point LORO_WASM_NODEJS at one. +const { writeFileSync } = require("node:fs"); +const path = require("node:path"); + +const wasmPath = + process.env.LORO_WASM_NODEJS ?? + path.resolve(__dirname, "../../crates/loro-wasm/nodejs/index.js"); +const { LoroDoc } = require(wasmPath); +const fixture = (name) => path.resolve(__dirname, "../tests/fixtures/rust", name); + +function newDoc() { + const doc = new LoroDoc(); + doc.setPeerId(1); + doc.configTextStyle({ bold: { expand: "after" }, link: { expand: "none" } }); + return doc; +} + +const scenarios = { + // loro-dev/loro#1133: positions written after a mark count its anchors. + "richtext-mark-positions"(doc) { + const text = doc.getText("text"); + text.insert(0, "abcd"); + doc.commit(); + text.mark({ start: 0, end: 2 }, "bold", true); + doc.commit(); + text.insert(3, "X"); + doc.commit(); + text.delete(1, 1); + doc.commit(); + }, + "richtext-unmark-emoji"(doc) { + const text = doc.getText("text"); + text.insert(0, "a😀bc"); + doc.commit(); + text.mark({ start: 0, end: 3 }, "bold", true); + doc.commit(); + text.unmark({ start: 0, end: 1 }, "bold"); + doc.commit(); + text.delete(3, 1); + doc.commit(); + }, + "richtext-checkout-mark"(doc, versions) { + const text = doc.getText("text"); + text.insert(0, "abc"); + doc.commit(); + versions.push(doc.frontiers()); + text.mark({ start: 0, end: 3 }, "bold", true); + text.insert(3, "d"); + doc.commit(); + }, +}; + +for (const [name, edit] of Object.entries(scenarios)) { + const doc = newDoc(); + const versions = []; + edit(doc, versions); + const checkouts = versions.map((frontiers) => { + doc.checkout(frontiers); + const delta = doc.getText("text").toDelta(); + doc.checkoutToLatest(); + return { frontiers, delta }; + }); + writeFileSync( + fixture(`${name}.json`), + `${JSON.stringify( + { + delta: doc.getText("text").toDelta(), + checkouts, + updates: doc.exportJsonUpdates(), + }, + null, + 2, + )}\n`, + ); + writeFileSync(fixture(`${name}.blob`), doc.export({ mode: "update" })); + writeFileSync(fixture(`${name}.snapshot.blob`), doc.export({ mode: "snapshot" })); + console.log(name, JSON.stringify(doc.getText("text").toDelta())); +} diff --git a/loro-js/src/runtime/containers.ts b/loro-js/src/runtime/containers.ts index caa37a646..6c4b41fd2 100644 --- a/loro-js/src/runtime/containers.ts +++ b/loro-js/src/runtime/containers.ts @@ -590,6 +590,8 @@ export class LoroList extends LoroContainer { export interface TextElement extends SequenceElement { value: string; + /** Set on a zero-width rich-text style anchor. */ + anchor?: TextAnchor | undefined; attributes?: Map | undefined; attributeMeta?: Map | undefined; attributeHistory?: Map | undefined; @@ -1361,6 +1363,29 @@ export interface TextStyleMeta { readonly value: RuntimeValue; } +/** + * A rich-text mark. As in Rust, its start and end anchors are zero-width + * elements of the text sequence, and the style covers every element between + * them, including elements inserted there later. See + * context/loro-js-richtext-anchors.md. + */ +export interface TextStyle extends TextStyleMeta { + readonly key: string; + /** + * The mark op's end entity position in its causal view; the end anchor goes + * to `end + 1` once the start anchor is in place. + */ + readonly end: number; +} + +export interface TextAnchor { + readonly style: TextStyle; + readonly isEnd: boolean; +} + +/** Style info byte of detached marks: Rust's `TextStyleInfoFlag::BOLD`. */ +const DETACHED_STYLE_INFO = 0x84; + export class LoroText extends LoroContainer { readonly _sequence = new SequenceIndex( (element) => ({ @@ -1370,12 +1395,13 @@ export class LoroText extends LoroContainer { (element) => countLineBreaks(element.value), ); _detachedCounter = 0; - _detachedStyleCounter = 0; _attributeHistoryComplete = true; - readonly _styleIndex = new TextStyleIndex(); + readonly _styleIndex = new TextStyleIndex(); _styleVersion: CausalVersion | undefined; + /** Start-anchor counters by peer, ascending; the end anchor is the next counter. */ + readonly #anchorStarts = new Map(); readonly #attributeValuesCache = new WeakMap< - ReadonlyMap, + ReadonlyMap, ReadonlyMap >(); @@ -1387,7 +1413,11 @@ export class LoroText extends LoroContainer { /** Rebuilds fragmented physical text spans without changing CRDT history. */ compact(): void { this._ensureHydrated(); - this._sequence.compact((elements) => TextSequenceSpan.fromElements(elements)); + this._sequence.compact((elements) => + elements.some((element) => element.anchor !== undefined) + ? undefined + : TextSequenceSpan.fromElements(elements), + ); } kind(): "Text" { @@ -1411,10 +1441,10 @@ export class LoroText extends LoroContainer { if (!Number.isSafeInteger(line) || line < 0 || line >= this.lineCount) { return undefined; } - const unicodePosition = this._sequence.visibleIndexAfterLineBreaks(line); - return unicodePosition === undefined + const entity = this._sequence.visibleIndexAfterLineBreaks(line); + return entity === undefined ? undefined - : this._sequence.metricOffsetAtVisibleIndex(unicodePosition, "utf16"); + : this._sequence.metricOffsetAtVisibleIndex(entity, "utf16"); } /** Returns the zero-based line containing a UTF-16 position. */ @@ -1423,10 +1453,10 @@ export class LoroText extends LoroContainer { if (!Number.isSafeInteger(position) || position < 0 || position > this.length) { return undefined; } - const unicodePosition = this.convertPos(position, "utf16", "unicode"); - return unicodePosition === undefined + const entity = this.#entityAt(position); + return entity === undefined ? undefined - : this._sequence.lineBreakOffsetAtVisibleIndex(unicodePosition); + : this._sequence.lineBreakOffsetAtVisibleIndex(entity); } /** Returns a line without its LF or CRLF terminator. */ @@ -1444,6 +1474,7 @@ export class LoroText extends LoroContainer { return this._stringRange(0, this._sequence.visibleLength); } + /** Text of the visible entity range; style anchors contribute nothing. */ _stringRange(start: number, end: number): string { this._ensureHydrated(); const chunks: string[] = []; @@ -1503,6 +1534,7 @@ export class LoroText extends LoroContainer { if (!(span instanceof TextSequenceSpan)) { for (let offset = spanStart; offset < spanEnd; offset += 1) { const element = span.elementAt(offset); + if (element.anchor !== undefined) continue; if (!append(element.value, element.id.peer, element.id.counter)) { return false; } @@ -1538,13 +1570,15 @@ export class LoroText extends LoroContainer { previousRunCounter, ); }, - (element) => append(element.value, element.id.peer, element.id.counter), + (element) => + element.anchor !== undefined || + append(element.value, element.id.peer, element.id.counter), ); if (!stopped && chunk.length > 0) callback(chunk.join("")); } insert(pos: number, text: string): void { - const unicodePosition = this._validateInsertPosition(pos); + const entity = this.#insertPosition(pos); if (text.length === 0) return; if (this._doc === undefined) { const span = TextSequenceSpan.fromText( @@ -1555,10 +1589,11 @@ export class LoroText extends LoroContainer { undefined, ); this._detachedCounter += span.length; - this._sequence.insertSpanAtVisible(unicodePosition, span); + this._sequence.insertSpanAtVisible(entity, span); + this.#inheritGapStyles(span.idAt(0), span.length); return; } - this._doc._textInsert(this, unicodePosition, text); + this._doc._textInsert(this, entity, text); } push(text: string): void { @@ -1574,14 +1609,21 @@ export class LoroText extends LoroContainer { delete(pos: number, len: number): void { validateRange(pos, len, this.length); if (len === 0) return; - const start = this._unicodePosition(pos); - const end = this._unicodePosition(pos + len); + this.#requireBoundary(pos); + this.#requireBoundary(pos + len); if (this._doc === undefined) { - for (const element of this._sequence.visibleRange(start, end)) - this._sequence.setDeleted(element, true); + for (const { run } of this._deleteRuns(pos, pos + len)) { + for (let offset = 0; offset < run.length; offset += 1) { + const element = this._sequence.findById({ + peer: run.start.peer, + counter: run.start.counter + offset, + }); + if (element !== undefined) this._sequence.setDeleted(element, true); + } + } return; } - this._doc._textDelete(this, start, end - start); + this._doc._textDelete(this, pos, len); } deleteUtf8(index: number, length: number): void { @@ -1594,14 +1636,14 @@ export class LoroText extends LoroContainer { slice(start: number, end: number): string { validateRange(start, end - start, this.length); - const unicodeStart = this._unicodePosition(start); - const unicodeEnd = this._unicodePosition(end); - return this._stringRange(unicodeStart, unicodeEnd); + return this._stringRange(this.#requireBoundary(start), this.#requireBoundary(end)); } charAt(pos: number): string { if (pos === this.length) return ""; - return this._sequence.atVisible(this._unicodePosition(pos))?.value ?? ""; + this.#requireBoundary(pos); + const entity = this._sequence.visibleIndexOfMetricUnit(pos, "utf16"); + return entity === undefined ? "" : (this._sequence.atVisible(entity)?.value ?? ""); } splice(pos: number, len: number, text: string): string { @@ -1611,65 +1653,71 @@ export class LoroText extends LoroContainer { return removed; } + /** + * Marks a UTF-16 range like Rust's `TextHandler::mark`: both ends go where + * text inserted there would go, and a mark that would not change any + * position's value (or an unmark of a key the range never had) is skipped. + */ mark(range: { start: number; end: number }, key: string, value: unknown): void { - validateRange(range.start, range.end - range.start, this.length); - const start = this._unicodePosition(range.start); - const end = this._unicodePosition(range.end); + this._ensureHydrated(); + const { start, end } = range; + if (!Number.isSafeInteger(start) || !Number.isSafeInteger(end) || start >= end) { + throw new RangeError("mark start must be less than its end"); + } + validateRange(start, end - start, this.length); + const entityStart = this.#insertPosition(start); + const entityEnd = this.#insertPosition(end); + const normalized = normalizeDetachedValue(value); + if (this.#markIsRedundant(entityStart, entityEnd, key, normalized)) return; if (this._doc === undefined) { - this._applyMark(start, end, key, normalizeDetachedValue(value)); + this.#markDetached(entityStart, entityEnd, key, normalized); return; } - this._doc._textMark(this, start, end, key, value); + this._doc._textMark(this, entityStart, entityEnd, key, value); } unmark(range: { start: number; end: number }, key: string): void { - validateRange(range.start, range.end - range.start, this.length); - const start = this._unicodePosition(range.start); - const end = this._unicodePosition(range.end); - if ( - !this._styleIndex.rangeHasKey(this._styleRuns(start, end), key, this._styleVersion) - ) { - return; - } this.mark(range, key, null); } + /** + * Applies a Quill delta like Rust's `apply_delta`: inserted text without an + * attribute explicitly drops the style it would inherit, and every attribute + * is marked after the whole delta is applied. + */ applyDelta(delta: readonly Delta[]): void { let position = 0; const marks: { readonly start: number; readonly end: number; - readonly attributes: Readonly>; + readonly attributes: readonly (readonly [string, unknown])[]; }[] = []; for (const operation of delta) { if ("insert" in operation) { - this.insert(position, operation.insert); const length = operation.insert.length; - if (operation.attributes !== undefined) { - marks.push({ - start: position, - end: position + length, - attributes: operation.attributes, - }); + if (length === 0) continue; + if (position > this.length) { + throw new RangeError(`delta position ${position} is out of range`); } + const attributes = operation.attributes ?? {}; + const overrides = this.#insertOverrides(position, attributes); + this.insert(position, operation.insert); + marks.push({ start: position, end: position + length, attributes: overrides }); position += length; } else if ("delete" in operation) { this.delete(position, operation.delete); } else { - if (operation.attributes !== undefined) { - marks.push({ - start: position, - end: position + operation.retain, - attributes: operation.attributes, - }); + const end = position + operation.retain; + if (end > this.length) { + throw new RangeError(`delta range ${position}..${end} is out of range`); } - position += operation.retain; + const entries = Object.entries(operation.attributes ?? {}); + if (entries.length > 0) marks.push({ start: position, end, attributes: entries }); + position = end; } } for (const { start, end, attributes } of marks) { - for (const [key, value] of Object.entries(attributes)) { - this.mark({ start, end }, key, value); - } + for (const [key, value] of attributes) this.mark({ start, end }, key, value); } } @@ -1680,10 +1728,11 @@ export class LoroText extends LoroContainer { sliceDelta(start: number, end: number): Delta[] { this._ensureHydrated(); validateRange(start, end - start, this.length); - const unicodeStart = this._unicodePosition(start); - const unicodeEnd = this._unicodePosition(end); return textElementsToDelta( - this._sequence.visibleRange(unicodeStart, unicodeEnd), + this._sequence.visibleRange( + this.#requireBoundary(start), + this.#requireBoundary(end), + ), this.#attributeResolver(), ); } @@ -1701,25 +1750,8 @@ export class LoroText extends LoroContainer { convertPos(index: number, from: TextPosType, to: TextPosType): number | undefined { this._ensureHydrated(); if (!isTextPosType(from) || !isTextPosType(to)) return undefined; - const visibleLength = this._sequence.visibleLength; - const directMetric = - from === "unicode" || - (from === "utf16" && this._sequence.visibleUtf16Length === visibleLength) || - (from === "utf8" && this._sequence.visibleUtf8Length === visibleLength); - const unicodeIndex = directMetric - ? Number.isSafeInteger(index) && index >= 0 && index <= visibleLength - ? index - : undefined - : this._sequence.visibleIndexAtMetricOffset(index, from); - if (unicodeIndex === undefined) return undefined; - if ( - to === "unicode" || - (to === "utf16" && this._sequence.visibleUtf16Length === visibleLength) || - (to === "utf8" && this._sequence.visibleUtf8Length === visibleLength) - ) { - return unicodeIndex; - } - return this._sequence.metricOffsetAtVisibleIndex(unicodeIndex, to); + const entity = this.#entityFrom(index, from); + return entity === undefined ? undefined : this.#offsetAt(entity, to); } /** @@ -1733,21 +1765,14 @@ export class LoroText extends LoroContainer { const length = this.length; if (length === 0) return new Cursor(this.id, undefined, side === 0 ? -1 : side, 0); if (pos >= length) return new Cursor(this.id, undefined, 1, length); + const element = this.#elementAt(pos); const unicodePosition = this.convertPos(pos, "utf16", "unicode"); - if (unicodePosition === undefined) return undefined; - return new Cursor( - this.id, - this._sequence.atVisible(unicodePosition)!.id, - side, - unicodePosition, - ); + if (element === undefined || unicodePosition === undefined) return undefined; + return new Cursor(this.id, element.id, side, unicodePosition); } getEditorOf(pos: number): string | undefined { - const unicodePosition = this.convertPos(pos, "utf16", "unicode"); - return unicodePosition === undefined - ? undefined - : this._sequence.atVisible(unicodePosition)?.id.peer.toString(); + return this.#elementAt(pos)?.id.peer.toString(); } update(text: string, _options?: TextUpdateOptions): void { @@ -1766,6 +1791,7 @@ export class LoroText extends LoroContainer { return this.toString(); } + /** Visible elements in order, style anchors included. */ _visibleElements(): TextElement[] { this._ensureHydrated(); return this._sequence.visible(); @@ -1781,23 +1807,32 @@ export class LoroText extends LoroContainer { return this._sequence.atVisible(position); } - _insertVisible( - position: number, - characters: readonly string[], - ids: readonly CodecId[], - lamports: readonly number[], - ): void { - this._sequence.insertAtVisible( - position, - characters.map((value, index) => ({ - value, - id: ids[index]!, - lamport: lamports[index]!, - deleted: false, - originLeft: undefined, - originRight: undefined, - })), - ); + /** + * Appends snapshot state: text scalars and style anchors in document order, + * all visible. Styles cover the elements between their anchors. + */ + _appendElements(elements: readonly TextElement[]): void { + if (elements.length === 0) return; + const base = this._sequence.allLength; + this._sequence.insertAtPhysical(base, elements); + const starts = new Map(); + for (let index = 0; index < elements.length; index += 1) { + const element = elements[index]!; + const anchor = element.anchor; + if (anchor === undefined) continue; + this.#registerAnchor(element); + if (!anchor.isEnd) { + starts.set(anchor.style, base + index); + continue; + } + const start = starts.get(anchor.style); + if (start === undefined) continue; + this._styleIndex.add( + this._sequence.physicalIdRuns(start, base + index + 1), + anchor.style.key, + anchor.style, + ); + } } _insertFugue( @@ -1807,37 +1842,13 @@ export class LoroText extends LoroContainer { lamport: number, causalVersion: CausalVersion, ): void { - const current = this._sequence.isFullyIncluded(causalVersion); - const authored = current ? undefined : this._sequence.causalView(causalVersion); - const authoredLength = current ? this._sequence.visibleLength : authored!.length; - const authoredPosition = Math.min(position, authoredLength); - const currentContext = current - ? this._sequence.visibleInsertionContext(authoredPosition) - : undefined; - const authoredAt = (index: number): TextElement | undefined => - current ? this._sequence.atVisible(index) : authored!.at(index); - const authoredLeft = currentContext?.left ?? authoredAt(authoredPosition - 1); const insertion = fugueInsertion( this._sequence, position, startId, causalVersion, true, - currentContext ?? { current: false, left: authoredLeft }, ); - let inherited: Map | undefined; - if (!this._styleIndex.isEmpty) { - inherited = new Map(); - for (const [key, meta] of this._attributeMetasAt(authoredLeft, causalVersion)) { - if ((meta.info & 0b100) !== 0) inherited.set(key, meta); - } - for (const [key, meta] of this._attributeMetasAt( - authoredAt(authoredPosition), - causalVersion, - )) { - if ((meta.info & 0b010) !== 0) inherited.set(key, meta); - } - } const firstCodePoint = text.codePointAt(0)!; const singleScalar = text.length === 1 || (text.length === 2 && firstCodePoint > 0xffff); @@ -1866,14 +1877,120 @@ export class LoroText extends LoroContainer { recordFugueInsertionRun(this._sequence, insertion, startId); insertedLength = span.length; } - if (inherited !== undefined && inherited.size > 0) { - const insertedRun = [{ start: { ...startId }, length: insertedLength }]; - for (const [key, meta] of inherited) { - this._styleIndex.add(insertedRun, key, meta); + this.#inheritGapStyles(startId, insertedLength, insertion.insertIndex); + } + + /** Inserts a mark's start anchor at its entity position in the causal view. */ + _applyStyleStart( + position: number, + style: TextStyle, + lamport: number, + causalVersion: CausalVersion, + ): void { + this.#insertAnchor( + position, + { + value: "", + id: { peer: style.startId.peer, counter: style.startId.counter }, + lamport, + deleted: false, + originLeft: undefined, + originRight: undefined, + anchor: { style, isEnd: false }, + }, + causalVersion, + ); + } + + /** + * Inserts a mark's end anchor at `end + 1` in the causal view (which now + * contains the start anchor) and applies the style to everything between. + */ + _applyStyleEnd( + id: CodecId, + lamport: number, + causalVersion: CausalVersion, + ): TextStyle | undefined { + const start = this._sequence.findById({ peer: id.peer, counter: id.counter - 1 }); + const anchor = start?.anchor; + if (start === undefined || anchor === undefined || anchor.isEnd) return undefined; + const style = anchor.style; + const end: TextElement = { + value: "", + id: { peer: id.peer, counter: id.counter }, + lamport, + deleted: false, + originLeft: undefined, + originRight: undefined, + anchor: { style, isEnd: true }, + }; + this.#insertAnchor(style.end + 1, end, causalVersion); + this.#applyStyleRange(start, end, style); + if (this._styleVersion !== undefined) { + // A detached document edited in place keeps showing its own new style. + const next = new Map(this._styleVersion); + next.set(id.peer, Math.max(next.get(id.peer) ?? 0, id.counter + 1)); + this._styleVersion = next; + } + return style; + } + + /** ID runs of every element a style covers, its anchors included. */ + _styleMemberRuns(style: TextStyle): SequenceIdRun[] { + const start = this._sequence.findById(style.startId); + const end = this._sequence.findById({ + peer: style.startId.peer, + counter: style.startId.counter + 1, + }); + if (start === undefined || end === undefined) return []; + const from = this._sequence.physicalIndexOf(start); + const to = this._sequence.physicalIndexOf(end); + if (from === undefined || to === undefined || from > to) return []; + return this._sequence.physicalIdRuns(from, to + 1); + } + + /** Runs of the elements whose attribute an applied style changes. */ + _styleChangeRuns(style: TextStyle): SequenceIdRun[] { + return this._styleIndex.winningRuns( + this._styleMemberRuns(style), + style.key, + style, + this._styleVersion, + styleValuesEqual, + ); + } + + /** + * Undoes a style like Rust's undo of its op: wherever the style changed the + * resolved value (from `before` to `after`, the versions around its op), the + * text gets the `before` value back (`null` drops the attribute). + */ + _undoStyle(style: TextStyle, before: CausalVersion, after: CausalVersion): void { + const changes = this._styleIndex + .transitions(this._styleMemberRuns(style), style.key, before, after) + .filter( + ({ before: previous, after: next }) => + !styleValuesEqual(previous?.value ?? null, next?.value ?? null), + ) + .flatMap(({ run, before: previous }) => + this._sequence + .visibleMetricRangesForIdRuns([run], "utf16") + .map((range) => ({ range, value: previous?.value ?? null })), + ) + .sort((left, right) => left.range.start - right.range.start); + for (const { range, value } of changes) { + if (range.start < range.end) { + this.mark({ start: range.start, end: range.end }, style.key, value); } } } + /** The anchored style whose start anchor has this ID. */ + _styleAt(startId: CodecId): TextStyle | undefined { + const anchor = this._sequence.findById(startId)?.anchor; + return anchor === undefined || anchor.isEnd ? undefined : anchor.style; + } + _deleteIdSpan(startId: CodecId, length: number, deletedBy?: CodecId): void { if (Math.abs(length) === 1) { const element = this._sequence.findById(startId); @@ -1942,54 +2059,68 @@ export class LoroText extends LoroContainer { } } - _applyMark( - start: number, - end: number, - key: string, - value: RuntimeValue, - meta?: TextStyleMeta, - causalVersion?: CausalVersion, - ): void { - const appliedMeta = - meta ?? - ({ - startId: { peer: -1n, counter: this._detachedStyleCounter }, - lamport: this._detachedStyleCounter++, - info: 0, - value, - } satisfies TextStyleMeta); - this._styleIndex.add(this._styleRuns(start, end, causalVersion), key, appliedMeta); - if (this._styleVersion !== undefined) { - const next = new Map(this._styleVersion); - next.set( - appliedMeta.startId.peer, - Math.max( - next.get(appliedMeta.startId.peer) ?? 0, - appliedMeta.startId.counter + 1, - ), - ); - this._styleVersion = next; + /** + * Text runs covering a UTF-16 range, like Rust's `get_text_entity_ranges`: + * each run is consecutive in both entity position and ID. Style anchors are + * never deleted and split runs. Positions are entity indexes in the current + * state. + */ + _deleteRuns(start: number, end: number): { position: number; run: SequenceIdRun }[] { + if (start >= end) return []; + const sequence = this._sequence; + if (sequence.visibleZeroWidthLength === 0) { + const first = this.#entityAt(start); + const stop = this.#entityAt(end); + if (first === undefined || stop === undefined) return []; + let position = first; + return sequence.visibleIdRuns(first, stop).map((run) => { + const entry = { position, run }; + position += run.length; + return entry; + }); } - } - - _styleRuns(start: number, end: number, causalVersion?: CausalVersion): SequenceIdRun[] { - if (causalVersion === undefined || this._sequence.isFullyIncluded(causalVersion)) { - return this._sequence.visibleIdRuns(start, end); + const first = sequence.visibleIndexOfMetricUnit(start, "utf16"); + const last = sequence.visibleIndexOfMetricUnit(end - 1, "utf16"); + if (first === undefined || last === undefined) return []; + const output: { position: number; run: SequenceIdRun }[] = []; + let position = first; + for (const run of sequence.visibleIdRuns(first, last + 1)) { + const peer = run.start.peer; + let counter = run.start.counter; + const runEnd = counter + run.length; + for (const anchor of this.#anchorCounters(peer, counter, runEnd)) { + if (anchor > counter) { + output.push({ + position, + run: { start: { peer, counter }, length: anchor - counter }, + }); + position += anchor - counter; + } + position += 1; + counter = anchor + 1; + } + if (counter < runEnd) { + output.push({ + position, + run: { start: { peer, counter }, length: runEnd - counter }, + }); + position += runEnd - counter; + } } - return this._sequence.causalView(causalVersion).idRuns(start, end); + return output; } _attributeHistoryAt( element: TextElement, key: string, - ): readonly TextStyleMeta[] | undefined { + ): readonly TextStyle[] | undefined { return this._styleIndex.historyAt(element.id, key); } _attributeMetasAt( element: TextElement | undefined, version: CausalVersion | undefined = this._styleVersion, - ): ReadonlyMap { + ): ReadonlyMap { return element === undefined ? new Map() : this._styleIndex.metasAt(element.id, version); @@ -1997,7 +2128,7 @@ export class LoroText extends LoroContainer { _attributeMetasResolver( version: CausalVersion | undefined = this._styleVersion, - ): (element: TextElement) => ReadonlyMap { + ): (element: TextElement) => ReadonlyMap { const metasAt = this._styleIndex.resolver(version); return (element) => metasAt(element.id); } @@ -2010,13 +2141,27 @@ export class LoroText extends LoroContainer { this._styleVersion = version === undefined ? undefined : new Map(version); } + _ensureLineIndex(): void { + this._ensureHydrated(); + this._sequence.enableLineBreaks(); + } + + _reset(): void { + this._sequence.reset(); + this._detachedCounter = 0; + this._attributeHistoryComplete = true; + this._styleIndex.reset(); + this._styleVersion = undefined; + this.#anchorStarts.clear(); + } + #attributeResolver(): (element: TextElement) => ReadonlyMap { const metasAt = this._attributeMetasResolver(); return (element) => this.#attributeValues(metasAt(element)); } #attributeValues( - metas: ReadonlyMap, + metas: ReadonlyMap, ): ReadonlyMap { const cached = this.#attributeValuesCache.get(metas); if (cached !== undefined) return cached; @@ -2028,37 +2173,363 @@ export class LoroText extends LoroContainer { return attributes; } - _validateInsertPosition(position: number): number { - this._ensureHydrated(); - const unicodePosition = this.convertPos(position, "utf16", "unicode"); - if (unicodePosition === undefined) { - throw new RangeError(`text position ${position} is out of range`); + /** An entity index at a UTF-16 boundary; any one within a run of anchors. */ + #entityAt(position: number): number | undefined { + return this.#entityFrom(position, "utf16"); + } + + #entityFrom(index: number, from: TextPosType): number | undefined { + const sequence = this._sequence; + const length = sequence.visibleLength; + if (sequence.visibleZeroWidthLength === 0) { + const direct = + from === "unicode" || + (from === "utf16" && sequence.visibleUtf16Length === length) || + (from === "utf8" && sequence.visibleUtf8Length === length); + if (direct) { + return Number.isSafeInteger(index) && index >= 0 && index <= length + ? index + : undefined; + } + } + return from === "unicode" + ? sequence.visibleIndexOfWidthElement(index) + : sequence.visibleIndexAtMetricOffset(index, from); + } + + #offsetAt(entity: number, to: TextPosType): number | undefined { + const sequence = this._sequence; + if (to === "unicode") { + return sequence.visibleZeroWidthLength === 0 + ? entity + : entity - sequence.zeroWidthBeforeVisibleIndex(entity)!; } - return unicodePosition; + if ( + sequence.visibleZeroWidthLength === 0 && + (to === "utf16" ? sequence.visibleUtf16Length : sequence.visibleUtf8Length) === + sequence.visibleLength + ) { + return entity; + } + return sequence.metricOffsetAtVisibleIndex(entity, to); } - _unicodePosition(position: number): number { + #requireBoundary(position: number): number { this._ensureHydrated(); - const unicodePosition = this.convertPos(position, "utf16", "unicode"); - if (unicodePosition === undefined) { + const entity = this.#entityAt(position); + if (entity === undefined) { throw new RangeError(`text position ${position} is not on a UTF-16 boundary`); } - return unicodePosition; + return entity; } - _ensureLineIndex(): void { + /** The text scalar that starts at a UTF-16 position. */ + #elementAt(position: number): TextElement | undefined { this._ensureHydrated(); - this._sequence.enableLineBreaks(); + if (this.#entityAt(position) === undefined) return undefined; + const entity = this._sequence.visibleIndexOfMetricUnit(position, "utf16"); + return entity === undefined ? undefined : this._sequence.atVisible(entity); } - _reset(): void { - this._sequence.reset(); - this._detachedCounter = 0; - this._detachedStyleCounter = 0; - this._attributeHistoryComplete = true; - this._styleIndex.reset(); - this._styleVersion = undefined; + /** + * The entity index where text inserted at a UTF-16 position goes: Rust's + * `find_best_insert_pos`. Among the style anchors at that position, the text + * goes before the first anchor that must stay after it (see + * `anchorStaysAfterInsert`), or after all of them. + */ + #insertPosition(position: number): number { + const entity = this.#entityAt(position); + if (entity === undefined) { + throw new RangeError(`text position ${position} is out of range`); + } + const sequence = this._sequence; + if (sequence.visibleZeroWidthLength === 0) return entity; + let index = + position === 0 ? 0 : sequence.visibleIndexOfMetricUnit(position - 1, "utf16")! + 1; + sequence.forEachVisibleRange(index, sequence.visibleLength, (element) => { + const anchor = element.anchor; + if (anchor === undefined || anchorStaysAfterInsert(anchor)) return false; + index += 1; + return undefined; + }); + return index; + } + + /** + * Whether a mark would change nothing, as Rust checks before writing one: + * every entity position of the range already resolves the key to the value, + * or it is an unmark and no position has the key at all. + */ + #markIsRedundant( + start: number, + end: number, + key: string, + value: RuntimeValue, + ): boolean { + const runs = this._sequence.visibleIdRuns(start, end); + const version = this._styleVersion; + if ( + this._styleIndex.everyWinner(runs, key, version, (meta) => + styleValuesEqual(meta.value, value), + ) + ) { + return true; + } + return value === null && !this._styleIndex.someHasKey(runs, key, version); + } + + /** Detached text marks like Rust's `mark_for_detached`, with local anchors. */ + #markDetached(start: number, end: number, key: string, value: RuntimeValue): void { + const counter = this._detachedCounter; + this._detachedCounter += 2; + const style: TextStyle = { + startId: { peer: 0n, counter }, + lamport: counter, + info: DETACHED_STYLE_INFO, + value, + key, + end, + }; + const startAnchor: TextElement = { + value: "", + id: { peer: 0n, counter }, + lamport: counter, + deleted: false, + originLeft: undefined, + originRight: undefined, + anchor: { style, isEnd: false }, + }; + const endAnchor: TextElement = { + value: "", + id: { peer: 0n, counter: counter + 1 }, + lamport: counter + 1, + deleted: false, + originLeft: undefined, + originRight: undefined, + anchor: { style, isEnd: true }, + }; + this._sequence.insertAtVisible(end, [endAnchor]); + this._sequence.insertAtVisible(start, [startAnchor]); + this.#registerAnchor(startAnchor); + this.#registerAnchor(endAnchor); + this.#inheritGapStyles(startAnchor.id, 1); + this.#inheritGapStyles(endAnchor.id, 1); + this.#applyStyleRange(startAnchor, endAnchor, style); + } + + /** + * Rust's `insert_with_txn_and_attr`: the styles text inserted at a position + * would get, compared with the requested attributes. Returns the marks that + * make the inserted text carry exactly those attributes. + */ + #insertOverrides( + position: number, + attributes: Readonly>, + ): [string, unknown][] { + const entity = this.#insertPosition(position); + const inherited = new Map(); + if (entity > 0 && entity < this._sequence.visibleLength) { + // Rust's `get_styles_for_insert`: the styles shared by both visible + // neighbors, resolved per key. + const left = this._styleIndex.membershipAt( + this._sequence.atVisible(entity - 1)!.id, + ); + const right = this._styleIndex.membershipAt(this._sequence.atVisible(entity)!.id); + if (left !== undefined && right !== undefined) { + const version = this._styleVersion; + for (const [key, history] of intersectHistories(left, right)) { + for (let index = history.length - 1; index >= 0; index -= 1) { + const style = history[index]!; + if ( + version === undefined || + style.startId.counter + 1 < (version.get(style.startId.peer) ?? 0) + ) { + inherited.set(key, style.value); + break; + } + } + } + } + } + const overrides: [string, unknown][] = []; + for (const [key, value] of inherited) { + const requested = attributes[key]; + if ( + Object.hasOwn(attributes, key) && + styleValuesEqual(value, normalizeDetachedValue(requested)) + ) { + continue; + } + overrides.push([key, requested ?? null]); + } + for (const [key, value] of Object.entries(attributes)) { + if (!inherited.has(key)) overrides.push([key, value]); + } + return overrides; + } + + #insertAnchor( + position: number, + element: TextElement, + causalVersion: CausalVersion, + ): void { + const insertion = fugueInsertion( + this._sequence, + position, + element.id, + causalVersion, + true, + ); + element.originLeft = insertion.originLeft; + element.originRight = insertion.originRight; + this._sequence.insertAtPhysical(insertion.insertIndex, [element]); + recordFugueInsertion(this._sequence, insertion, [element.id]); + this.#registerAnchor(element); + this.#inheritGapStyles(element.id, 1, insertion.insertIndex); + } + + #registerAnchor(element: TextElement): void { + if (element.anchor === undefined || element.anchor.isEnd) return; + let starts = this.#anchorStarts.get(element.id.peer); + if (starts === undefined) { + starts = []; + this.#anchorStarts.set(element.id.peer, starts); + } + const counter = element.id.counter; + let low = 0; + let high = starts.length; + while (low < high) { + const middle = (low + high) >>> 1; + if (starts[middle]! < counter) low = middle + 1; + else high = middle; + } + if (starts[low] !== counter) starts.splice(low, 0, counter); + } + + /** Counters of the style anchors of `peer` in `[start, end)`, ascending. */ + #anchorCounters(peer: bigint, start: number, end: number): number[] { + const starts = this.#anchorStarts.get(peer); + if (starts === undefined) return []; + let low = 0; + let high = starts.length; + while (low < high) { + const middle = (low + high) >>> 1; + if (starts[middle]! + 1 < start) low = middle + 1; + else high = middle; + } + const output: number[] = []; + for (let index = low; index < starts.length && starts[index]! < end; index += 1) { + const counter = starts[index]!; + if (counter >= start) output.push(counter); + if (counter + 1 >= start && counter + 1 < end) output.push(counter + 1); + } + return output; + } + + /** + * Gives newly inserted elements every style that covers their physical gap: + * the styles shared by the physical neighbors, which is exactly the set of + * styles whose start anchor precedes and whose end anchor follows the gap. + */ + #inheritGapStyles(startId: CodecId, length: number, insertIndex?: number): void { + if (this._styleIndex.isEmpty) return; + const sequence = this._sequence; + let index = insertIndex; + if (index === undefined) { + const first = sequence.findByIdRaw(startId); + index = first === undefined ? undefined : sequence.physicalIndexOf(first); + if (index === undefined) return; + } + if (index === 0) return; + const left = sequence.atPhysicalRaw(index - 1); + const right = sequence.atPhysicalRaw(index + length); + if (left === undefined || right === undefined) return; + const leftStyles = this._styleIndex.membershipAt(left.id); + const rightStyles = this._styleIndex.membershipAt(right.id); + if (leftStyles === undefined || rightStyles === undefined) return; + const shared = + leftStyles === rightStyles + ? leftStyles + : intersectHistories(leftStyles, rightStyles); + if (shared.size > 0) { + this._styleIndex.addMembership([{ start: startId, length }], shared); + } + } + + #applyStyleRange(start: TextElement, end: TextElement, style: TextStyle): void { + const from = this._sequence.physicalIndexOf(start); + const to = this._sequence.physicalIndexOf(end); + if (from === undefined || to === undefined || from > to) return; + this._styleIndex.add(this._sequence.physicalIdRuns(from, to + 1), style.key, style); + } +} + +/** + * Rust's `prefer_insert_before` plus its first rule: text inserted at an + * anchor's position goes before a start anchor of an unmark (a null or false + * value), before a start anchor whose style does not expand before, and before + * an end anchor whose style expands after. + */ +function anchorStaysAfterInsert(anchor: TextAnchor): boolean { + const { style, isEnd } = anchor; + if (isEnd) return (style.info & 0b100) !== 0; + return style.value === null || style.value === false || (style.info & 0b010) === 0; +} + +function intersectHistories( + left: ReadonlyMap, + right: ReadonlyMap, +): Map { + const shared = new Map(); + for (const [key, history] of left) { + const other = right.get(key); + if (other === undefined) continue; + const members = new Set(other); + const common = history.filter((meta) => members.has(meta)); + if (common.length > 0) shared.set(key, common); + } + return shared; +} + +/** Deep equality of style values; `undefined` equals `null`. */ +function styleValuesEqual(left: unknown, right: unknown): boolean { + if (left === undefined) left = null; + if (right === undefined) right = null; + if (left === right) return true; + if (left instanceof Uint8Array && right instanceof Uint8Array) { + return ( + left.length === right.length && left.every((byte, index) => byte === right[index]) + ); + } + if (Array.isArray(left) && Array.isArray(right)) { + return ( + left.length === right.length && + left.every((item, index) => styleValuesEqual(item, right[index])) + ); + } + if ( + typeof left === "object" && + typeof right === "object" && + left !== null && + right !== null && + !Array.isArray(left) && + !Array.isArray(right) && + !(left instanceof Uint8Array) && + !(right instanceof Uint8Array) && + !isContainer(left) && + !isContainer(right) + ) { + const leftEntries = Object.entries(left); + const rightRecord = right as Record; + return ( + leftEntries.length === Object.keys(rightRecord).length && + leftEntries.every( + ([key, value]) => + Object.hasOwn(rightRecord, key) && styleValuesEqual(value, rightRecord[key]), + ) + ); } + return false; } export class LoroMovableList extends LoroList { @@ -3415,6 +3886,7 @@ function textElementsToDelta( values = []; }; for (const element of elements) { + if (element.anchor !== undefined) continue; const nextAttributes = attributesAt(element); if (values.length > 0 && !textAttributesEqual(attributes, nextAttributes)) { flush(); @@ -3443,6 +3915,7 @@ function isTextPosType(value: string): value is TextPosType { } function utf8CodePointLength(value: string): number { + if (value.length === 0) return 0; const codePoint = value.codePointAt(0) ?? 0; if (codePoint <= 0x7f) return 1; if (codePoint <= 0x7ff) return 2; diff --git a/loro-js/src/runtime/document.ts b/loro-js/src/runtime/document.ts index a28d1d17f..422fb2261 100644 --- a/loro-js/src/runtime/document.ts +++ b/loro-js/src/runtime/document.ts @@ -69,7 +69,7 @@ import { type SequenceMoveMeta, type SequenceValueMeta, type TextElement, - type TextStyleMeta, + type TextStyle, type TreeNodeRecord, } from "./containers"; import { SequenceEventDiff } from "./event-diff"; @@ -2195,7 +2195,22 @@ export class LoroDoc = Record = Record 0) positions.push({ start: offset, length }); + } else { + const index = container._sequence.visibleIndexOf(element as never); + if (index !== undefined) positions.push({ start: index, length: 1 }); + } } - positions.sort((left, right) => left - right); - const ranges: { start: number; length: number }[] = []; + positions.sort((left, right) => left.start - right.start); for (const position of positions) { const previous = ranges.at(-1); - if (previous !== undefined && previous.start + previous.length === position) { - previous.length += 1; + if (previous !== undefined && previous.start + previous.length === position.start) { + previous.length += position.length; } else { - ranges.push({ start: position, length: 1 }); + ranges.push({ ...position }); } } for (const selected of ranges.reverse()) { @@ -2527,7 +2552,7 @@ export class LoroDoc = Record = Record { + const style = container._styleAt({ + peer: changeId.peer, + counter: operation.counter - 1, + }); + if (style === undefined) return; + const value = + style.value === null ? null : (runtimeValueToJson(style.value) as Value); + for (const range of container._sequence.visibleMetricRangesForIdRuns( + container._styleChangeRuns(style), + "utf16", + )) { + state.diff.formatText(range.start, range.end - range.start, style.key, value); + } + }; } - if (content.type === "text-mark-end") return undefined; } else if (container instanceof LoroList) { const state = this.#sequenceEventState(recording, container, "list"); if (content.type === "list-insert" || content.type === "movable-list-insert") { @@ -3178,19 +3201,20 @@ export class LoroDoc = Record= 0; index -= 1) { - const run = runs[index]!; - end -= run.length; + const { position: entity, run } = runs[index]!; this.#appendAndApply( container, { type: "text-delete", - position: end, + position: entity, length: BigInt(run.length), startId: run.start, }, @@ -3417,31 +3441,30 @@ export class LoroDoc = Record = Record viewLength) { - return false; - } - const runs = container._styleRuns(content.start, content.end, causalVersion); if ( - !container._styleIndex.runsContainMeta(runs, content.key, { - peer: change.id.peer, - counter: operation.counter, - }) + !container._sequence.containsIdRuns([ + { start: { peer: change.id.peer, counter: operation.counter }, length: 1 }, + ]) ) { return false; } - } else if (content.type === "text-mark-end") { - if (!(container instanceof LoroText)) return false; } else if (content.type === "text-insert") { if (!(container instanceof LoroText)) return false; if ( @@ -3979,8 +3994,23 @@ export class LoroDoc = Record = Record = Record = Record right.position - left.position); for (const { element, position } of removals) { - state?.diff.delete( - position, - container instanceof LoroText ? (element.value as string).length : 1, - ); + const length = + container instanceof LoroText ? (element.value as string).length : 1; + // Style anchors have no width and no event of their own. + if (length > 0) state?.diff.delete(position, length); container._sequence.setDeleted(element as never, true); } @@ -4240,6 +4266,7 @@ export class LoroDoc = Record = Record(); - for (let index = 0; index <= visible.length; index += 1) { - const current = index < visible.length ? metasAt(visible[index]!) : new Map(); - for (const [key, meta] of active) { - if (current.get(key) === meta) continue; - spans.push({ - peerIndex: peerIndex(meta.startId.peer), - counter: meta.startId.counter + 1, - lamportSub: meta.lamport - meta.startId.counter, - length: -1, - }); - } - for (const [key, meta] of current) { - if (active.get(key) === meta) continue; + // Rust's `encode_snapshot_fast`: text and style anchors in document + // order. A start anchor is a zero-length span followed by its mark, an + // end anchor a span of length -1 at the start counter + 1 that keeps the + // start's lamport offset. + const text: string[] = []; + for (const element of visible) { + const anchor = element.anchor; + if (anchor !== undefined) { + const style = anchor.style; spans.push({ - peerIndex: peerIndex(meta.startId.peer), - counter: meta.startId.counter, - lamportSub: meta.lamport - meta.startId.counter, - length: 0, - }); - marks.push({ - keyIndex: keyIndex(key), - value: this.#encodeSnapshotValue(meta.value), - info: meta.info, + peerIndex: peerIndex(style.startId.peer), + counter: style.startId.counter + (anchor.isEnd ? 1 : 0), + lamportSub: style.lamport - style.startId.counter, + length: anchor.isEnd ? -1 : 0, }); + if (!anchor.isEnd) { + marks.push({ + keyIndex: keyIndex(style.key), + value: this.#encodeSnapshotValue(style.value), + info: style.info, + }); + } + continue; } - active = new Map(current); - if (index < visible.length) { - const element = visible[index]!; + text.push(element.value); + const index = peerIndex(element.id.peer); + const lamportSub = element.lamport - element.id.counter; + const previous = spans.at(-1); + if ( + previous !== undefined && + previous.length > 0 && + previous.peerIndex === index && + previous.lamportSub === lamportSub && + previous.counter + previous.length === element.id.counter + ) { + previous.length += 1; + } else { spans.push({ - peerIndex: peerIndex(element.id.peer), + peerIndex: index, counter: element.id.counter, - lamportSub: element.lamport - element.id.counter, + lamportSub, length: 1, }); } } return { kind: CodecContainerType.Text, - text: visible.map((element) => element.value).join(""), + text: text.join(""), peers, spans, keys, @@ -5930,69 +5964,66 @@ export class LoroDoc = Record(); - const active = new Map(); + const elements: TextElement[] = []; + const styles = new Map(); let characterIndex = 0; let markIndex = 0; for (const span of state.spans) { const peer = state.peers[Number(span.peerIndex)]!; + const lamport = span.counter + span.lamportSub; if (span.length === 0) { const mark = state.marks[markIndex++]!; - const key = state.keys[mark.keyIndex]!; - const value = this.#decodeSnapshotValue(mark.value); - const meta: TextStyleMeta = { + const style: TextStyle = { startId: { peer, counter: span.counter }, - lamport: span.counter + span.lamportSub, + lamport, info: mark.info, - value, + value: this.#decodeSnapshotValue(mark.value), + key: state.keys[mark.keyIndex]!, + end: -1, }; - stylesById.set(idKey(meta.startId), { key, meta }); - const stack = active.get(key) ?? []; - stack.push(meta); - active.set(key, stack); + styles.set(idKey(style.startId), style); + elements.push({ + value: "", + id: { peer, counter: span.counter }, + lamport, + deleted: false, + originLeft: undefined, + originRight: undefined, + anchor: { style, isEnd: false }, + }); continue; } if (span.length === -1) { - const style = stylesById.get(idKey({ peer, counter: span.counter - 1 })); - if (style !== undefined) { - const stack = active.get(style.key); - if (stack !== undefined) { - const index = stack.lastIndexOf(style.meta); - if (index >= 0) stack.splice(index, 1); - if (stack.length === 0) active.delete(style.key); - } - } + const style = styles.get(idKey({ peer, counter: span.counter - 1 })); + if (style === undefined) continue; + elements.push({ + value: "", + id: { peer, counter: span.counter }, + lamport, + deleted: false, + originLeft: undefined, + originRight: undefined, + anchor: { style, isEnd: true }, + }); continue; } if (span.length < -1) continue; - for (const [key, stack] of active) { - const meta = stack.at(-1); - if (meta !== undefined) { - styleRuns.push({ - run: { start: { peer, counter: span.counter }, length: span.length }, - key, - meta, - }); - } - } for (let offset = 0; offset < span.length; offset += 1) { - ids.push({ peer, counter: span.counter + offset }); - lamports.push(span.counter + offset + span.lamportSub); - characterIndex += 1; + elements.push({ + value: characters[characterIndex++]!, + id: { peer, counter: span.counter + offset }, + lamport: lamport + offset, + deleted: false, + originLeft: undefined, + originRight: undefined, + }); } } - container._insertVisible(0, characters.slice(0, characterIndex), ids, lamports); - for (const { run, key, meta } of styleRuns) { - container._styleIndex.add([run], key, meta); - } + container._appendElements(elements); container._attributeHistoryComplete = false; } else if (container instanceof LoroTree && state.kind === CodecContainerType.Tree) { const records: TreeNodeRecord[] = []; diff --git a/loro-js/src/runtime/sequence-index.ts b/loro-js/src/runtime/sequence-index.ts index 9d7d92290..f3d8d99d1 100644 --- a/loro-js/src/runtime/sequence-index.ts +++ b/loro-js/src/runtime/sequence-index.ts @@ -99,6 +99,9 @@ interface SequenceNode { ownIdRunCount: number; ownUtf16: number; ownUtf8: number; + /** Elements with no UTF-16 width, such as rich-text style anchors. */ + ownZeroWidth: number; + ownVisibleZeroWidth: number; visibleOffsets: number[] | undefined; visibleUtf16Prefix: number[] | undefined; visibleUtf8Prefix: number[] | undefined; @@ -114,6 +117,8 @@ interface SequenceNode { idRunCount: number; allUtf16: number; allUtf8: number; + visibleZeroWidth: number; + allZeroWidth: number; lazyDeleted: boolean; lazyVisible: boolean; } @@ -228,6 +233,16 @@ export class SequenceIndex { return visibleLineBreakMetric(this.#root); } + /** Visible elements with no UTF-16 width, such as rich-text style anchors. */ + get visibleZeroWidthLength(): number { + return this.#root?.visibleZeroWidth ?? 0; + } + + /** Physical elements with no UTF-16 width, including hidden ones. */ + get allZeroWidthLength(): number { + return this.#root?.allZeroWidth ?? 0; + } + get spanCount(): number { let count = 0; visitInOrder(this.#root, () => { @@ -715,6 +730,137 @@ export class SequenceIndex { return undefined; } + /** Number of visible zero-width elements before a visible index. */ + zeroWidthBeforeVisibleIndex(index: number): number | undefined { + if (!Number.isSafeInteger(index) || index < 0 || index > this.visibleLength) { + return undefined; + } + let node = this.#root; + let remaining = index; + let count = 0; + while (node !== undefined) { + pushNodeDeletion(node, this.#metrics); + const leftCount = visibleCount(node.left); + if (remaining < leftCount) { + node = node.left; + continue; + } + remaining -= leftCount; + count += node.left?.visibleZeroWidth ?? 0; + if (remaining < ownVisibleCount(node)) { + if (node.ownVisibleZeroWidth === 0) return count; + for (let offset = 0; offset < nodeLength(node) && remaining > 0; offset += 1) { + if (nodeDeleted(node, offset)) continue; + if (nodeMetrics(node, offset, this.#metrics).utf16 === 0) count += 1; + remaining -= 1; + } + return count; + } + remaining -= ownVisibleCount(node); + count += node.ownVisibleZeroWidth; + node = node.right; + } + return count; + } + + /** + * Visible index of the `offset`-th visible element that has a width, or the + * visible length when `offset` equals the number of such elements. + */ + visibleIndexOfWidthElement(offset: number): number | undefined { + const total = this.visibleLength - this.visibleZeroWidthLength; + if (!Number.isSafeInteger(offset) || offset < 0 || offset > total) return undefined; + if (offset === total) return this.visibleLength; + let node = this.#root; + let remaining = offset; + let index = 0; + while (node !== undefined) { + pushNodeDeletion(node, this.#metrics); + const leftWidth = visibleCount(node.left) - (node.left?.visibleZeroWidth ?? 0); + if (remaining < leftWidth) { + node = node.left; + continue; + } + remaining -= leftWidth; + index += visibleCount(node.left); + const ownWidth = ownVisibleCount(node) - node.ownVisibleZeroWidth; + if (remaining < ownWidth) { + if (node.ownVisibleZeroWidth === 0) { + return index + remaining; + } + for (let physical = 0; physical < nodeLength(node); physical += 1) { + if (nodeDeleted(node, physical)) continue; + if (nodeMetrics(node, physical, this.#metrics).utf16 !== 0) { + if (remaining === 0) return index; + remaining -= 1; + } + index += 1; + } + return undefined; + } + remaining -= ownWidth; + index += ownVisibleCount(node); + node = node.right; + } + return undefined; + } + + /** Visible index of the element that contains a metric unit (zero-based). */ + visibleIndexOfMetricUnit(unit: number, metric: Metric): number | undefined { + if (!Number.isSafeInteger(unit) || unit < 0) return undefined; + let node = this.#root; + let remaining = unit; + let index = 0; + while (node !== undefined) { + pushNodeDeletion(node, this.#metrics); + const leftMetric = visibleMetric(node.left, metric); + if (remaining < leftMetric) { + node = node.left; + continue; + } + remaining -= leftMetric; + index += visibleCount(node.left); + const ownMetric = ownVisibleMetric(node, metric, this.#metrics); + if (remaining < ownMetric) { + const prefix = + metric === "utf16" ? node.visibleUtf16Prefix : node.visibleUtf8Prefix; + if (prefix === undefined) return index; + let low = 0; + let high = prefix.length - 1; + while (low < high) { + const middle = (low + high) >>> 1; + if (prefix[middle + 1]! > remaining) high = middle; + else low = middle + 1; + } + return index + visibleElementsBefore(node, low); + } + remaining -= ownMetric; + index += ownVisibleCount(node); + node = node.right; + } + return undefined; + } + + /** ID runs of every physical element in `[start, end)`, hidden ones included. */ + physicalIdRuns(start: number, end: number): SequenceIdRun[] { + const boundedStart = Math.max(0, Math.min(start, this.allLength)); + const boundedEnd = Math.max(boundedStart, Math.min(end, this.allLength)); + const runs: { start: SequenceId; length: number }[] = []; + visitPhysicalIdRuns(this.#root, boundedStart, boundedEnd, (id, length) => { + const previous = runs[runs.length - 1]; + if ( + previous !== undefined && + previous.start.peer === id.peer && + previous.start.counter + previous.length === id.counter + ) { + previous.length += length; + } else { + runs.push({ start: { ...id }, length }); + } + }); + return runs; + } + visibleIndexAfterLineBreaks(count: number): number | undefined { const total = visibleLineBreakMetric(this.#root); if (!Number.isSafeInteger(count) || count < 0 || count > total) return undefined; @@ -1616,6 +1762,8 @@ export class SequenceIndex { ownIdRunCount: 0, ownUtf16: 0, ownUtf8: 0, + ownZeroWidth: 0, + ownVisibleZeroWidth: 0, visibleOffsets: undefined, visibleUtf16Prefix: undefined, visibleUtf8Prefix: undefined, @@ -1631,6 +1779,8 @@ export class SequenceIndex { idRunCount: 0, allUtf16: 0, allUtf8: 0, + visibleZeroWidth: 0, + allZeroWidth: 0, lazyDeleted: false, lazyVisible: false, }; @@ -1881,6 +2031,8 @@ export class SequenceIndex { const metrics = this.#metrics(element); node.ownUtf16 += metrics.utf16; node.ownUtf8 += metrics.utf8; + const zeroWidth = metrics.utf16 === 0 ? 1 : 0; + node.ownZeroWidth += zeroWidth; if (!element.deleted) { if (node.ownVisibleCount === 0) { node.ownFirstVisibleId = element.id; @@ -1892,6 +2044,7 @@ export class SequenceIndex { node.ownVisibleCount += 1; node.ownVisibleUtf16 += metrics.utf16; node.ownVisibleUtf8 += metrics.utf8; + node.ownVisibleZeroWidth += zeroWidth; } visibleOffsets.push(node.ownVisibleCount); visibleUtf16Prefix.push(node.ownVisibleUtf16); @@ -1944,6 +2097,8 @@ export class SequenceIndex { const metrics = this.#metrics(element); node.ownUtf16 += metrics.utf16; node.ownUtf8 += metrics.utf8; + const zeroWidth = metrics.utf16 === 0 ? 1 : 0; + node.ownZeroWidth += zeroWidth; const lineBreaks = metrics.lineBreaks ?? 0; nodeLineBreaks.own += lineBreaks; if (!element.deleted) { @@ -1957,6 +2112,7 @@ export class SequenceIndex { node.ownVisibleCount += 1; node.ownVisibleUtf16 += metrics.utf16; node.ownVisibleUtf8 += metrics.utf8; + node.ownVisibleZeroWidth += zeroWidth; nodeLineBreaks.ownVisible += lineBreaks; visibleLineBreakOffsets = appendLineBreakOffsets( visibleLineBreakOffsets, @@ -2030,6 +2186,8 @@ export class SequenceIndex { const value = nodeMetrics(node, offsetInNode, this.#metrics); node.ownUtf16 += value.utf16; node.ownUtf8 += value.utf8; + const zeroWidth = value.utf16 === 0 ? 1 : 0; + node.ownZeroWidth += zeroWidth; const lineBreaks = trackLineBreaks ? (value.lineBreaks ?? 0) : 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.own += lineBreaks; if (!nodeDeleted(node, offsetInNode)) { @@ -2043,6 +2201,7 @@ export class SequenceIndex { node.ownVisibleCount += 1; node.ownVisibleUtf16 += value.utf16; node.ownVisibleUtf8 += value.utf8; + node.ownVisibleZeroWidth += zeroWidth; if (nodeLineBreaks !== undefined) { nodeLineBreaks.ownVisible += lineBreaks; visibleLineBreakOffsets = appendLineBreakOffsets( @@ -2567,6 +2726,7 @@ function updateNodeElementVisibility( node.ownVisibleCount += direction; node.ownVisibleUtf16 += direction * value.utf16; node.ownVisibleUtf8 += direction * value.utf8; + if (value.utf16 === 0) node.ownVisibleZeroWidth += direction; recomputeOwnIdRuns(node); if (nodeLength(node) === 1) return; for (let index = offset + 1; index < node.visibleOffsets!.length; index += 1) { @@ -2598,6 +2758,7 @@ function recomputeOwn( const value = nodeMetrics(node, 0, metrics); node.ownUtf16 = value.utf16; node.ownUtf8 = value.utf8; + node.ownZeroWidth = value.utf16 === 0 ? 1 : 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.own = value.lineBreaks ?? 0; node.visibleOffsets = undefined; node.visibleUtf16Prefix = undefined; @@ -2611,6 +2772,7 @@ function recomputeOwn( node.ownVisibleCount = 0; node.ownVisibleUtf16 = 0; node.ownVisibleUtf8 = 0; + node.ownVisibleZeroWidth = 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.ownVisible = 0; node.ownFirstVisibleId = undefined; node.ownLastVisibleId = undefined; @@ -2619,6 +2781,7 @@ function recomputeOwn( node.ownVisibleCount = 1; node.ownVisibleUtf16 = value.utf16; node.ownVisibleUtf8 = value.utf8; + node.ownVisibleZeroWidth = node.ownZeroWidth; if (nodeLineBreaks !== undefined) { nodeLineBreaks.ownVisible = value.lineBreaks ?? 0; } @@ -2631,9 +2794,11 @@ function recomputeOwn( node.ownVisibleCount = 0; node.ownVisibleUtf16 = 0; node.ownVisibleUtf8 = 0; + node.ownVisibleZeroWidth = 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.ownVisible = 0; node.ownUtf16 = 0; node.ownUtf8 = 0; + node.ownZeroWidth = 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.own = 0; node.ownFirstVisibleId = undefined; node.ownLastVisibleId = undefined; @@ -2647,6 +2812,8 @@ function recomputeOwn( const value = nodeMetrics(node, offset, metrics); node.ownUtf16 += value.utf16; node.ownUtf8 += value.utf8; + const zeroWidth = value.utf16 === 0 ? 1 : 0; + node.ownZeroWidth += zeroWidth; const lineBreaks = trackLineBreaks ? (value.lineBreaks ?? 0) : 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.own += lineBreaks; if (!nodeDeleted(node, offset)) { @@ -2660,6 +2827,7 @@ function recomputeOwn( node.ownVisibleCount += 1; node.ownVisibleUtf16 += value.utf16; node.ownVisibleUtf8 += value.utf8; + node.ownVisibleZeroWidth += zeroWidth; if (nodeLineBreaks !== undefined) { nodeLineBreaks.ownVisible += lineBreaks; visibleLineBreakOffsets = appendLineBreakOffsets( @@ -2705,6 +2873,7 @@ function recomputeOwnElements( const value = metrics(storage as T); node.ownUtf16 = value.utf16; node.ownUtf8 = value.utf8; + node.ownZeroWidth = value.utf16 === 0 ? 1 : 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.own = value.lineBreaks ?? 0; node.visibleOffsets = undefined; node.visibleUtf16Prefix = undefined; @@ -2718,6 +2887,7 @@ function recomputeOwnElements( node.ownVisibleCount = 0; node.ownVisibleUtf16 = 0; node.ownVisibleUtf8 = 0; + node.ownVisibleZeroWidth = 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.ownVisible = 0; node.ownFirstVisibleId = undefined; node.ownLastVisibleId = undefined; @@ -2726,6 +2896,7 @@ function recomputeOwnElements( node.ownVisibleCount = 1; node.ownVisibleUtf16 = value.utf16; node.ownVisibleUtf8 = value.utf8; + node.ownVisibleZeroWidth = node.ownZeroWidth; if (nodeLineBreaks !== undefined) { nodeLineBreaks.ownVisible = value.lineBreaks ?? 0; } @@ -2738,9 +2909,11 @@ function recomputeOwnElements( node.ownVisibleCount = 0; node.ownVisibleUtf16 = 0; node.ownVisibleUtf8 = 0; + node.ownVisibleZeroWidth = 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.ownVisible = 0; node.ownUtf16 = 0; node.ownUtf8 = 0; + node.ownZeroWidth = 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.own = 0; node.ownFirstVisibleId = undefined; node.ownLastVisibleId = undefined; @@ -2754,6 +2927,8 @@ function recomputeOwnElements( const value = metrics(element); node.ownUtf16 += value.utf16; node.ownUtf8 += value.utf8; + const zeroWidth = value.utf16 === 0 ? 1 : 0; + node.ownZeroWidth += zeroWidth; const lineBreaks = trackLineBreaks ? (value.lineBreaks ?? 0) : 0; if (nodeLineBreaks !== undefined) nodeLineBreaks.own += lineBreaks; if (!element.deleted) { @@ -2767,6 +2942,7 @@ function recomputeOwnElements( node.ownVisibleCount += 1; node.ownVisibleUtf16 += value.utf16; node.ownVisibleUtf8 += value.utf8; + node.ownVisibleZeroWidth += zeroWidth; if (nodeLineBreaks !== undefined) { nodeLineBreaks.ownVisible += lineBreaks; visibleLineBreakOffsets = appendLineBreakOffsets( @@ -2811,6 +2987,10 @@ function recomputeVisibility( (left?.visibleUtf16 ?? 0) + node.ownVisibleUtf16 + (right?.visibleUtf16 ?? 0); node.visibleUtf8 = (left?.visibleUtf8 ?? 0) + node.ownVisibleUtf8 + (right?.visibleUtf8 ?? 0); + node.visibleZeroWidth = + (left?.visibleZeroWidth ?? 0) + + node.ownVisibleZeroWidth + + (right?.visibleZeroWidth ?? 0); node.firstVisibleId = left?.firstVisibleId ?? node.ownFirstVisibleId ?? right?.firstVisibleId; node.lastVisibleId = @@ -2864,6 +3044,8 @@ function recompute( node.visibleCount = node.ownVisibleCount; node.visibleUtf16 = node.ownVisibleUtf16; node.visibleUtf8 = node.ownVisibleUtf8; + node.visibleZeroWidth = node.ownVisibleZeroWidth; + node.allZeroWidth = node.ownZeroWidth; node.firstVisibleId = node.ownFirstVisibleId; node.lastVisibleId = node.ownLastVisibleId; node.visibleIdRunCount = node.ownVisibleIdRunCount; @@ -2877,6 +3059,8 @@ function recompute( node.allCount = (left?.allCount ?? 0) + nodeLength(node) + (right?.allCount ?? 0); node.allUtf16 = (left?.allUtf16 ?? 0) + node.ownUtf16 + (right?.allUtf16 ?? 0); node.allUtf8 = (left?.allUtf8 ?? 0) + node.ownUtf8 + (right?.allUtf8 ?? 0); + node.allZeroWidth = + (left?.allZeroWidth ?? 0) + node.ownZeroWidth + (right?.allZeroWidth ?? 0); recomputeVisibility(node); node.firstId = left?.firstId ?? node.ownFirstId; node.lastId = right?.lastId ?? node.ownLastId; @@ -2939,6 +3123,7 @@ function applyNodeDeleted( node.visibleCount = 0; node.visibleUtf16 = 0; node.visibleUtf8 = 0; + node.visibleZeroWidth = 0; if (tracksLineBreaks(metrics)) lineBreakMetrics(node).visible = 0; node.firstVisibleId = undefined; node.lastVisibleId = undefined; @@ -2959,6 +3144,7 @@ function applyNodeVisible( node.visibleCount = node.allCount; node.visibleUtf16 = node.allUtf16; node.visibleUtf8 = node.allUtf8; + node.visibleZeroWidth = node.allZeroWidth; if (tracksLineBreaks(metrics)) { const value = lineBreakMetrics(node); value.visible = value.all; @@ -3147,6 +3333,44 @@ function visitInOrder( } } +function visitPhysicalIdRuns( + node: SequenceNode | undefined, + start: number, + end: number, + visit: (start: SequenceId, length: number) => void, +): void { + if (node === undefined || start >= end) return; + if (start === 0 && end === node.allCount && node.idRunCount === 1) { + visit(node.firstId, node.allCount); + return; + } + const leftCount = allCount(node.left); + if (start < leftCount) { + visitPhysicalIdRuns(node.left, start, Math.min(end, leftCount), visit); + } + const ownCount = nodeLength(node); + const ownStart = Math.max(0, start - leftCount); + const ownEnd = Math.min(ownCount, end - leftCount); + if (ownStart < ownEnd) { + if (ownStart === 0 && ownEnd === ownCount && node.ownIdRunCount === 1) { + visit(node.ownFirstId, ownCount); + } else { + for (let offset = ownStart; offset < ownEnd; offset += 1) { + visit(nodeId(node, offset), 1); + } + } + } + const rightStart = leftCount + ownCount; + if (end > rightStart) { + visitPhysicalIdRuns( + node.right, + Math.max(0, start - rightStart), + end - rightStart, + visit, + ); + } +} + function visitVisibleIdRuns( node: SequenceNode | undefined, start: number, diff --git a/loro-js/src/runtime/text-style-index.ts b/loro-js/src/runtime/text-style-index.ts index 98a6ec993..50f0b3dd2 100644 --- a/loro-js/src/runtime/text-style-index.ts +++ b/loro-js/src/runtime/text-style-index.ts @@ -89,6 +89,144 @@ export class TextStyleIndex { return this.#segmentAt(id)?.histories.get(key); } + /** Every style that covers an element, by key, oldest first. */ + membershipAt(id: SequenceId): ReadonlyMap | undefined { + return this.#segmentAt(id)?.histories; + } + + /** Adds every style of `histories` to a run. */ + addMembership( + runs: readonly SequenceIdRun[], + histories: ReadonlyMap, + ): void { + for (const [key, history] of histories) { + for (const meta of history) this.add(runs, key, meta); + } + } + + /** + * Whether every element of the runs resolves `key` to a style whose value + * satisfies `matches`. An element without a style for the key does not match. + */ + everyWinner( + runs: readonly SequenceIdRun[], + key: string, + version: ReadonlyMap | undefined, + matches: (meta: Meta) => boolean, + ): boolean { + for (const run of normalizeRuns(runs)) { + const segments = this.#segmentsByPeer.get(run.start.peer); + if (segments === undefined) return false; + const end = run.start.counter + run.length; + let cursor = run.start.counter; + const first = Math.max( + 0, + segments._lowerBoundBy((segment) => segment.start - cursor) - 1, + ); + for (let index = first; index < segments.size && cursor < end; index += 1) { + const segment = segments.at(index)!; + if (segment.end <= cursor) continue; + if (segment.start > cursor) return false; + const meta = latestIncluded(segment.histories.get(key), version); + if (meta === undefined || !matches(meta)) return false; + cursor = Math.min(end, segment.end); + } + if (cursor < end) return false; + } + return true; + } + + /** Whether some element of the runs has any style for `key`, even an unmark. */ + someHasKey( + runs: readonly SequenceIdRun[], + key: string, + version: ReadonlyMap | undefined, + ): boolean { + for (const run of normalizeRuns(runs)) { + const segments = this.#segmentsByPeer.get(run.start.peer); + if (segments === undefined) continue; + const end = run.start.counter + run.length; + const first = Math.max( + 0, + segments._lowerBoundBy((segment) => segment.start - run.start.counter) - 1, + ); + for (let index = first; index < segments.size; index += 1) { + const segment = segments.at(index)!; + if (segment.start >= end) break; + if (segment.end <= run.start.counter) continue; + if (latestIncluded(segment.histories.get(key), version) !== undefined) + return true; + } + } + return false; + } + + /** + * Sub-runs where `meta` now resolves `key` and changed the resolved value: + * the elements whose attribute an applied style changes. + */ + winningRuns( + runs: readonly SequenceIdRun[], + key: string, + meta: Meta, + version: ReadonlyMap | undefined, + valuesEqual: (left: Meta["value"] | null, right: Meta["value"] | null) => boolean, + ): SequenceIdRun[] { + return this.replacedRuns(runs, key, meta, version, valuesEqual).map( + ({ start, length }) => ({ start, length }), + ); + } + + /** + * Like `winningRuns`, with the style each run would resolve to without `meta`. + */ + replacedRuns( + runs: readonly SequenceIdRun[], + key: string, + meta: Meta, + version: ReadonlyMap | undefined, + valuesEqual: (left: Meta["value"] | null, right: Meta["value"] | null) => boolean, + ): (SequenceIdRun & { readonly previous: Meta | undefined })[] { + const output: { start: SequenceId; length: number; previous: Meta | undefined }[] = + []; + for (const run of normalizeRuns(runs)) { + const segments = this.#segmentsByPeer.get(run.start.peer); + if (segments === undefined) continue; + const end = run.start.counter + run.length; + const first = Math.max( + 0, + segments._lowerBoundBy((segment) => segment.start - run.start.counter) - 1, + ); + for (let index = first; index < segments.size; index += 1) { + const segment = segments.at(index)!; + if (segment.start >= end) break; + const start = Math.max(run.start.counter, segment.start); + const segmentEnd = Math.min(end, segment.end); + if (start >= segmentEnd) continue; + const history = segment.histories.get(key); + if (latestIncluded(history, version) !== meta) continue; + const previous = latestIncluded(history, version, meta); + if (valuesEqual(previous?.value ?? null, meta.value)) continue; + const last = output.at(-1); + if ( + last !== undefined && + last.previous === previous && + last.start.peer === run.start.peer && + last.start.counter + last.length === start + ) { + last.length += segmentEnd - start; + } else { + output.push({ + start: { peer: run.start.peer, counter: start }, + length: segmentEnd - start, + previous, + }); + } + } + } + return output; + } + metasAt( id: SequenceId, version?: ReadonlyMap, @@ -130,30 +268,6 @@ export class TextStyleIndex { }; } - rangeHasKey( - runs: readonly SequenceIdRun[], - key: string, - version?: ReadonlyMap, - ): boolean { - for (const run of normalizeRuns(runs)) { - const segments = this.#segmentsByPeer.get(run.start.peer); - if (segments === undefined) continue; - const end = run.start.counter + run.length; - const first = Math.max( - 0, - segments._lowerBoundBy((segment) => segment.start - run.start.counter) - 1, - ); - for (let index = first; index < segments.size; index += 1) { - const segment = segments.at(index)!; - if (segment.start >= end) break; - if (segment.end <= run.start.counter) continue; - const meta = latestIncluded(segment.histories.get(key), version); - if (meta !== undefined && meta.value !== null) return true; - } - } - return false; - } - transitions( runs: readonly SequenceIdRun[], key: string, @@ -211,37 +325,6 @@ export class TextStyleIndex { return transitions; } - runsContainMeta(runs: readonly SequenceIdRun[], key: string, id: SequenceId): boolean { - for (const run of runs) { - const segments = this.#segmentsByPeer.get(run.start.peer); - if (segments === undefined) return false; - const end = run.start.counter + run.length; - let cursor = run.start.counter; - const first = Math.max( - 0, - segments._lowerBoundBy((segment) => segment.start - cursor) - 1, - ); - for (let index = first; index < segments.size && cursor < end; index += 1) { - const segment = segments.at(index)!; - if (segment.start > cursor || segment.start >= end) return false; - if (segment.end <= cursor) continue; - if ( - !segment.histories - .get(key) - ?.some( - (item) => - item.startId.peer === id.peer && item.startId.counter === id.counter, - ) - ) { - return false; - } - cursor = Math.min(end, segment.end); - } - if (cursor < end) return false; - } - return true; - } - reset(): void { this.#segmentsByPeer.clear(); } @@ -383,15 +466,25 @@ function lowerBoundMeta( return low; } +/** + * The winning style of a history at a version, optionally ignoring one style. + * A style applies once its end anchor (the op after `startId`) is included. + */ function latestIncluded( history: readonly Meta[] | undefined, version: ReadonlyMap | undefined, + excluded?: Meta, ): Meta | undefined { if (history === undefined) return undefined; - if (version === undefined) return history.at(-1); for (let index = history.length - 1; index >= 0; index -= 1) { const meta = history[index]!; - if (meta.startId.counter < (version.get(meta.startId.peer) ?? 0)) return meta; + if (meta === excluded) continue; + if ( + version === undefined || + meta.startId.counter + 1 < (version.get(meta.startId.peer) ?? 0) + ) { + return meta; + } } return undefined; } diff --git a/loro-js/tests/fixtures/rust/richtext-base.ts.blob b/loro-js/tests/fixtures/rust/richtext-base.ts.blob new file mode 100644 index 0000000000000000000000000000000000000000..85accd6eef79ccd4cd4ac009a13ef0c3224e4ae7 GIT binary patch literal 150 zcmd1FFUn^?0*PI-#28rW83Y*w83hO4{A`@8JPbSxY@ED|K)|3S;E|e>ldte${)`5N@cg2jlonP7 Z#s)@4R)LI?l7eC@ef=DuYxGi5%K%*}7)$^F literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/rust/richtext-checkout-mark.blob b/loro-js/tests/fixtures/rust/richtext-checkout-mark.blob new file mode 100644 index 0000000000000000000000000000000000000000..7220f074e5880341e68584619cb4d42f8f0f3aa5 GIT binary patch literal 104 zcmd1FFUn^?0_>YT9x|{5GO#hQF$yp;LS>)~Rv@1dD8a(W#K6SGl9Zp5!cvl2QNkd~ l$il?Iz{1AB$id3a%EQ3Q!pO|V$iT&%n3UYY%)rQ)0stUl3M~Kt literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/rust/richtext-checkout-mark.json b/loro-js/tests/fixtures/rust/richtext-checkout-mark.json new file mode 100644 index 000000000..e4344606b --- /dev/null +++ b/loro-js/tests/fixtures/rust/richtext-checkout-mark.json @@ -0,0 +1,80 @@ +{ + "delta": [ + { + "insert": "abcd", + "attributes": { + "bold": true + } + } + ], + "checkouts": [ + { + "frontiers": [ + { + "peer": "1", + "counter": 2 + } + ], + "delta": [ + { + "insert": "abc" + } + ] + } + ], + "updates": { + "schema_version": 1, + "start_version": {}, + "peers": [ + "1" + ], + "changes": [ + { + "id": "0@0", + "timestamp": 0, + "deps": [], + "lamport": 0, + "msg": null, + "ops": [ + { + "container": "cid:root-text:Text", + "content": { + "type": "insert", + "pos": 0, + "text": "abc" + }, + "counter": 0 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark", + "start": 0, + "end": 3, + "style_key": "bold", + "style_value": true, + "info": 132 + }, + "counter": 3 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark_end" + }, + "counter": 4 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "insert", + "pos": 4, + "text": "d" + }, + "counter": 5 + } + ] + } + ] + } +} diff --git a/loro-js/tests/fixtures/rust/richtext-checkout-mark.snapshot.blob b/loro-js/tests/fixtures/rust/richtext-checkout-mark.snapshot.blob new file mode 100644 index 0000000000000000000000000000000000000000..e3683d2d0f8602f8d80cfbcc6450d449978b3d41 GIT binary patch literal 278 zcmYk0Jqp555QJymUnHck@dg$iz*;XL1nsQE7{wwG2qI!B0Z*W%#}MyeZRtg<+?OA5 zVR`drm;ENoiyVK|zdy#%7jY)>ECw7LyJWM`2rGZez)(SjCdsqZ?AC`JdNis7Z>5JYD-yCFskvC$iNB|Gug1p;E>0X(=VEF50(9+UE!@!L3b9dt%OU`u|_ sCsvy9R0<^On$xb=abM`E5@8-Lny;*ZRZ0!4>4XqUq1I*H_D4WYFAOUQQ2+n{ literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/rust/richtext-mark-positions.json b/loro-js/tests/fixtures/rust/richtext-mark-positions.json new file mode 100644 index 000000000..2b84857f4 --- /dev/null +++ b/loro-js/tests/fixtures/rust/richtext-mark-positions.json @@ -0,0 +1,79 @@ +{ + "delta": [ + { + "insert": "a", + "attributes": { + "bold": true + } + }, + { + "insert": "cXd" + } + ], + "checkouts": [], + "updates": { + "schema_version": 1, + "start_version": {}, + "peers": [ + "1" + ], + "changes": [ + { + "id": "0@0", + "timestamp": 0, + "deps": [], + "lamport": 0, + "msg": null, + "ops": [ + { + "container": "cid:root-text:Text", + "content": { + "type": "insert", + "pos": 0, + "text": "abcd" + }, + "counter": 0 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark", + "start": 0, + "end": 2, + "style_key": "bold", + "style_value": true, + "info": 132 + }, + "counter": 4 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark_end" + }, + "counter": 5 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "insert", + "pos": 5, + "text": "X" + }, + "counter": 6 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "delete", + "pos": 2, + "len": 1, + "start_id": "1@0" + }, + "counter": 7 + } + ] + } + ] + } +} diff --git a/loro-js/tests/fixtures/rust/richtext-mark-positions.snapshot.blob b/loro-js/tests/fixtures/rust/richtext-mark-positions.snapshot.blob new file mode 100644 index 0000000000000000000000000000000000000000..beb978c37e9ae3002b64b561e0317b686bfcece9 GIT binary patch literal 296 zcmYk1v1$TA5QhJMcJ^+)JH#r%H}C^iFMr0}-(l5%LD9(g}H>K0;X%qJ~hz_*kdn9~U%6g#;tmSuGjkW&cOJF;!};XrnPIW}sy) IdT5BKUt(z?l>h($ literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/rust/richtext-right.ts.blob b/loro-js/tests/fixtures/rust/richtext-right.ts.blob new file mode 100644 index 0000000000000000000000000000000000000000..16b12ee9611a25d825785cae25ff04c8c36ddba3 GIT binary patch literal 138 zcmd1FFUn^?0&9eHjx(?nGq4G=F-kLuL1jdtG$R-&0C}v83=E76Y>X_7ObkpMOsTmn zC8-r93`&eF?A#E=K$?$(nVXfJo0XN5lZSzw8={+!k(rs9iGi7!2gqPz;9#_4v~OZ! GU<3dH6A3f` literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/rust/richtext-snapshot.ts.blob b/loro-js/tests/fixtures/rust/richtext-snapshot.ts.blob new file mode 100644 index 0000000000000000000000000000000000000000..80d5dae1f9b146bd1ef8f15c693c96c99c042d29 GIT binary patch literal 598 zcmZWm&ubG=5dLQ7z1<|c#3`O2M-=BtzLRn$x2ICS7O-OQ$q;7 z^ybN%N3VhhK|Ixez(2r?_z#4F;K76Qwk;GK80N=&Z{~g9mu7;V3dWY-SKJYE#Jq+N@c+}t!9pJP3RipU@Pr1a&3-b}%Z8Cn)#nb|R zfNDts>j&1B%c0hH@tvqjBEk>}sifFS)T50t>mbJS0oS{6MYBBT`f?lsAJC>{G6i!x zr(ZitgaU%Xy0E&7N=u#}o&5N=qUXP~QIM)^RC@*9*2Xzlj<3GbTuij2>)Zh6E~sg5 zVg6>rcBmnXHx`&@7U!d-b_=+07m9mKy2Oan2Et?{G=)0KhSSDrWCcVM0VYC3V`1YR jIt>nvMa+M!j~u0Ee}DhE1nOaYF7aiOtoO3hVmtg2cuNWikT3SDy1&F=R8*5Zvv*_1uAC^ A-2eap literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/rust/richtext-unmark-emoji.json b/loro-js/tests/fixtures/rust/richtext-unmark-emoji.json new file mode 100644 index 000000000..a2c7383f4 --- /dev/null +++ b/loro-js/tests/fixtures/rust/richtext-unmark-emoji.json @@ -0,0 +1,92 @@ +{ + "delta": [ + { + "insert": "a" + }, + { + "insert": "😀", + "attributes": { + "bold": true + } + }, + { + "insert": "c" + } + ], + "checkouts": [], + "updates": { + "schema_version": 1, + "start_version": {}, + "peers": [ + "1" + ], + "changes": [ + { + "id": "0@0", + "timestamp": 0, + "deps": [], + "lamport": 0, + "msg": null, + "ops": [ + { + "container": "cid:root-text:Text", + "content": { + "type": "insert", + "pos": 0, + "text": "a😀bc" + }, + "counter": 0 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark", + "start": 0, + "end": 2, + "style_key": "bold", + "style_value": true, + "info": 132 + }, + "counter": 4 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark_end" + }, + "counter": 5 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark", + "start": 0, + "end": 2, + "style_key": "bold", + "style_value": null, + "info": 132 + }, + "counter": 6 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "mark_end" + }, + "counter": 7 + }, + { + "container": "cid:root-text:Text", + "content": { + "type": "delete", + "pos": 6, + "len": 1, + "start_id": "2@0" + }, + "counter": 8 + } + ] + } + ] + } +} diff --git a/loro-js/tests/fixtures/rust/richtext-unmark-emoji.snapshot.blob b/loro-js/tests/fixtures/rust/richtext-unmark-emoji.snapshot.blob new file mode 100644 index 0000000000000000000000000000000000000000..63970eb128244890e8cfb75e580f0aa463c3d2af GIT binary patch literal 309 zcmYjMy-EXN5S-cFyUX2u35SiXjc^ZO?E{Dy@D0q5jRyxI1WO4QtZl?D*jd^8AcBu! zWAA>+MI0Eu+5MKC*`{5z_=l_d;}rVloy+m{7|4;cBKwJ6BwiCKjRGlb(l#?&&hM8v zrb&PmV93fOARsqd!ICP3_%Ef(ZoaoK50hyppc4?>F38H?)rvd" + }, + { + "insert": "H", + "attributes": { + "bold": true + } + }, + { + "insert": "el", + "attributes": { + "em": true + } + }, + { + "insert": "lo!?", + "attributes": { + "bold": true, + "hl": "yellow" + } + }, + { + "insert": " ", + "attributes": { + "hl": "yellow" + } + }, + { + "insert": "World" + } + ], + "base": { + "frontiers": [ + { + "peer": "21", + "counter": 16 + } + ], + "delta": [ + { + "insert": "Hello", + "attributes": { + "bold": true + } + }, + { + "insert": " " + }, + { + "insert": "😀", + "attributes": { + "link": "https://loro.dev" + } + }, + { + "insert": " World" + } + ] + } +} diff --git a/loro-js/tests/fixtures/rust/shallow.ts.blob b/loro-js/tests/fixtures/rust/shallow.ts.blob index d6c9e752ac7d10352634eb8123c00048b30270e7..9b74b2990acc9dc32d8ffc5a8d8ebd2aef39e827 100644 GIT binary patch delta 157 zcmbQo{EqPshPQjC8IA?Ei(&H z0)|9Df{YCJGQP$z0$Hp;%*N2fQj%Ix0%xR}*yWo5C88!9GHOV`bOMdw0tzxPGcYmS Q)!!a9xsg$l#TF<40E1;2r~m)} delta 230 zcmey!G>=&%C%-740SQc7wTx$?$g=t(pqP(;kUs;9lCMO9Lz5+t|A~o-k%7y=(8$=t z)Xdz%lF^rek->qPg9B)=`drN7}AR1 NW?M)+UG5811OV!yBd!1d diff --git a/loro-js/tests/indexes.test.ts b/loro-js/tests/indexes.test.ts index 72834f149..b777164e5 100644 --- a/loro-js/tests/indexes.test.ts +++ b/loro-js/tests/indexes.test.ts @@ -34,7 +34,7 @@ describe("runtime indexes", () => { const operation = current.length === 0 ? 0 : nextRandom() % 3; if (operation === 0) { const position = nextRandom() % (current.length + 1); - const value = ["a", "文", "🙂"][nextRandom() % 3]!; + const value = ["a", "文", "🙂", ""][nextRandom() % 4]!; const element: TestElement = { id: { peer: 1n, counter: counter++ }, deleted: false, @@ -89,6 +89,38 @@ describe("runtime indexes", () => { expect(sequence.atVisible(index)).toBe(element); expect(sequence.visibleIndexOf(element)).toBe(index); } + // Zero-width elements (rich-text style anchors) are skipped by width + // queries and counted separately. + const widths = expectedVisible.filter((element) => element.value.length > 0); + expect(sequence.visibleZeroWidthLength).toBe( + expectedVisible.length - widths.length, + ); + let zeroWidthBefore = 0; + let utf16 = 0; + for (const [index, element] of expectedVisible.entries()) { + expect(sequence.zeroWidthBeforeVisibleIndex(index)).toBe(zeroWidthBefore); + if (element.value.length === 0) { + zeroWidthBefore += 1; + continue; + } + expect(sequence.visibleIndexOfWidthElement(index - zeroWidthBefore)).toBe(index); + for (let unit = 0; unit < element.value.length; unit += 1) { + expect(sequence.visibleIndexOfMetricUnit(utf16 + unit, "utf16")).toBe(index); + } + utf16 += element.value.length; + } + expect(sequence.zeroWidthBeforeVisibleIndex(expectedVisible.length)).toBe( + zeroWidthBefore, + ); + expect(sequence.visibleIndexOfWidthElement(widths.length)).toBe( + expectedVisible.length, + ); + expect(sequence.visibleIndexOfMetricUnit(utf16, "utf16")).toBeUndefined(); + const physicalStart = physical.length >>> 3; + const physicalEnd = physical.length - physicalStart; + expect(sequence.physicalIdRuns(physicalStart, physicalEnd)).toEqual( + compressIdRuns(physical.slice(physicalStart, physicalEnd)), + ); const start = expectedVisible.length >>> 2; const end = expectedVisible.length - start; expect(sequence.visibleRange(start, end)).toEqual( @@ -208,34 +240,54 @@ describe("runtime indexes", () => { expect(styles.metasAt({ peer: 1n, counter: 10 }).get("bold")).toBe(outer); expect(styles.metasAt({ peer: 1n, counter: 50 }).get("bold")).toBe(inner); + // A style applies once its end anchor (the op after its start) is included. expect( styles.metasAt({ peer: 1n, counter: 50 }, new Map([[7n, 1]])).get("bold"), - ).toBe(outer); + ).toBeUndefined(); expect( - styles.rangeHasKey([{ start: { peer: 1n, counter: 30 }, length: 20 }], "bold"), - ).toBe(false); + styles.metasAt({ peer: 1n, counter: 50 }, new Map([[7n, 2]])).get("bold"), + ).toBe(outer); + const whole = [{ start: { peer: 1n, counter: 0 }, length: 100 }]; + const middle = [{ start: { peer: 1n, counter: 30 }, length: 20 }]; + expect(styles.everyWinner(middle, "bold", undefined, (meta) => meta === inner)).toBe( + true, + ); + expect(styles.everyWinner(whole, "bold", undefined, (meta) => meta === inner)).toBe( + false, + ); expect( - styles.runsContainMeta( - [{ start: { peer: 1n, counter: 0 }, length: 100 }], + styles.everyWinner( + [{ start: { peer: 1n, counter: 90 }, length: 20 }], "bold", - outer.startId, + undefined, + () => true, ), - ).toBe(true); + ).toBe(false); + expect(styles.someHasKey(middle, "bold", undefined)).toBe(true); + expect(styles.someHasKey(middle, "italic", undefined)).toBe(false); + expect(styles.someHasKey(middle, "bold", new Map())).toBe(false); expect( - styles.runsContainMeta( - [{ start: { peer: 1n, counter: 0 }, length: 100 }], + styles.winningRuns( + whole, "bold", - inner.startId, + inner, + undefined, + (left, right) => left === right, ), - ).toBe(false); + ).toEqual([{ start: { peer: 1n, counter: 25 }, length: 50 }]); expect( - styles.transitions( - [{ start: { peer: 1n, counter: 0 }, length: 100 }], + styles.winningRuns( + whole, "bold", - new Map([[7n, 1]]), + outer, undefined, + (left, right) => left === right, ), ).toEqual([ + { start: { peer: 1n, counter: 0 }, length: 25 }, + { start: { peer: 1n, counter: 75 }, length: 25 }, + ]); + expect(styles.transitions(whole, "bold", new Map([[7n, 2]]), undefined)).toEqual([ { run: { start: { peer: 1n, counter: 25 }, length: 50 }, before: outer, diff --git a/loro-js/tests/richtext-anchors.test.ts b/loro-js/tests/richtext-anchors.test.ts new file mode 100644 index 000000000..577f91c3f --- /dev/null +++ b/loro-js/tests/richtext-anchors.test.ts @@ -0,0 +1,297 @@ +import { readFileSync } from "node:fs"; + +import { describe, expect, test } from "vitest"; + +import { + LoroDoc, + LoroText, + UndoManager, + type Delta, + type Frontiers, + type JsonSchema, +} from "../src/index"; + +/** + * Rust-side fixtures from `scripts/write-rust-richtext-fixtures.cjs`: loro-crdt + * ran the same edits and recorded its JSON updates, binary updates, snapshot, + * resulting delta, and deltas at recorded versions. + */ +interface RustFixture { + readonly delta: Delta[]; + readonly checkouts: { + readonly frontiers: Frontiers; + readonly delta: Delta[]; + }[]; + readonly updates: JsonSchema; +} + +const fixtureUrl = (name: string): URL => + new URL(`./fixtures/rust/${name}`, import.meta.url); +const rustFixture = (name: string): RustFixture => + JSON.parse(readFileSync(fixtureUrl(`${name}.json`), "utf8")) as RustFixture; +const rustBlob = (name: string): Uint8Array => + new Uint8Array(readFileSync(fixtureUrl(name))); + +function newDoc(): LoroDoc { + const doc = new LoroDoc(); + doc.setPeerId(1); + doc.configTextStyle({ bold: { expand: "after" }, link: { expand: "none" } }); + return doc; +} + +const scenarios: Record Frontiers[]> = { + "richtext-mark-positions"(doc) { + const text = doc.getText("text"); + text.insert(0, "abcd"); + doc.commit(); + text.mark({ start: 0, end: 2 }, "bold", true); + doc.commit(); + text.insert(3, "X"); + doc.commit(); + text.delete(1, 1); + doc.commit(); + return []; + }, + "richtext-unmark-emoji"(doc) { + const text = doc.getText("text"); + text.insert(0, "a😀bc"); + doc.commit(); + text.mark({ start: 0, end: 3 }, "bold", true); + doc.commit(); + text.unmark({ start: 0, end: 1 }, "bold"); + doc.commit(); + text.delete(3, 1); + doc.commit(); + return []; + }, + "richtext-checkout-mark"(doc) { + const text = doc.getText("text"); + text.insert(0, "abc"); + doc.commit(); + const version = doc.frontiers(); + text.mark({ start: 0, end: 3 }, "bold", true); + text.insert(3, "d"); + doc.commit(); + return [version]; + }, +}; + +function expectCheckouts(doc: LoroDoc, fixture: RustFixture): void { + for (const { frontiers, delta } of fixture.checkouts) { + doc.checkout(frontiers); + expect(doc.getText("text").toDelta()).toEqual(delta); + doc.checkoutToLatest(); + } + expect(doc.getText("text").toDelta()).toEqual(fixture.delta); +} + +describe("rich-text style anchors match Rust positions", () => { + for (const [name, edit] of Object.entries(scenarios)) { + test(`${name}: writes Rust's ops and reads Rust's updates`, () => { + const fixture = rustFixture(name); + const doc = newDoc(); + edit(doc); + expect(doc.getText("text").toDelta()).toEqual(fixture.delta); + // Every op position (insert, delete, mark start/end) matches Rust's. + expect(doc.exportJsonUpdates().changes).toEqual(fixture.updates.changes); + expectCheckouts(doc, fixture); + + for (const source of [ + (target: LoroDoc) => target.importJsonUpdates(fixture.updates), + (target: LoroDoc) => target.import(rustBlob(`${name}.blob`)), + ]) { + const imported = new LoroDoc(); + source(imported); + expect(imported.getText("text").toDelta()).toEqual(fixture.delta); + expectCheckouts(imported, fixture); + } + const fromSnapshot = new LoroDoc(); + fromSnapshot.import(rustBlob(`${name}.snapshot.blob`)); + expect(fromSnapshot.getText("text").toDelta()).toEqual(fixture.delta); + + // loro.js snapshots keep the anchors, so their history checks out too. + const reloaded = new LoroDoc(); + reloaded.import(doc.export({ mode: "update" })); + expectCheckouts(reloaded, fixture); + const snapshot = new LoroDoc(); + snapshot.import(doc.export({ mode: "snapshot" })); + expect(snapshot.getText("text").toDelta()).toEqual(fixture.delta); + }); + } + + test("text inserted next to anchors follows Rust's expand rules", () => { + const doc = newDoc(); + doc.configTextStyle({ + bold: { expand: "after" }, + link: { expand: "none" }, + em: { expand: "before" }, + hl: { expand: "both" }, + }); + const text = doc.getText("text"); + text.insert(0, "--AB--"); + for (const key of ["bold", "link", "em", "hl"]) { + text.mark({ start: 2, end: 4 }, key, true); + } + // At the start: only em and hl expand before. At the end: bold and hl + // expand after. + text.insert(4, "e"); + text.insert(2, "s"); + expect(text.toDelta()).toEqual([ + { insert: "--" }, + { insert: "s", attributes: { em: true, hl: true } }, + { insert: "AB", attributes: { bold: true, em: true, hl: true, link: true } }, + { insert: "e", attributes: { bold: true, hl: true } }, + { insert: "--" }, + ]); + // The unmark's end anchor expands after (the reverse of none), so "u" stays + // unbolded; before "A", the bold start anchor keeps "v" outside bold and + // link. The expected delta is Rust's output for the same edits. + text.unmark({ start: 3, end: 4 }, "bold"); + text.insert(4, "u"); + text.insert(3, "v"); + expect(text.toDelta()).toEqual([ + { insert: "--" }, + { insert: "sv", attributes: { em: true, hl: true } }, + { insert: "Au", attributes: { link: true, em: true, hl: true } }, + { insert: "B", attributes: { bold: true, link: true, em: true, hl: true } }, + { insert: "e", attributes: { bold: true, hl: true } }, + { insert: "--" }, + ]); + }); + + test("marks that change nothing write no anchors", () => { + const doc = newDoc(); + const text = doc.getText("text"); + text.insert(0, "abc"); + doc.commit(); + const ops = (): number => + doc.exportJsonUpdates().changes.flatMap((c) => c.ops).length; + const before = ops(); + text.unmark({ start: 0, end: 2 }, "bold"); + doc.commit(); + expect(ops()).toBe(before); + text.mark({ start: 0, end: 2 }, "bold", true); + doc.commit(); + expect(ops()).toBe(before + 2); + text.mark({ start: 0, end: 1 }, "bold", true); + doc.commit(); + expect(ops()).toBe(before + 2); + expect(() => text.mark({ start: 1, end: 1 }, "bold", true)).toThrow(RangeError); + }); + + test("positions convert across Unicode, UTF-16, and UTF-8 around anchors", () => { + const doc = newDoc(); + const text = doc.getText("text"); + text.insert(0, "a😀b中c"); + text.mark({ start: 1, end: 4 }, "bold", true); + text.mark({ start: 3, end: 5 }, "link", "x"); + expect(text.length).toBe(6); + expect(text.convertPos(5, "unicode", "utf16")).toBe(6); + expect(text.convertPos(3, "unicode", "utf8")).toBe(6); + expect(text.convertPos(6, "utf8", "unicode")).toBe(3); + expect(text.convertPos(3, "utf16", "unicode")).toBe(2); + expect(text.convertPos(2, "utf16", "unicode")).toBeUndefined(); + expect(text.slice(1, 4)).toBe("😀b"); + expect(text.charAt(3)).toBe("b"); + expect(text.getCursor(3)?.pos()).toEqual({ peer: "1", counter: 2 }); + let chunks = ""; + text.iter((chunk) => { + chunks += chunk; + }); + expect(chunks).toBe("a😀b中c"); + text.delete(1, 3); // 😀b: the anchors stay + expect(text.toString()).toBe("a中c"); + expect(text.toDelta()).toEqual([ + { insert: "a" }, + { insert: "中", attributes: { link: "x" } }, + { insert: "c" }, + ]); + }); + + test("concurrent inserts at a style boundary follow the anchors' Fugue order", () => { + const base = newDoc(); + base.getText("text").insert(0, "ab"); + base.commit(); + const left = new LoroDoc(); + left.setPeerId(2); + left.import(base.export({ mode: "update" })); + const right = new LoroDoc(); + right.setPeerId(3); + right.import(base.export({ mode: "update" })); + left.configTextStyle({ bold: { expand: "after" } }); + left.getText("text").mark({ start: 0, end: 1 }, "bold", true); + left.commit(); + right.getText("text").insert(1, "X"); + right.commit(); + left.import(right.export({ mode: "update" })); + right.import(left.export({ mode: "update" })); + // X was inserted concurrently after "a" without seeing the mark; Fugue + // places it after the end anchor, outside the style. + for (const doc of [left, right]) { + expect(doc.getText("text").toDelta()).toEqual([ + { insert: "a", attributes: { bold: true } }, + { insert: "Xb" }, + ]); + } + }); + + test("undo and redo restore the attributes a mark replaced", () => { + const doc = newDoc(); + const undo = new UndoManager(doc, { mergeInterval: 0 }); + const text = doc.getText("text"); + text.insert(0, "Hello World"); + doc.commit(); + text.mark({ start: 0, end: 5 }, "bold", true); + doc.commit(); + doc.configTextStyle({ bold: { expand: "after" }, italic: { expand: "after" } }); + text.mark({ start: 3, end: 8 }, "italic", true); + doc.commit(); + text.unmark({ start: 1, end: 4 }, "bold"); + doc.commit(); + // Each state below is what Rust's UndoManager produces for the same edits. + const steps = [ + () => undo.undo(), + () => undo.undo(), + () => undo.undo(), + () => undo.redo(), + () => undo.redo(), + ]; + const deltas = steps.map((step) => { + step(); + return text.toDelta(); + }); + expect(deltas).toEqual([ + [ + { insert: "Hel", attributes: { bold: true } }, + { insert: "lo", attributes: { bold: true, italic: true } }, + { insert: " Wo", attributes: { italic: true } }, + { insert: "rld" }, + ], + [{ insert: "Hello", attributes: { bold: true } }, { insert: " World" }], + [{ insert: "Hello World" }], + [{ insert: "Hello", attributes: { bold: true } }, { insert: " World" }], + [ + { insert: "Hel", attributes: { bold: true } }, + { insert: "lo", attributes: { bold: true, italic: true } }, + { insert: " Wo", attributes: { italic: true } }, + { insert: "rld" }, + ], + ]); + }); + + test("detached text marks with anchors and keeps them when attached", () => { + const doc = newDoc(); + const map = doc.getMap("map"); + const detached = new LoroText(); + detached.insert(0, "hello"); + detached.mark({ start: 0, end: 5 }, "bold", true); + detached.insert(5, "!"); + expect(detached.toDelta()).toEqual([ + { insert: "hello!", attributes: { bold: true } }, + ]); + const attached = map.setContainer("text", detached); + expect(attached.toDelta()).toEqual([ + { insert: "hello!", attributes: { bold: true } }, + ]); + }); +}); diff --git a/loro-js/tests/richtext-differential.test.ts b/loro-js/tests/richtext-differential.test.ts index 3c84c83c2..5764c65d2 100644 --- a/loro-js/tests/richtext-differential.test.ts +++ b/loro-js/tests/richtext-differential.test.ts @@ -7,13 +7,19 @@ const rust = loadRustReference(); const seeds = Number(process.env.LORO_DIFF_SEEDS ?? 40); const steps = Number(process.env.LORO_DIFF_STEPS ?? 80); const firstSeed = Number(process.env.LORO_DIFF_FIRST_SEED ?? 1); -// Marks need loro.js to keep Rust's style anchors in the text sequence. -const options = { marks: false }; -describe.skipIf(rust === undefined)("text differential against Rust", () => { +describe.skipIf(rust === undefined)("rich text differential against Rust", () => { test.each(Array.from({ length: seeds }, (_, index) => firstSeed + index))( - "random concurrent text edits converge with Rust (seed %i)", + "random concurrent rich-text edits converge with Rust (seed %i)", (seed) => { + expect(runActions(rust!, generateActions(seed, steps))).toBeUndefined(); + }, + ); + // Plain text keeps its own seeds so a rich-text failure does not hide one. + test.each(Array.from({ length: seeds >> 1 }, (_, index) => firstSeed + index))( + "random concurrent plain-text edits converge with Rust (seed %i)", + (seed) => { + const options = { marks: false }; expect(runActions(rust!, generateActions(seed, steps, options), options)).toBe( undefined, ); From 829db4fe16e7c2073529cc735074b7b5b6c11c7a Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 04:29:44 +0800 Subject: [PATCH 03/25] fix(loro.js): stop a concurrent insert at the end of its origin's subtree The indexed Fugue path jumped from the last direct child of `originLeft` to the origin-right element. Rust's scan stops at the first concurrent element whose origin is further left, so the insert could land after that element. Replicas then disagreed depending on import order, and all of them could disagree with loro-crdt. Check whether the interval's last element descends from `originLeft` and otherwise binary-search the boundary. The descent test skips implicit runs through per-peer sorted explicit counters. Rationale and cost: context/loro-js-performance.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- .changeset/loro-js-fugue-origin-subtree.md | 5 + context/loro-js-performance.md | 11 ++- loro-js/src/runtime/containers.ts | 104 +++++++++++++++++++- loro-js/tests/fugue-index.test.ts | 106 ++++++++++++++++++++- 4 files changed, 223 insertions(+), 3 deletions(-) create mode 100644 .changeset/loro-js-fugue-origin-subtree.md diff --git a/.changeset/loro-js-fugue-origin-subtree.md b/.changeset/loro-js-fugue-origin-subtree.md new file mode 100644 index 000000000..fecda9748 --- /dev/null +++ b/.changeset/loro-js-fugue-origin-subtree.md @@ -0,0 +1,5 @@ +--- +"loro.js": patch +--- + +Fix a `loro.js` Text/List convergence bug with concurrent inserts. When an insert was concurrent with a run typed after the same character and with another concurrent element whose origin was further left, `loro.js` put the insert after that element instead of at the end of the run. Replicas could disagree with each other (depending on the order in which they received the updates) and with `loro-crdt`. `loro.js` now places these inserts where Rust does. diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 896482166..14dda5920 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -103,7 +103,16 @@ JavaScript constant factor. keep their single-child edge implicit, and `SequenceIndex` can skip an entire future ID run while finding the next causally included element. Ordinary local edits keep the smaller unindexed path. MovableList continues to use the scan - because moves break the origin-tree physical preorder. + because moves break the origin-tree physical preorder. Sibling subtrees are + contiguous, so the gap between two direct children belongs to the earlier + child. After the last child the interval can also hold concurrent elements + whose origin is left of `originLeft`; Rust's scan stops before them. The index + therefore checks whether the interval's last element descends from + `originLeft` and otherwise binary-searches the boundary. The descent test + walks origin-left links but jumps over each implicit run through per-peer + sorted counters of explicit (non-consecutive) elements, so it costs + O(explicit links · log n), like Rust's span-based scan, instead of one probe + per scalar in a long concurrent run. - Merging adjacent changes appends only the new operations and key-table entries to the retained record. The cached operation length, peer end, frontier set, operation indexes, and subscriber update slice are updated incrementally, so diff --git a/loro-js/src/runtime/containers.ts b/loro-js/src/runtime/containers.ts index 9b3979b91..c6a49a6e2 100644 --- a/loro-js/src/runtime/containers.ts +++ b/loro-js/src/runtime/containers.ts @@ -2870,6 +2870,12 @@ interface FugueOriginEntry { interface FugueOriginIndex { structureVersion: number; readonly explicitChildren: Map; + /** + * Sorted counters, per peer, of the elements that are not the implicit child + * of their predecessor ID. Every other element is linked to `counter - 1`, so + * an origin-left walk can jump from any element to the head of its run. + */ + readonly explicitCounters: Map; } interface FugueInsertionResult { @@ -3036,6 +3042,7 @@ function indexedFugueInsertion( const parentRightIndex = directRightParentIndex(sequence, originLeft, originRight); let insertIndex = startIndex; let scanning = false; + let afterLastChild = false; for (let childIndex = 0; childIndex < candidates.length; childIndex += 1) { const { entry: other } = candidates[childIndex]!; if (sameOptionalId(other.originRight, originRight)) { @@ -3054,7 +3061,38 @@ function indexedFugueInsertion( } if (!scanning) { - insertIndex = candidates[childIndex + 1]?.index ?? originRightIndex; + const next = candidates[childIndex + 1]; + insertIndex = next?.index ?? originRightIndex; + afterLastChild = next === undefined; + } + } + + // Sibling subtrees are contiguous, so the gap between two direct children + // belongs to the earlier one. After the last child, the interval can also + // hold concurrent elements whose origin is left of `originLeft`; the scan + // stops before them, so the insertion must too. + const lastChild = candidates.at(-1); + if (afterLastChild && originLeft !== undefined && lastChild !== undefined) { + const left = sequence.findByIdRaw(originLeft); + const leftIndex = left === undefined ? undefined : sequence.physicalIndexOf(left); + if (leftIndex === undefined) return undefined; + const inSubtree = (index: number): boolean => + fugueDescendsFrom( + sequence, + originIndex, + sequence.atPhysicalRaw(index)!.id, + originLeft, + leftIndex, + ); + if (!inSubtree(originRightIndex - 1)) { + let low = lastChild.index + 1; + let high = originRightIndex - 1; + while (low < high) { + const middle = (low + high) >>> 1; + if (inSubtree(middle)) low = middle + 1; + else high = middle; + } + insertIndex = low; } } @@ -3066,6 +3104,54 @@ function indexedFugueInsertion( }; } +/** + * Whether `ancestor`, at physical index `ancestorIndex`, is on the origin-left + * chain of `id`. Ancestors precede their descendants, so the walk stops at the + * first origin left of `ancestorIndex`; runs of implicit children are skipped + * through `explicitCounters`. + */ +function fugueDescendsFrom( + sequence: SequenceIndex, + index: FugueOriginIndex, + id: CodecId, + ancestor: CodecId, + ancestorIndex: number, +): boolean { + let current = id; + for (;;) { + const counters = index.explicitCounters.get(current.peer); + let head = current.counter; + if (counters !== undefined) { + let low = 0; + let high = counters.length; + while (low < high) { + const middle = (low + high) >>> 1; + if (counters[middle]! <= current.counter) low = middle + 1; + else high = middle; + } + if (low > 0) head = counters[low - 1]!; + } + if ( + ancestor.peer === current.peer && + head <= ancestor.counter && + ancestor.counter < current.counter + ) { + return true; + } + const parent = sequence.findByIdRaw({ + peer: current.peer, + counter: head, + })?.originLeft; + if (parent === undefined) return false; + if (parent.peer === ancestor.peer && parent.counter === ancestor.counter) return true; + const element = sequence.findByIdRaw(parent); + const parentIndex = + element === undefined ? undefined : sequence.physicalIndexOf(element); + if (parentIndex === undefined || parentIndex < ancestorIndex) return false; + current = parent; + } +} + function getFugueOriginIndex( sequence: SequenceIndex, ): FugueOriginIndex { @@ -3079,6 +3165,7 @@ function getFugueOriginIndex( const rebuilt: FugueOriginIndex = { structureVersion: sequence.structureVersion, explicitChildren: new Map(), + explicitCounters: new Map(), }; sequence.forEachPhysicalRaw((element) => { recordFugueOriginEntry(rebuilt, element.id, element.originLeft, element.originRight); @@ -3147,6 +3234,21 @@ function recordFugueOriginEntry( ) { return; } + const counters = index.explicitCounters.get(id.peer); + if (counters === undefined) { + index.explicitCounters.set(id.peer, [id.counter]); + } else if (counters.at(-1)! < id.counter) { + counters.push(id.counter); + } else { + let low = 0; + let high = counters.length; + while (low < high) { + const middle = (low + high) >>> 1; + if (counters[middle]! < id.counter) low = middle + 1; + else high = middle; + } + if (counters[low] !== id.counter) counters.splice(low, 0, id.counter); + } const key = optionalSequenceIdKey(originLeft); const children = index.explicitChildren.get(key); const entry = { diff --git a/loro-js/tests/fugue-index.test.ts b/loro-js/tests/fugue-index.test.ts index 7b59fa27c..dcac55c6c 100644 --- a/loro-js/tests/fugue-index.test.ts +++ b/loro-js/tests/fugue-index.test.ts @@ -1,6 +1,6 @@ import { describe, expect, test, vi } from "vitest"; -import { LoroText } from "../src/index"; +import { LoroDoc, LoroText } from "../src/index"; import { SequenceIndex } from "../src/runtime/sequence-index"; describe("Fugue origin index", () => { @@ -23,4 +23,108 @@ describe("Fugue origin index", () => { expect(longRunProbes).toBeLessThanOrEqual(shortRunProbes + 2); expect(longRunProbes).toBeLessThan(8); }); + + test("finds the end of a concurrent run under the origin without probing it", () => { + const probeCount = (length: number): number => { + const text = new LoroText(); + text._insertFugue(0, "x", { peer: 1n, counter: 0 }, 0, new Map()); + // A concurrent root: it sorts after x and stays after x's subtree. + text._insertFugue(0, "n", { peer: 4n, counter: 0 }, 0, new Map()); + text._insertFugue( + 1, + "a".repeat(length), + { peer: 1n, counter: 1 }, + 1, + new Map([[1n, 1]]), + ); + expect(text.toString()).toBe(`x${"a".repeat(length)}n`); + + const atPhysical = vi.spyOn(SequenceIndex.prototype, "atPhysicalRaw"); + text._insertFugue(1, "b", { peer: 2n, counter: 0 }, 1, new Map([[1n, 1]])); + const probes = atPhysical.mock.calls.length; + atPhysical.mockRestore(); + + expect(text.toString()).toBe(`x${"a".repeat(length)}bn`); + return probes; + }; + + const shortRunProbes = probeCount(128); + const longRunProbes = probeCount(16_384); + // A binary search over the interval, not a scan of the run. + expect(longRunProbes).toBeLessThanOrEqual(shortRunProbes + 16); + expect(longRunProbes).toBeLessThan(64); + }); + + // An insert concurrent with a run under its origin must stop at the first + // concurrent element whose origin is further left, as Rust's scan does. + // The indexed path used to jump to the origin-right element instead. + test.each(["list", "text"] as const)( + "orders a concurrent %s insert before an element from an older origin", + (kind) => { + const peers = [1, 2, 3, 4].map((peer) => { + const doc = new LoroDoc(); + doc.setPeerId(peer); + return doc; + }); + const [p1, p2, p3, p4] = peers as [LoroDoc, LoroDoc, LoroDoc, LoroDoc]; + const insert = (doc: LoroDoc, pos: number, value: string): void => { + if (kind === "list") doc.getList("seq").insert(pos, value); + else doc.getText("seq").insert(pos, value); + doc.commit(); + }; + const read = (doc: LoroDoc): string => + kind === "list" + ? (doc.getList("seq").toArray() as string[]).join("") + : doc.getText("seq").toString(); + + insert(p1, 0, "A"); + insert(p2, 0, "B"); + insert(p4, 0, "N"); + p3.import(p1.export({ mode: "update" })); + p3.import(p4.export({ mode: "update" })); + expect(read(p3)).toBe("AN"); + // Y: origin left A, origin right N. + insert(p3, 1, "Y"); + p1.import(p2.export({ mode: "update" })); + p1.import(p4.export({ mode: "update" })); + expect(read(p1)).toBe("ABN"); + // X: origin left A, origin right B. + insert(p1, 1, "X"); + p1.import(p3.export({ mode: "update" })); + + const replay = new LoroDoc(); + for (const peer of [p4, p3, p2, p1]) replay.import(peer.export({ mode: "update" })); + // loro-crdt gives AXYBN for every import order. + expect(read(replay)).toBe("AXYBN"); + expect(read(p1)).toBe("AXYBN"); + }, + ); + + test("places an insert concurrent with a run before a concurrent root, as Rust does", () => { + const [p1, p2, p4] = [1, 2, 4].map((peer) => { + const doc = new LoroDoc(); + doc.setPeerId(peer); + return doc; + }) as [LoroDoc, LoroDoc, LoroDoc]; + p1.getText("text").insert(0, "x"); + p1.commit(); + p2.import(p1.export({ mode: "update" })); + p1.getText("text").insert(1, "aaaa"); + p1.commit(); + p4.getText("text").insert(0, "n"); + p4.commit(); + p2.getText("text").insert(1, "b"); + p2.commit(); + + for (const order of [ + [p1, p4, p2], + [p2, p4, p1], + [p4, p1, p2], + ]) { + const doc = new LoroDoc(); + for (const peer of order) doc.import(peer.export({ mode: "update" })); + // loro-crdt gives xaaaabn; the indexed path used to give xaaaanb. + expect(doc.getText("text").toString()).toBe("xaaaabn"); + } + }); }); From ae2a67fd04999302e45c3438b5edaf63d58924e8 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:08:43 +0800 Subject: [PATCH 04/25] perf(loro.js): jump between ID runs when finding a Fugue subtree end An element right after its ID predecessor has it as origin-left, so only run starts can end a sibling's subtree (context/loro-js-performance.md). Importing a concurrent character after a 512k-character run drops from ~9.5 ms to ~0.2 ms. Adds text-concurrent-insert-after-long-run to bench:complexity. Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 4 ++ loro-js/benchmarks/complexity.mjs | 18 +++++ loro-js/src/runtime/containers.ts | 38 ++++++----- loro-js/src/runtime/sequence-index.ts | 96 ++++++++++++++------------- loro-js/tests/indexes.test.ts | 45 +++++++++++++ loro-js/tests/text-interop.test.ts | 39 +++++++++++ 6 files changed, 177 insertions(+), 63 deletions(-) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 38a37474a..4c1ec6283 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -108,6 +108,10 @@ JavaScript constant factor. the origin-right bound: when that bound is only the next causally included element, later concurrent elements of an ancestor's subtree can sit before it. The walk stops at the first element whose origin-left precedes the sibling. + Inside a physical ID run every element's origin-left is its predecessor, so + the walk jumps between run starts (`SequenceIndex.nextPhysicalIdRunStart`, + skipping single-run treap subtrees): O((runs in the subtree + 1) log n), and + O(log n) after a long typed run (`text-concurrent-insert-after-long-run`). - An imported Text delete is resolved by its position in the op's causal view, like Rust's tracker (`LoroText._deleteTargets`): O(log n + runs) through `visibleIdRuns` or the cached causal view. The recorded `start_id` is only a diff --git a/loro-js/benchmarks/complexity.mjs b/loro-js/benchmarks/complexity.mjs index 46abfa8e1..489164db0 100644 --- a/loro-js/benchmarks/complexity.mjs +++ b/loro-js/benchmarks/complexity.mjs @@ -509,6 +509,24 @@ for (const size of sizes) { }); unsubscribeImport(); + // A concurrent character that Fugue orders after a sibling's whole run of + // 64 * size characters. Peer 1's character first builds the origin index. + const concurrentUpdate = (peer, text) => { + const doc = new LoroDoc(); + doc.setPeerId(peer); + doc.getText("text").insert(0, text); + doc.commit(); + return doc.export({ mode: "update" }); + }; + const concurrentTarget = new LoroDoc(); + concurrentTarget.import(concurrentUpdate(2, "x".repeat(size * 64))); + concurrentTarget.import(concurrentUpdate(1, "y")); + const concurrentTail = concurrentUpdate(3, "z"); + measure("text-concurrent-insert-after-long-run", size, () => { + concurrentTarget.import(concurrentTail); + return concurrentTarget.getText("text").length; + }); + const checkoutTextDoc = new LoroDoc(); checkoutTextDoc.setPeerId(3); checkoutTextDoc.getText("text").insert(0, "x".repeat(size)); diff --git a/loro-js/src/runtime/containers.ts b/loro-js/src/runtime/containers.ts index caa37a646..51c59c6d6 100644 --- a/loro-js/src/runtime/containers.ts +++ b/loro-js/src/runtime/containers.ts @@ -3139,30 +3139,32 @@ function indexedFugueInsertion( * the next causally included element, later concurrent elements that belong to * an ancestor's subtree can sit before it. An element is a descendant exactly * when its origin-left lies inside the scanned subtree. + * + * Only the first element of a physical ID run needs that check. An element + * right after its ID predecessor has that predecessor as origin-left: the op + * was placed right after its origin-left in its causal view, the predecessor + * is in that view, and only concurrent elements can come between an element + * and its origin-left. So the walk jumps from run start to run start and costs + * O((runs in the subtree + 1) log n), not one step per scalar. */ function fugueSubtreeEnd( sequence: SequenceIndex, rootIndex: number, limit: number, ): number { - let end = rootIndex + 1; - if (end >= limit) return limit; - let previous = sequence.atPhysicalRaw(rootIndex)!.id; - sequence.forEachPhysicalFrom(end, (element, index) => { - if (index >= limit) return false; - const left = element.originLeft; - if (left === undefined) return false; - if (left.peer !== previous.peer || left.counter !== previous.counter) { - const leftElement = sequence.findByIdRaw(left); - const leftIndex = - leftElement === undefined ? undefined : sequence.physicalIndexOf(leftElement); - if (leftIndex === undefined || leftIndex < rootIndex) return false; - } - previous = element.id; - end = index + 1; - return undefined; - }); - return end; + let index = rootIndex + 1; + while (index < limit) { + const start = sequence.nextPhysicalIdRunStart(index, limit); + if (start >= limit) return limit; + const left = sequence.atPhysicalRaw(start)!.originLeft; + if (left === undefined) return start; + const leftElement = sequence.findByIdRaw(left); + const leftIndex = + leftElement === undefined ? undefined : sequence.physicalIndexOf(leftElement); + if (leftIndex === undefined || leftIndex < rootIndex) return start; + index = start + 1; + } + return limit; } function getFugueOriginIndex( diff --git a/loro-js/src/runtime/sequence-index.ts b/loro-js/src/runtime/sequence-index.ts index 9d7d92290..249eef1fc 100644 --- a/loro-js/src/runtime/sequence-index.ts +++ b/loro-js/src/runtime/sequence-index.ts @@ -423,51 +423,16 @@ export class SequenceIndex { }); } - /** Visits elements in physical order from `start` until `visit` returns false. */ - forEachPhysicalFrom( - start: number, - visit: (element: T, index: number) => boolean | void, - ): void { - if (!Number.isSafeInteger(start) || start < 0 || start >= this.allLength) return; - const stack: SequenceNode[] = []; - let node = this.#root; - let base = 0; - let current: SequenceNode | undefined; - let offset = 0; - while (node !== undefined) { - const leftCount = allCount(node.left); - if (start < base + leftCount) { - stack.push(node); - node = node.left; - continue; - } - const ownStart = base + leftCount; - if (start < ownStart + nodeLength(node)) { - current = node; - offset = start - ownStart; - break; - } - base = ownStart + nodeLength(node); - node = node.right; - } - let index = start; - while (current !== undefined) { - for (; offset < nodeLength(current); offset += 1) { - if (visit(nodeElement(current, offset), index) === false) return; - index += 1; - } - let next = current.right; - if (next === undefined) { - current = stack.pop(); - } else { - while (next.left !== undefined) { - stack.push(next); - next = next.left; - } - current = next; - } - offset = 0; - } + /** + * Returns the first physical index in `[start, limit)` whose element does not + * continue the ID run of the element before it, or `limit` if there is none. + * Expected O(log n): treap subtrees that are one ID run are skipped whole. + */ + nextPhysicalIdRunStart(start: number, limit = this.allLength): number { + const end = Math.min(limit, this.allLength); + if (start >= end) return end; + const found = findPhysicalIdRunStart(this.#root, 0, start, undefined); + return found === undefined || found > end ? end : found; } findNextIncludedPhysical( @@ -2233,6 +2198,47 @@ function nodeCounter( : (node.element as T).id.counter; } +/** + * Finds the first index `>= start` in this subtree whose element starts an ID + * run. `before` is the ID of the element physically before the subtree. + */ +function findPhysicalIdRunStart( + node: SequenceNode | undefined, + nodeStart: number, + start: number, + before: SequenceId | undefined, +): number | undefined { + if (node === undefined || nodeStart + node.allCount <= start) return undefined; + if (node.idRunCount === 1) { + return start <= nodeStart && + (before === undefined || !sequenceIdsContinue(before, node.firstId)) + ? nodeStart + : undefined; + } + + const fromLeft = findPhysicalIdRunStart(node.left, nodeStart, start, before); + if (fromLeft !== undefined) return fromLeft; + + const ownStart = nodeStart + allCount(node.left); + let previous = node.left?.lastId ?? before; + const length = nodeLength(node); + const firstOffset = Math.max(0, start - ownStart); + if (firstOffset < length) { + if (firstOffset > 0) previous = nodeId(node, firstOffset - 1); + for (let offset = firstOffset; offset < length; offset += 1) { + const id = nodeId(node, offset); + if (previous === undefined || !sequenceIdsContinue(previous, id)) { + return ownStart + offset; + } + // The rest of a single-run node continues this element. + if (node.ownIdRunCount === 1) break; + previous = id; + } + } + + return findPhysicalIdRunStart(node.right, ownStart + length, start, node.ownLastId); +} + function findNextIncludedPhysical( node: SequenceNode | undefined, nodeStart: number, diff --git a/loro-js/tests/indexes.test.ts b/loro-js/tests/indexes.test.ts index 72834f149..9bb9aea08 100644 --- a/loro-js/tests/indexes.test.ts +++ b/loro-js/tests/indexes.test.ts @@ -571,6 +571,51 @@ describe("runtime indexes", () => { } }); + test("finds physical ID run starts", () => { + const sequence = new SequenceIndex(); + const physical: TestElement[] = []; + let random = 0x51_7c_c1_b7; + const nextRandom = (): number => { + random ^= random << 13; + random ^= random >>> 17; + random ^= random << 5; + return random >>> 0; + }; + const nextCounter = new Map(); + for (let step = 0; step < 400; step += 1) { + const peer = BigInt(nextRandom() % 3); + const length = 1 + (nextRandom() % 40); + const start = nextCounter.get(peer) ?? 0; + nextCounter.set(peer, start + length); + const elements = Array.from({ length }, (_, offset) => ({ + id: { peer, counter: start + offset }, + deleted: false, + deletedBy: [], + value: "x", + })); + const position = nextRandom() % (physical.length + 1); + sequence.insertAtPhysical(position, elements); + physical.splice(position, 0, ...elements); + } + + const startsRun = (index: number): boolean => { + if (index === 0) return true; + const previous = physical[index - 1]!.id; + const current = physical[index]!.id; + return previous.peer !== current.peer || previous.counter + 1 !== current.counter; + }; + for (let start = 0; start <= physical.length; start += 1) { + const limit = start + (nextRandom() % 200); + let expected = start; + while (expected < Math.min(limit, physical.length) && !startsRun(expected)) { + expected += 1; + } + expect(sequence.nextPhysicalIdRunStart(start, limit)).toBe( + Math.min(expected, limit, physical.length), + ); + } + }); + test("ranges across an excluded element", () => { const sequence = new SequenceIndex(); const base = Array.from({ length: 6 }, (_, counter) => ({ diff --git a/loro-js/tests/text-interop.test.ts b/loro-js/tests/text-interop.test.ts index 944b7f5f9..1a10fa42b 100644 --- a/loro-js/tests/text-interop.test.ts +++ b/loro-js/tests/text-interop.test.ts @@ -4,6 +4,7 @@ import { describe, expect, test } from "vitest"; import { LoroDoc, type Delta, type JsonSchema } from "../src/index"; import { applyDelta, normalizeDelta } from "./support/richtext-differential"; +import { loadRustReference, type RustReference } from "./support/rust-reference"; const fixture = (name: string): Uint8Array => new Uint8Array(readFileSync(new URL(`./fixtures/rust/${name}`, import.meta.url))); @@ -65,6 +66,44 @@ describe("text interoperability with Rust", () => { } }); + test("orders a concurrent insert after a sibling subtree of several ID runs", () => { + // z goes after the whole subtree of peer 2's first "a": its later run of a, + // the B run, and peer 4's c run, whose starts are the only elements + // `fugueSubtreeEnd` checks. + const scenario = ({ LoroDoc: Doc }: { LoroDoc: typeof LoroDoc }): string => { + const make = (peer: number): LoroDoc => { + const created = new Doc(); + created.setPeerId(peer); + return created; + }; + const [p1, p2, p3, p4, target] = [1, 2, 3, 4, 9].map(make) as [ + LoroDoc, + LoroDoc, + LoroDoc, + LoroDoc, + LoroDoc, + ]; + p2.getText("t").insert(0, "a".repeat(100)); + p2.commit(); + sync(p2, p4); + p2.getText("t").insert(50, "B".repeat(40)); + p2.commit(); + p4.getText("t").insert(80, "c".repeat(30)); + p4.commit(); + p1.getText("t").insert(0, "y"); + p1.commit(); + p3.getText("t").insert(0, "z"); + p3.commit(); + for (const source of [p2, p4, p1, p3]) sync(source, target); + return target.getText("t").toString(); + }; + const expected = `ya${"a".repeat(49)}${"B".repeat(40)}${"a".repeat(30)}${"c".repeat(30)}${"a".repeat(20)}z`; + expect(scenario({ LoroDoc })).toBe(expected); + // The same edits in Rust, when its WASM build is available. + const rust: RustReference | undefined = loadRustReference(); + expect(rust === undefined ? expected : scenario(rust)).toBe(expected); + }); + test("writes multi-run deletes last run first, like Rust", () => { const owner = doc(1); const text = owner.getText("t"); From b6db4d10a712dee76301fe4f2c06f15c3b401f51 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:13:50 +0800 Subject: [PATCH 05/25] docs(loro.js): read 0.2 data like Rust and document the upgrade Decision and rationale: context/loro-js-rust-differential.md ("Data written by loro.js 0.2"). The changeset becomes major; README "Upgrading from 0.2" lists what changes and how to migrate. Fixtures written by loro.js 0.2.1 pin how loro.js and Rust read its updates, snapshots, and cursors. Co-Authored-By: Claude Opus 5.5 (1M context) --- .changeset/loro-js-text-interop.md | 13 ++- context/loro-js-rust-differential.md | 38 +++++--- loro-js/AGENTS.md | 5 + loro-js/README.md | 41 ++++++++ loro-js/scripts/write-legacy-fixtures.mjs | 65 +++++++++++++ .../loro-js-0.2/cursor-emoji.cursor.blob | Bin 0 -> 9 bytes .../loro-js-0.2/cursor-emoji.snapshot.blob | Bin 0 -> 248 bytes .../tests/fixtures/loro-js-0.2/expected.json | 4 + .../fixtures/loro-js-0.2/text-concurrent.json | 88 ++++++++++++++++++ .../loro-js-0.2/text-concurrent.snapshot.blob | Bin 0 -> 369 bytes .../loro-js-0.2/text-concurrent.update.blob | Bin 0 -> 282 bytes loro-js/tests/legacy-data.test.ts | 62 ++++++++++++ 12 files changed, 300 insertions(+), 16 deletions(-) create mode 100644 loro-js/scripts/write-legacy-fixtures.mjs create mode 100644 loro-js/tests/fixtures/loro-js-0.2/cursor-emoji.cursor.blob create mode 100644 loro-js/tests/fixtures/loro-js-0.2/cursor-emoji.snapshot.blob create mode 100644 loro-js/tests/fixtures/loro-js-0.2/expected.json create mode 100644 loro-js/tests/fixtures/loro-js-0.2/text-concurrent.json create mode 100644 loro-js/tests/fixtures/loro-js-0.2/text-concurrent.snapshot.blob create mode 100644 loro-js/tests/fixtures/loro-js-0.2/text-concurrent.update.blob create mode 100644 loro-js/tests/legacy-data.test.ts diff --git a/.changeset/loro-js-text-interop.md b/.changeset/loro-js-text-interop.md index ec51a6bc3..7e9b31ab6 100644 --- a/.changeset/loro-js-text-interop.md +++ b/.changeset/loro-js-text-interop.md @@ -1,11 +1,14 @@ --- -"loro.js": patch +"loro.js": major --- -Text edits now converge with Rust (`loro-crdt`) in cases a randomized comparison with the Rust WASM build found: +Text now follows Rust (`loro-crdt`) everywhere, including when reading documents written by earlier loro.js versions. This breaks documents edited only with loro.js that have concurrent text edits or stored cursors: read "Upgrading from 0.2" in the README before upgrading. -- A concurrent insert after a sibling's subtree was placed after later, unrelated concurrent text. Replicas that merged such edits in different orders could end with different text. -- An imported delete is now applied by its position, as Rust does, instead of by its recorded `start_id`. Rust's WASM build records a wrong `start_id` when the deleted text spans astral characters (for example, emoji) inserted in separate edits, so loro.js deleted different characters than Rust. +- **Breaking:** concurrent text inserts are ordered like Rust. Earlier versions could place an insert after a sibling's subtree behind later, unrelated concurrent text, so a history with such inserts can now read differently (for example `béa9` instead of `bé9a`). Earlier loro.js replicas that merged such edits in different orders, and documents shared with `loro-crdt` peers, had already diverged; they now agree with Rust. +- **Breaking:** text cursors follow Rust. An encoded cursor stores the Unicode position of its target, not the UTF-16 position, so a cursor encoded by an earlier version can resolve to another offset. A cursor at the end or in empty text has no target, `getCursorPos` returns the target's own offset for every side, and a deleted target reports the length of the text before it with side `-1`. Get new cursors after upgrading. +- An imported delete is applied by its position, as Rust does, instead of by its recorded `start_id` (Rust's WASM build up to 1.16.3 can record a wrong one around astral characters, loro-dev/loro#1149). - `checkout`, `diff`, and `revertTo` no longer restore a character twice when two concurrent deletes removed it. -- Text cursors follow Rust: a cursor at the end or in empty text has no target, `getCursorPos` returns the target's own offset for every side, and a deleted target reports the length of the text before it with side `-1`. The encoded cursor matches Rust's bytes. - Consecutive inserts in one transaction are merged into one op, and a delete that spans several ID runs writes the last run first, as Rust does. +- Importing a concurrent insert next to a long run of text no longer walks the whole run. + +Upgrading: a snapshot written by an earlier version keeps its current text, but its history (updates, JSON updates, checkout of older versions) is read like Rust, and a replica loaded from a snapshot and one loaded from updates can keep different text after new edits. Migrate every replica the same way; the safest path is to read the final state with the old version and build a new document from it. diff --git a/context/loro-js-rust-differential.md b/context/loro-js-rust-differential.md index 898e19ce4..c72addc6f 100644 --- a/context/loro-js-rust-differential.md +++ b/context/loro-js-rust-differential.md @@ -60,20 +60,36 @@ regression test. ## Rust behaviors the harness works around -The Rust reference has bugs of its own in UTF-16 (WASM) builds: +WASM builds before loro-dev/loro#1135 had bugs of their own. The harness still +tolerates them, so it also runs against older reference builds: -- A subscribed transaction that merges deletes of astral text can panic with +- A subscribed transaction that merges deletes of astral text could panic with "Op/hint length mismatch" (`change_to_diff` in - `crates/loro-internal/src/txn.rs`), even after unsubscribing. The harness - subscribes only to loro.js documents. + `crates/loro-internal/src/txn.rs`). The harness subscribes only to loro.js + documents. - `get_text_entity_ranges` advanced the recorded delete `start_id` by the UTF-16 - length of astral text, so Rust can write a delete whose `start_id` names the - wrong elements. Rust applies deletes by position, and so does loro.js - (`LoroText._deleteTargets`). Rust still resolves a deleted cursor target - through the recorded IDs, so the harness skips cursors Rust cannot resolve. -- Rust's shallow import falls back to the recorded delete IDs when they disagree - with the positions, which can diverge from its own full-history import. The - harness compares loro.js with the source document instead. + length of astral text, so Rust could write a delete whose `start_id` names the + wrong elements; such deletes stay in existing histories (loro-dev/loro#1149). + Rust applies deletes by position, and so does loro.js + (`LoroText._deleteTargets`). Rust resolves a deleted cursor target through the + recorded IDs, so the harness skips cursors Rust cannot resolve. +- Rust's shallow import labels placeholders with the recorded delete IDs, which + can diverge from its own full-history import when they are wrong. The harness + compares loro.js with the source document instead. Snapshot-import checkout (`snapshotCheckout`) is off by default until loro.js can check out a document imported from a snapshot (loro-dev/loro#1126). + +## Data written by loro.js 0.2 + +loro.js 0.2 ordered some concurrent text inserts differently from Rust (a +concurrent insert after a sibling's subtree, now `fugueSubtreeEnd`) and encoded +text cursors by UTF-16 position. Decision (2026-09-28): later versions read all +data with Rust's semantics and add no version marker. The same bytes must mean +the same thing in both runtimes, documents shared with `loro-crdt` peers had +already diverged, and a marker would need a format change that Rust does not +have. The cost falls on documents edited only with 0.2: their history can read +differently, and a replica loaded from a 0.2 snapshot keeps the 0.2 text while +one loaded from updates follows Rust. `README.md` ("Upgrading from 0.2") +describes the migration paths; `tests/legacy-data.test.ts` pins the readings +with fixtures written by `scripts/write-legacy-fixtures.mjs`. diff --git a/loro-js/AGENTS.md b/loro-js/AGENTS.md index cd5b3661c..61232601b 100644 --- a/loro-js/AGENTS.md +++ b/loro-js/AGENTS.md @@ -20,6 +20,11 @@ the same binary and JSON data as Rust (`loro-crdt`) and give the same results. `crates/loro/tests/loro_js_interop.rs` imports them. Container state encoders live in `src/runtime/document.ts` (`#containerState`) and must follow `docs/encoding-container-states.md`. +- loro.js 0.2 output read by later versions: `scripts/write-legacy-fixtures.mjs` + writes `tests/fixtures/loro-js-0.2/` with a 0.2 build; `tests/legacy-data.test.ts` + pins that loro.js and Rust read it the same way. Later versions read 0.2 data + with Rust's semantics, with no version marker (README "Upgrading from 0.2"; the + decision is in [context/loro-js-rust-differential.md](../context/loro-js-rust-differential.md)). ## Commands diff --git a/loro-js/README.md b/loro-js/README.md index 6e037ad0e..ccbb38d73 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -121,6 +121,47 @@ Treat the package as experimental when data can be produced by untrusted or olde clients. Keep a `loro-crdt` interoperability test for the formats and operations your application depends on. +## Upgrading from 0.2 + +1.0 reads every document the way Rust (`loro-crdt`) does, including documents +written by 0.2. The data has no version marker, so a document edited only with 0.2 +can read differently after the upgrade: + +- **Concurrent text inserts.** 0.2 could order an insert after a sibling's subtree + differently from Rust. 1.0 orders it like Rust, so such a history can give other + text (`bé9a` in 0.2, `béa9` in 1.0 and Rust). +- **Cursors.** 0.2 encoded the UTF-16 position of a cursor's target; Rust and 1.0 use + the Unicode position. A cursor encoded by 0.2 can resolve to another offset (on + `ab😀`, 4 in 0.2 and 2 in 1.0). Get new cursors from the upgraded document. + +Documents also edited by `loro-crdt` peers already disagreed with Rust in these cases; +1.0 makes them agree. + +How 1.0 reads data written by 0.2 (measured on the example above): + +| 0.2 data | Current text in 1.0 | History in 1.0 | +| --------------------- | ----------------------- | -------------------------------------------------------------------------- | +| Updates, JSON | Rust's reading (`béa9`) | Rust's reading | +| Full/shallow snapshot | Kept (`bé9a`) | Rust's reading: exported updates and checkouts of older versions follow it | + +**Migrate every replica the same way.** Replicas loaded from the same 0.2 snapshot +agree with each other, but a replica loaded from 0.2 updates holds other text, and +the two stay different after exchanging new edits (`béQ9aZ` and `béQa9Z`). A peer that +later joins through exported updates instead of a snapshot diverges the same way. + +Recommended paths: + +1. **Keep the content, drop the history (safest).** With 0.2, check out the latest + version and read the final state (`doc.toJSON()`, or `text.toDelta()` for rich + text). In 1.0, build one new document from it and share that document with every + replica. Do not import old updates or snapshots into it. +2. **Keep the history, accept Rust's reading.** Import the 0.2 updates (not snapshots) + on every replica, and check the content, because it can differ from what 0.2 + showed. + +A 0.2 snapshot keeps its current text, so loading it is only safe when every replica +loads that snapshot and new peers join through snapshots, never through updates. + ## Development checks From this directory: diff --git a/loro-js/scripts/write-legacy-fixtures.mjs b/loro-js/scripts/write-legacy-fixtures.mjs new file mode 100644 index 000000000..6889f676d --- /dev/null +++ b/loro-js/scripts/write-legacy-fixtures.mjs @@ -0,0 +1,65 @@ +// Writes documents with loro.js 0.2 into tests/fixtures/loro-js-0.2, so that +// tests/legacy-data.test.ts pins how later versions read data written before +// loro.js adopted Rust's text semantics ("Upgrading from 0.2" in README.md). +// +// npm install --prefix /tmp/loro-js-0.2 loro.js@0.2.1 +// node scripts/write-legacy-fixtures.mjs /tmp/loro-js-0.2/node_modules/loro.js/dist/index.js + +import { mkdirSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const legacyEntry = process.argv[2]; +if (legacyEntry === undefined) { + throw new Error("pass the path of loro.js 0.2's dist/index.js"); +} +const { LoroDoc } = await import(pathToFileURL(legacyEntry).href); + +const directory = new URL("../tests/fixtures/loro-js-0.2/", import.meta.url); +mkdirSync(directory, { recursive: true }); +const write = (name, bytes) => writeFileSync(new URL(name, directory), bytes); +const expected = {}; + +function doc(peer) { + const created = new LoroDoc(); + created.setPeerId(peer); + return created; +} + +function sync(from, to) { + to.import(from.export({ mode: "update" })); +} + +// Concurrent inserts that 0.2 ordered differently from Rust: "a" (peer 3) and +// "é" (peer 2) both follow "b", and 0.2 put "a" after "9". +{ + const [peer1, peer2, peer3] = [1, 2, 3].map(doc); + peer2.getText("t").insert(0, "9"); + peer2.commit(); + peer1.getText("t").insert(0, "b"); + peer1.commit(); + sync(peer1, peer2); + peer2.getText("t").insert(1, "é"); + peer2.commit(); + sync(peer1, peer3); + peer3.getText("t").insert(1, "a"); + peer3.commit(); + sync(peer3, peer2); + write("text-concurrent.update.blob", peer2.export({ mode: "update" })); + write("text-concurrent.snapshot.blob", peer2.export({ mode: "snapshot" })); + write("text-concurrent.json", JSON.stringify(peer2.exportJsonUpdates(), null, 2)); + expected["text-concurrent"] = peer2.getText("t").toString(); +} + +// A cursor on the end of "ab😀": 0.2 encoded its UTF-16 position. +{ + const owner = doc(1); + const text = owner.getText("c"); + text.insert(0, "ab😀"); + owner.commit(); + const cursor = text.getCursor(4, 1); + write("cursor-emoji.snapshot.blob", owner.export({ mode: "snapshot" })); + write("cursor-emoji.cursor.blob", cursor.encode()); + expected["cursor-emoji"] = owner.getCursorPos(cursor).offset; +} + +write("expected.json", `${JSON.stringify(expected, null, 2)}\n`); diff --git a/loro-js/tests/fixtures/loro-js-0.2/cursor-emoji.cursor.blob b/loro-js/tests/fixtures/loro-js-0.2/cursor-emoji.cursor.blob new file mode 100644 index 0000000000000000000000000000000000000000..584f5e8f57c9e871df7db184d03af9c823164a77 GIT binary patch literal 9 QcmZQ%WMN=TW?*6g00GYcaR2}S literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/loro-js-0.2/cursor-emoji.snapshot.blob b/loro-js/tests/fixtures/loro-js-0.2/cursor-emoji.snapshot.blob new file mode 100644 index 0000000000000000000000000000000000000000..964b55e069a71bc2a4a41660b40ca504fa9348ca GIT binary patch literal 248 zcmd1FFUn^?0xcC=_A@Xy14VrNgZvp7m>HNE1sEBjDxeHjAfJ(ejgf_siGhKMF`0pn zk%fs7$O0l(CProkcDBT%5A$a(z3n#R@k1Ry&Yh{4VPnhZ4&?i6Mg7LaZr!Nde*0UZMZj10|Px#wZVGcz;+ Q9SA0cVkaE4fT&>r0KEboD*ylh literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/loro-js-0.2/expected.json b/loro-js/tests/fixtures/loro-js-0.2/expected.json new file mode 100644 index 000000000..79106f599 --- /dev/null +++ b/loro-js/tests/fixtures/loro-js-0.2/expected.json @@ -0,0 +1,4 @@ +{ + "text-concurrent": "bé9a", + "cursor-emoji": 4 +} diff --git a/loro-js/tests/fixtures/loro-js-0.2/text-concurrent.json b/loro-js/tests/fixtures/loro-js-0.2/text-concurrent.json new file mode 100644 index 000000000..c51d81803 --- /dev/null +++ b/loro-js/tests/fixtures/loro-js-0.2/text-concurrent.json @@ -0,0 +1,88 @@ +{ + "schema_version": 1, + "start_version": {}, + "peers": [ + "1", + "2", + "3" + ], + "changes": [ + { + "id": "0@0", + "timestamp": 0, + "deps": [], + "lamport": 0, + "msg": null, + "ops": [ + { + "container": "cid:root-t:Text", + "content": { + "type": "insert", + "pos": 0, + "text": "b" + }, + "counter": 0 + } + ] + }, + { + "id": "0@1", + "timestamp": 0, + "deps": [], + "lamport": 0, + "msg": null, + "ops": [ + { + "container": "cid:root-t:Text", + "content": { + "type": "insert", + "pos": 0, + "text": "9" + }, + "counter": 0 + } + ] + }, + { + "id": "1@1", + "timestamp": 0, + "deps": [ + "0@0", + "0@1" + ], + "lamport": 1, + "msg": null, + "ops": [ + { + "container": "cid:root-t:Text", + "content": { + "type": "insert", + "pos": 1, + "text": "é" + }, + "counter": 1 + } + ] + }, + { + "id": "0@2", + "timestamp": 0, + "deps": [ + "0@0" + ], + "lamport": 1, + "msg": null, + "ops": [ + { + "container": "cid:root-t:Text", + "content": { + "type": "insert", + "pos": 1, + "text": "a" + }, + "counter": 0 + } + ] + } + ] +} \ No newline at end of file diff --git a/loro-js/tests/fixtures/loro-js-0.2/text-concurrent.snapshot.blob b/loro-js/tests/fixtures/loro-js-0.2/text-concurrent.snapshot.blob new file mode 100644 index 0000000000000000000000000000000000000000..074087f5ee706d680895b9e7226cfe1e3b236672 GIT binary patch literal 369 zcmYjNu}T9$5PdVVduI+YN+d+IkgF9e6Dz$UO=6 zu(7rD2gF8e8{eHCI54v_Z{NI`-Scdk;lJJ8f5YC1^oOG%Tzu3zJ(xYRzJ)+!RTyD4 zd@Xqa(?FakpmKo~amqXFd6f{ylh8wVUb3iMv6Q)QFX>0_Hl(5v?P|Ge9`FU^@NS^ zbjti%0e~CldH3V&?U_YNFCWV(ezKT%##WLDQ?xXzPUk~{T11jV!ptj{AhpY8HPJBR NpOi7mp3wV~_yI);Clvqy literal 0 HcmV?d00001 diff --git a/loro-js/tests/fixtures/loro-js-0.2/text-concurrent.update.blob b/loro-js/tests/fixtures/loro-js-0.2/text-concurrent.update.blob new file mode 100644 index 0000000000000000000000000000000000000000..c13a0f884d79caf798865f63b4418b84a41aabd1 GIT binary patch literal 282 zcmd1FFUn^?0$<(Pbs1Q!7#JBC83h;_p)ybgE0E8~z{beJ$i%?F#8|?>$H>CO2xI{f zD-%!=6JrugD^LX(5U0NnBZEWMgW}x0w5kv7cfCxz|3@br7Kv2G!rvaKf)J8 J8^gqy2mptd3H1N~ literal 0 HcmV?d00001 diff --git a/loro-js/tests/legacy-data.test.ts b/loro-js/tests/legacy-data.test.ts new file mode 100644 index 000000000..e89f70a2b --- /dev/null +++ b/loro-js/tests/legacy-data.test.ts @@ -0,0 +1,62 @@ +import { readFileSync } from "node:fs"; + +import { describe, expect, test } from "vitest"; + +import { Cursor, LoroDoc, type JsonSchema } from "../src/index"; +import { loadRustReference } from "./support/rust-reference"; + +// Written by loro.js 0.2.1 (scripts/write-legacy-fixtures.mjs). Later versions +// read them the way Rust does; README.md "Upgrading from 0.2" documents the +// differences. +const fixtureUrl = (name: string): URL => + new URL(`./fixtures/loro-js-0.2/${name}`, import.meta.url); +const fixture = (name: string): Uint8Array => + new Uint8Array(readFileSync(fixtureUrl(name))); +const json = (name: string): unknown => + JSON.parse(readFileSync(fixtureUrl(name), "utf8")); +const legacy = json("expected.json") as Record; + +interface Runtime { + readonly LoroDoc: typeof LoroDoc; + readonly Cursor: typeof Cursor; +} + +const rust = loadRustReference(); +const runtimes: [string, Runtime][] = [["loro.js", { LoroDoc, Cursor }]]; +if (rust !== undefined) runtimes.push(["loro-crdt", rust]); + +describe.each(runtimes)("documents written by loro.js 0.2, read by %s", (_, runtime) => { + const read = (load: (doc: LoroDoc) => void): LoroDoc => { + const doc = new runtime.LoroDoc(); + load(doc); + return doc; + }; + + test("orders concurrent inserts from history like Rust", () => { + expect(legacy["text-concurrent"]).toBe("bé9a"); + const fromUpdate = read((doc) => doc.import(fixture("text-concurrent.update.blob"))); + expect(fromUpdate.getText("t").toString()).toBe("béa9"); + const fromJson = read((doc) => + doc.importJsonUpdates(json("text-concurrent.json") as JsonSchema), + ); + expect(fromJson.getText("t").toString()).toBe("béa9"); + }); + + test("keeps a snapshot's current text but reads its history like Rust", () => { + const fromSnapshot = read((doc) => + doc.import(fixture("text-concurrent.snapshot.blob")), + ); + expect(fromSnapshot.getText("t").toString()).toBe("bé9a"); + const history = read((doc) => doc.import(fromSnapshot.export({ mode: "update" }))); + expect(history.getText("t").toString()).toBe("béa9"); + }); + + test("resolves a 0.2 cursor by its Unicode position", () => { + expect(legacy["cursor-emoji"]).toBe(4); + const owner = read((doc) => doc.import(fixture("cursor-emoji.snapshot.blob"))); + const cursor = runtime.Cursor.decode(fixture("cursor-emoji.cursor.blob")); + expect(owner.getCursorPos(cursor)?.offset).toBe(2); + const fresh = owner.getText("c").getCursor(4, 1)!; + expect(owner.getCursorPos(fresh)?.offset).toBe(4); + }); +}); From 7ae6f5a9d5d29993723cae101d87bdab837e0efe Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:27:37 +0800 Subject: [PATCH 06/25] fix(loro.js): hydrate lazy text, redact dead styles, and keep checkout events - Insert positions hydrate a lazily loaded Text first; its anchors decide where text goes. - Shallow snapshots null the values of styles with no text left, like Rust's redact_dead_style_values (context/shallow-snapshot-style-redaction.md). - A transition that only toggles a start anchor starts the text's event baseline, so it no longer reports the whole text as inserted. - The differential suite checks out per-op versions, two in a row. Co-Authored-By: Claude Opus 5.5 (1M context) --- loro-js/src/runtime/containers.ts | 2 + loro-js/src/runtime/document.ts | 77 +++++++++- loro-js/tests/richtext-anchors.test.ts | 136 ++++++++++++++++++ .../tests/support/richtext-differential.ts | 37 ++++- 4 files changed, 242 insertions(+), 10 deletions(-) diff --git a/loro-js/src/runtime/containers.ts b/loro-js/src/runtime/containers.ts index 2bff88365..c68453272 100644 --- a/loro-js/src/runtime/containers.ts +++ b/loro-js/src/runtime/containers.ts @@ -2179,6 +2179,7 @@ export class LoroText extends LoroContainer { } #entityFrom(index: number, from: TextPosType): number | undefined { + this._ensureHydrated(); const sequence = this._sequence; const length = sequence.visibleLength; if (sequence.visibleZeroWidthLength === 0) { @@ -2238,6 +2239,7 @@ export class LoroText extends LoroContainer { * `anchorStaysAfterInsert`), or after all of them. */ #insertPosition(position: number): number { + // Hydrates a lazily loaded text; its anchors decide where the text goes. const entity = this.#entityAt(position); if (entity === undefined) { throw new RangeError(`text position ${position} is out of range`); diff --git a/loro-js/src/runtime/document.ts b/loro-js/src/runtime/document.ts index 422fb2261..f59dd3fdd 100644 --- a/loro-js/src/runtime/document.ts +++ b/loro-js/src/runtime/document.ts @@ -3997,6 +3997,12 @@ export class LoroDoc = Record = Record(); this.#rebuildFromHistory(rootVersion); - const rootStore = this.#buildStateStore(startFrontiers); + const rootStore = this.#buildStateStore(startFrontiers, { redacted }); this.#rebuildFromHistory(latestVersion); - const latestStore = this.#buildStateStore(); + const latestStore = this.#buildStateStore(undefined, { + redacted: new Set(), + only: redacted, + }); if (restoreVersion.compare(latestVersion) !== 0) { this.#rebuildFromHistory(restoreVersion); } @@ -5612,7 +5625,10 @@ export class LoroDoc = Record = Record = Record(); @@ -5741,6 +5760,7 @@ export class LoroDoc = Record; + /** Only pairs in this set may be nulled (the shallow root's pairs). */ + readonly only?: ReadonlySet; +} + +const STYLE_EXPAND_BOTH = 0x02 | 0x04; + +/** + * Rust's `redact_dead_style_values`: nulls the value of every style pair with + * no text span between its anchors, except both-expand pairs, which still + * style future inserts. Anchors and IDs stay so positions do not move. + */ +function redactDeadStyleValues( + peers: readonly bigint[], + spans: readonly { peerIndex: bigint; counter: number; length: number }[], + marks: { value: EncodedLoroValue; info: number }[], + redaction: StyleRedaction, +): void { + // Open pairs keyed by the ID their end span carries (start counter + 1). + const open = new Map(); + let starts = 0; + let textSpans = 0; + for (const span of spans) { + if (span.length > 0) { + textSpans += 1; + } else if (span.length === 0) { + const id = `${peers[Number(span.peerIndex)]}:${span.counter}`; + open.set(`${span.peerIndex}:${span.counter + 1}`, { mark: starts, id, textSpans }); + starts += 1; + } else { + const key = `${span.peerIndex}:${span.counter}`; + const pair = open.get(key); + if (pair === undefined) continue; + open.delete(key); + if (pair.textSpans !== textSpans) continue; + const mark = marks[pair.mark]!; + if ((mark.info & STYLE_EXPAND_BOTH) === STYLE_EXPAND_BOTH) continue; + if (mark.value.type === "null") continue; + if (redaction.only !== undefined && !redaction.only.has(pair.id)) continue; + mark.value = { type: "null" }; + redaction.redacted.add(pair.id); + } + } +} diff --git a/loro-js/tests/richtext-anchors.test.ts b/loro-js/tests/richtext-anchors.test.ts index 577f91c3f..9c4f94456 100644 --- a/loro-js/tests/richtext-anchors.test.ts +++ b/loro-js/tests/richtext-anchors.test.ts @@ -2,6 +2,11 @@ import { readFileSync } from "node:fs"; import { describe, expect, test } from "vitest"; +import { + ContainerType, + decodeFastSnapshot, + decodeStateSnapshotStore, +} from "../src/codec/index"; import { LoroDoc, LoroText, @@ -10,6 +15,8 @@ import { type Frontiers, type JsonSchema, } from "../src/index"; +import { applyDelta, normalizeDelta } from "./support/richtext-differential"; +import { loadRustReference } from "./support/rust-reference"; /** * Rust-side fixtures from `scripts/write-rust-richtext-fixtures.cjs`: loro-crdt @@ -295,3 +302,132 @@ describe("rich-text style anchors match Rust positions", () => { ]); }); }); + +interface Runtime { + readonly LoroDoc: typeof LoroDoc; + readonly LoroText: typeof LoroText; +} + +const rust = loadRustReference(); +const runtimes: [string, Runtime][] = [["loro.js", { LoroDoc, LoroText }]]; +if (rust !== undefined) runtimes.push(["loro-crdt", rust]); + +/** The values of the style marks in a Text state of an encoded state store. */ +function markValues(stateBytes: Uint8Array): unknown[] { + const store = decodeStateSnapshotStore(stateBytes); + if (store.kind !== "sstable") return []; + return store.containers.flatMap(({ wrapper: { state } }) => + state.kind === ContainerType.Text + ? state.marks.map(({ value }) => (value.type === "null" ? null : value.value)) + : [], + ); +} + +describe.each(runtimes)("rich-text anchors in %s", (runtimeName, runtime) => { + test("inserts into a nested text that a snapshot loaded lazily", () => { + const source = new runtime.LoroDoc(); + source.setPeerId(1); + source.configTextStyle({ bold: { expand: "before" } }); + const text = source.getMap("m").setContainer("t", new runtime.LoroText()); + text.insert(0, "abcd"); + text.mark({ start: 0, end: 4 }, "bold", true); + source.commit(); + const snapshot = source.export({ mode: "snapshot" }); + + const insertInto = (position: number): Delta[] => { + const doc = new runtime.LoroDoc(); + doc.setPeerId(2); + doc.configTextStyle({ bold: { expand: "before" } }); + doc.import(snapshot); + // Not read before the insert, so the text is still unhydrated. + const nested = doc.getMap("m").get("t") as LoroText; + nested.insert(position, "X"); + return nested.toDelta(); + }; + // bold expands before: X at 0 joins it, X at 4 does not. + expect(insertInto(0)).toEqual([{ insert: "Xabcd", attributes: { bold: true } }]); + expect(insertInto(2)).toEqual([{ insert: "abXcd", attributes: { bold: true } }]); + expect(insertInto(4)).toEqual([ + { insert: "abcd", attributes: { bold: true } }, + { insert: "X" }, + ]); + }); + + test.each(["none", "before", "after", "both"] as const)( + "shallow snapshots null a %s style that no text is left in", + (expand) => { + const shallow = (styleDiesAfterRoot: boolean): Uint8Array => { + const doc = new runtime.LoroDoc(); + doc.setPeerId(1); + doc.setChangeMergeInterval(0); + doc.configTextStyle({ link: { expand } }); + const text = doc.getText("t"); + text.insert(0, "x"); + doc.commit(); + text.mark({ start: 0, end: 1 }, "link", "private"); + doc.commit(); + let root: Frontiers | undefined; + if (styleDiesAfterRoot) { + doc.getMap("meta").set("v", 0); + doc.commit(); + root = doc.frontiers(); + } + text.delete(0, 1); + doc.commit(); + doc.getMap("meta").set("v", 1); + doc.commit(); + return doc.export({ + mode: "shallow-snapshot", + frontiers: root ?? doc.frontiers(), + }); + }; + + // Dead at the root: the value is nulled there and in the latest state, + // except for a both-expand style, which still styles future inserts. + const dead = decodeFastSnapshot(shallow(false)); + const kept = expand === "both" ? "private" : null; + expect(markValues(dead.shallowRootState)).toEqual([kept]); + // Alive at the root: the latest state keeps it for historical checkout. + const late = decodeFastSnapshot(shallow(true)); + expect(markValues(late.shallowRootState)).toEqual(["private"]); + // Rust writes no latest state for so few ops; loro.js always does. + const writesLatest = runtimeName === "loro.js"; + expect(markValues(dead.state)).toEqual(writesLatest ? [kept] : []); + expect(markValues(late.state)).toEqual(writesLatest ? ["private"] : []); + const replica = new runtime.LoroDoc(); + replica.import(shallow(false)); + expect(replica.getText("t").toString()).toBe(""); + }, + ); + + test("a checkout that adds only a mark's start anchor changes no text", () => { + const doc = new runtime.LoroDoc(); + doc.setPeerId(5); + doc.configTextStyle({ bold: { expand: "after" } }); + const text = doc.getText("t"); + text.insert(0, "abc"); + doc.commit(); + text.mark({ start: 1, end: 3 }, "bold", "x"); + doc.commit(); + text.delete(1, 2); + doc.commit(); + + // 2@5 is before the mark, 3@5 has its start anchor but not its end. + const versions: Frontiers[] = [2, 3, 4, 3, 2, 4, 5, 3].map((counter) => [ + { peer: "5", counter }, + ]); + doc.checkout(versions[0]!); + let mirror = text.toDelta(); + doc.subscribe((batch) => { + for (const event of batch.events) { + if (event.diff.type === "text") { + mirror = applyDelta(mirror, event.diff.diff as Delta[]); + } + } + }); + for (const version of versions.slice(1)) { + doc.checkout(version); + expect(normalizeDelta(mirror)).toEqual(normalizeDelta(text.toDelta())); + } + }); +}); diff --git a/loro-js/tests/support/richtext-differential.ts b/loro-js/tests/support/richtext-differential.ts index 47d0a75dd..c719b8380 100644 --- a/loro-js/tests/support/richtext-differential.ts +++ b/loro-js/tests/support/richtext-differential.ts @@ -397,12 +397,19 @@ export class RichtextScenario { return; case "checkout": { const peer = this.#peer(action.peer); - if (peer.versions.length === 0) return; this.#commit(peer); - const version = peer.versions[action.version % peer.versions.length]!; - this.#both(peer, `checkout(${JSON.stringify(version)})`, (r) => - r.doc.checkout(version), - ); + // Two versions in a row, so transitions also run between historical + // versions, not only from and back to the latest one. + for (const roll of [ + action.version, + Math.imul(action.version, 0x9e37_79b1) >>> 0, + ]) { + const version = this.#versionToVisit(peer, roll); + if (version === undefined) continue; + this.#both(peer, `checkout(${JSON.stringify(version)})`, (r) => + r.doc.checkout(version), + ); + } this.#both(peer, "checkoutToLatest", (r) => r.doc.checkoutToLatest()); return; } @@ -461,6 +468,26 @@ export class RichtextScenario { } } + /** + * A recorded commit version, or for odd rolls the version right after a + * random op in the history. The latter can split a change or an op, such as + * a mark whose end anchor is not included yet. + */ + #versionToVisit(peer: Peer, roll: number): Frontiers | undefined { + if (roll % 2 === 1) { + const counters = [...peer.rust.doc.oplogVersion().toJSON()].filter( + ([, end]) => end > 0, + ); + if (counters.length > 0) { + const [id, end] = counters[(roll >>> 1) % counters.length]!; + return [{ peer: id, counter: (roll >>> 5) % end }]; + } + } + return peer.versions.length === 0 + ? undefined + : peer.versions[roll % peer.versions.length]; + } + #text(replica: Replica): Text { return replica.doc.getText("t"); } From ee884206104b5d8da37850738bacf0149107384c Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:29:45 +0800 Subject: [PATCH 07/25] test(loro.js): check out per-op versions in the differential suite Visits two versions per checkout, including versions right after a random op, so transitions between historical versions and mid-change versions are compared with Rust. README "Upgrading from 0.2" now reflects what a document loaded from a 0.2 snapshot does after its history is used. Co-Authored-By: Claude Opus 5.5 (1M context) --- .changeset/loro-js-text-interop.md | 2 +- loro-js/README.md | 44 +++++++++---------- .../tests/support/richtext-differential.ts | 37 +++++++++++++--- 3 files changed, 55 insertions(+), 28 deletions(-) diff --git a/.changeset/loro-js-text-interop.md b/.changeset/loro-js-text-interop.md index 7e9b31ab6..74e6b59af 100644 --- a/.changeset/loro-js-text-interop.md +++ b/.changeset/loro-js-text-interop.md @@ -11,4 +11,4 @@ Text now follows Rust (`loro-crdt`) everywhere, including when reading documents - Consecutive inserts in one transaction are merged into one op, and a delete that spans several ID runs writes the last run first, as Rust does. - Importing a concurrent insert next to a long run of text no longer walks the whole run. -Upgrading: a snapshot written by an earlier version keeps its current text, but its history (updates, JSON updates, checkout of older versions) is read like Rust, and a replica loaded from a snapshot and one loaded from updates can keep different text after new edits. Migrate every replica the same way; the safest path is to read the final state with the old version and build a new document from it. +Upgrading: updates and JSON updates written by an earlier version are read like Rust. A snapshot written by an earlier version keeps its current text, but its history reads like Rust, so a document loaded from it must not stay in use (exported updates, replicas loaded from updates, and checkouts can show other content). Migrate every replica the same way: read the final state (with the old version, or right after importing an old snapshot) and build a new document from it, or import the old updates everywhere and accept Rust's reading. See "Upgrading from 0.2" in the README. diff --git a/loro-js/README.md b/loro-js/README.md index ccbb38d73..b966de1f1 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -139,28 +139,28 @@ Documents also edited by `loro-crdt` peers already disagreed with Rust in these How 1.0 reads data written by 0.2 (measured on the example above): -| 0.2 data | Current text in 1.0 | History in 1.0 | -| --------------------- | ----------------------- | -------------------------------------------------------------------------- | -| Updates, JSON | Rust's reading (`béa9`) | Rust's reading | -| Full/shallow snapshot | Kept (`bé9a`) | Rust's reading: exported updates and checkouts of older versions follow it | - -**Migrate every replica the same way.** Replicas loaded from the same 0.2 snapshot -agree with each other, but a replica loaded from 0.2 updates holds other text, and -the two stay different after exchanging new edits (`béQ9aZ` and `béQa9Z`). A peer that -later joins through exported updates instead of a snapshot diverges the same way. - -Recommended paths: - -1. **Keep the content, drop the history (safest).** With 0.2, check out the latest - version and read the final state (`doc.toJSON()`, or `text.toDelta()` for rich - text). In 1.0, build one new document from it and share that document with every - replica. Do not import old updates or snapshots into it. -2. **Keep the history, accept Rust's reading.** Import the 0.2 updates (not snapshots) - on every replica, and check the content, because it can differ from what 0.2 - showed. - -A 0.2 snapshot keeps its current text, so loading it is only safe when every replica -loads that snapshot and new peers join through snapshots, never through updates. +| 0.2 data | 1.0 reads | +| --------------------- | ----------------------------------------------------------------------- | +| Updates, JSON updates | Rust's reading: `béa9` | +| Full/shallow snapshot | The 0.2 text (`bé9a`) as current state, but the history reads like Rust | + +A document loaded from a 0.2 snapshot therefore holds a current state that its own +history disagrees with. Its current state is safe to read right after the import, +but do not keep using the document: exporting it as updates gives `béa9`, a +replica loaded from updates holds other text and stays different after new edits +(`béQ9aZ` and `béQa9Z`), and checking out older versions and back can change the +content (Rust can then even show text that matches neither reading). + +**Migrate every replica the same way.** Recommended paths: + +1. **Keep the content, drop the history (safest).** Read the final state, either + with 0.2 after checking out the latest version, or with 1.0 right after importing + a 0.2 snapshot: `doc.toJSON()`, or `text.toDelta()` for rich text. Build one new + document from it in 1.0 and share that document with every replica. Do not + import old updates or snapshots into it. +2. **Keep the history, accept Rust's reading.** Import the 0.2 updates (not + snapshots) on every replica, and check the content, because it can differ from + what 0.2 showed. ## Development checks diff --git a/loro-js/tests/support/richtext-differential.ts b/loro-js/tests/support/richtext-differential.ts index 47d0a75dd..c719b8380 100644 --- a/loro-js/tests/support/richtext-differential.ts +++ b/loro-js/tests/support/richtext-differential.ts @@ -397,12 +397,19 @@ export class RichtextScenario { return; case "checkout": { const peer = this.#peer(action.peer); - if (peer.versions.length === 0) return; this.#commit(peer); - const version = peer.versions[action.version % peer.versions.length]!; - this.#both(peer, `checkout(${JSON.stringify(version)})`, (r) => - r.doc.checkout(version), - ); + // Two versions in a row, so transitions also run between historical + // versions, not only from and back to the latest one. + for (const roll of [ + action.version, + Math.imul(action.version, 0x9e37_79b1) >>> 0, + ]) { + const version = this.#versionToVisit(peer, roll); + if (version === undefined) continue; + this.#both(peer, `checkout(${JSON.stringify(version)})`, (r) => + r.doc.checkout(version), + ); + } this.#both(peer, "checkoutToLatest", (r) => r.doc.checkoutToLatest()); return; } @@ -461,6 +468,26 @@ export class RichtextScenario { } } + /** + * A recorded commit version, or for odd rolls the version right after a + * random op in the history. The latter can split a change or an op, such as + * a mark whose end anchor is not included yet. + */ + #versionToVisit(peer: Peer, roll: number): Frontiers | undefined { + if (roll % 2 === 1) { + const counters = [...peer.rust.doc.oplogVersion().toJSON()].filter( + ([, end]) => end > 0, + ); + if (counters.length > 0) { + const [id, end] = counters[(roll >>> 1) % counters.length]!; + return [{ peer: id, counter: (roll >>> 5) % end }]; + } + } + return peer.versions.length === 0 + ? undefined + : peer.versions[roll % peer.versions.length]; + } + #text(replica: Replica): Text { return replica.doc.getText("t"); } From 076202502d63319cb9c58393bbce645990a4eedc Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:32:07 +0800 Subject: [PATCH 08/25] docs(loro.js): read 0.2 rich text with Rust's positions The decision and its reasons are in context/loro-js-richtext-anchors.md ("Data written by loro.js 0.2"). The changeset becomes major and lists the breaking changes; README "Upgrading from 0.2" covers rich text; a 0.2.1 fixture pins how loro.js and Rust read its rich-text ops. Co-Authored-By: Claude Opus 5.5 (1M context) --- .changeset/loro-js-richtext-anchors.md | 14 ++-- context/loro-js-richtext-anchors.md | 33 ++++++++- loro-js/README.md | 15 ++-- loro-js/scripts/write-legacy-fixtures.mjs | 20 ++++++ .../tests/fixtures/loro-js-0.2/expected.json | 13 +++- .../loro-js-0.2/richtext-positions.json | 65 ++++++++++++++++++ .../richtext-positions.snapshot.blob | Bin 0 -> 304 bytes .../richtext-positions.update.blob | Bin 0 -> 122 bytes loro-js/tests/legacy-data.test.ts | 31 ++++++++- 9 files changed, 177 insertions(+), 14 deletions(-) create mode 100644 loro-js/tests/fixtures/loro-js-0.2/richtext-positions.json create mode 100644 loro-js/tests/fixtures/loro-js-0.2/richtext-positions.snapshot.blob create mode 100644 loro-js/tests/fixtures/loro-js-0.2/richtext-positions.update.blob diff --git a/.changeset/loro-js-richtext-anchors.md b/.changeset/loro-js-richtext-anchors.md index 5e27ad710..b7c7530f3 100644 --- a/.changeset/loro-js-richtext-anchors.md +++ b/.changeset/loro-js-richtext-anchors.md @@ -1,15 +1,19 @@ --- -"loro.js": minor +"loro.js": major --- -Rich text now uses Rust's position model. A mark's start and end anchors are elements of the text sequence, as in Rust (`loro-crdt`), so every insert, delete, and mark position loro.js writes or reads counts them. Before, loro.js wrote and read positions without the anchors, so as soon as a document had a mark, loro.js and Rust applied each other's later edits at different places (for example, after marking `ab` in `abcd`, inserting `X` at 3 and deleting 1 gave `acXd` in Rust but `acdX` when loro.js imported Rust's update, and `bXcd` when Rust imported loro.js's). +Rich text now uses Rust's position model. A mark's start and end anchors are elements of the text sequence, as in Rust (`loro-crdt`), so every insert, delete, and mark position loro.js writes or reads counts them. + +**Breaking:** loro.js reads rich-text history written by earlier versions with Rust's positions. Earlier versions wrote and read positions without the anchors, so every edit made after a mark in a document edited only with loro.js can now apply elsewhere, and styles can cover other text (for example, after marking `ab` in `abcd`, inserting `X` at 3 and deleting 1 gave `acXd` in earlier versions; those ops now read as `bXcd`, as Rust has always read them). Documents shared with `loro-crdt` peers had already diverged there (Rust read `bXcd` from loro.js's updates, loro.js read `acdX` from Rust's); they now agree with Rust. There is no version marker in the data: read "Upgrading from 0.2" in the README before upgrading. Behavior now follows Rust: - Text inserted where a mark starts or ends goes inside or outside the mark according to the mark's expand setting, as Rust decides it; concurrent edits at a mark boundary converge with Rust. -- `mark` and `unmark` skip a mark that would change nothing, and throw when `start >= end`. +- **Breaking:** `mark` and `unmark` throw when `start >= end`, and skip a mark that would change nothing, as Rust does. +- **Breaking:** internal `_`-prefixed members of `LoroText` that were visible in the type declarations (`_insertVisible`, `_applyMark`, `_styleRuns`, `_validateInsertPosition`, `_unicodePosition`, `_detachedStyleCounter`) were removed. - `applyDelta` drops a style that inserted text would inherit when the insert does not list it, like Rust's `apply_delta`. - Undo and redo restore the attributes a mark replaced. -- Snapshots write and read the anchors like Rust, so checking out an older version of a styled text gives the same result in both runtimes. +- Snapshots write and read the anchors like Rust, so checking out an older version of a styled text gives the same result in both runtimes. Shallow snapshots null the values of styles that no text is left in, like Rust. +- Inserting into a nested text that a snapshot loaded lazily no longer throws or drops the styles at the insert position, and checking out a version that has a mark's start but not its end no longer reports the whole text as inserted. -Documents that loro.js edited after a mark with an earlier version contain positions that only loro.js read that way; Rust has always read them with the anchors, and loro.js now does too. +Upgrading: a snapshot written by an earlier version keeps its current rich text, but its history reads like Rust, so a document loaded from it must not stay in use. Read the final state (`text.toDelta()`, with the old version or right after importing an old snapshot) and build a new document from it, or import the old updates everywhere and accept Rust's reading. diff --git a/context/loro-js-richtext-anchors.md b/context/loro-js-richtext-anchors.md index c9e26be7b..375154cfe 100644 --- a/context/loro-js-richtext-anchors.md +++ b/context/loro-js-richtext-anchors.md @@ -98,7 +98,11 @@ Remote ops: - A mark's event is emitted with its end anchor and covers the text whose resolved value changed (`LoroText._styleChangeRuns`). - Checkout toggles anchors like inserted text and reports style changes through - `TextStyleIndex.transitions` over the style's member runs. + `TextStyleIndex.transitions` over the style's member runs. Anchors take the + bulk path of `#applyVersionTransition`, which records no event, so the text's + event baseline is started when an anchor op is collected; without it, a + version with a start anchor but no end anchor was diffed against an empty + text and reported the whole text as inserted. - Undoing a mark gives back, wherever it changed the value, the value from just before its op (`LoroText._undoStyle`). That matches Rust for local histories; loro.js undo still does not transform against remote edits the way Rust's @@ -112,7 +116,16 @@ document order. A start anchor is a zero-length span followed by its mark entry; an end anchor is a span of length -1 at the start counter + 1 that keeps the start's lamport offset. Contiguous text is merged into one span. `#hydrateContainerState` rebuilds the same elements and each style's range -(`LoroText._appendElements`). +(`LoroText._appendElements`). A Text loaded from a snapshot hydrates lazily; +every position lookup (`#entityFrom`, `#insertPosition`) hydrates first, because +the anchors decide where inserted text goes. + +Shallow snapshots null the value of every style with no text between its anchors +at the shallow root, except both-expand styles, and null the same styles in the +latest state (`redactDeadStyleValues` in `document.ts`, Rust's +`redact_dead_style_values`; see +[shallow-snapshot-style-redaction.md](shallow-snapshot-style-redaction.md)). +Anchors and IDs stay, so positions do not move. ## Complexity @@ -124,6 +137,22 @@ start's lamport offset. Contiguous text is merged into one span. - Inserting inside styled text intersects two style histories per insert, which is proportional to the number of styles covering that position. +## Data written by loro.js 0.2 + +loro.js 0.2 wrote and read Text positions without the anchors. Decision +(2026-09-28): later versions read all data with Rust's positions and add no +version marker, like the plain-text changes in +[loro-js-rust-differential.md](loro-js-rust-differential.md). Reasons: a Text +op position must mean the same thing in both runtimes; documents shared with +`loro-crdt` peers had already diverged (each runtime read the other's ops at +other positions); and a marker would need an encoding change that Rust does +not have. The cost: in a document edited only with 0.2, every op after a mark +can apply elsewhere, and a 0.2 snapshot keeps its 0.2 state while its history +reads like Rust (a checkout there and back switches to Rust's reading). The +package version is major; `loro-js/README.md` ("Upgrading from 0.2") gives the +migration paths, and `loro-js/tests/legacy-data.test.ts` pins the readings of +0.2 fixtures in both runtimes. + ## Testing - `loro-js/tests/richtext-anchors.test.ts` compares loro.js with Rust fixtures diff --git a/loro-js/README.md b/loro-js/README.md index b7c98480f..f7ee96230 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -132,6 +132,11 @@ can read differently after the upgrade: - **Concurrent text inserts.** 0.2 could order an insert after a sibling's subtree differently from Rust. 1.0 orders it like Rust, so such a history can give other text (`bé9a` in 0.2, `béa9` in 1.0 and Rust). +- **Rich text.** 0.2 counted text positions without a mark's start and end anchors; + Rust and 1.0 count them. Every insert, delete, and mark op that 0.2 wrote after a + mark can therefore apply elsewhere (with bold on `ab` in `abcd`, inserting `X` at + 3 and deleting 1 gave `acXd` in 0.2; its ops read as `bXcd` in 1.0 and Rust), and + styles can cover other text. - **Cursors.** 0.2 encoded the UTF-16 position of a cursor's target; Rust and 1.0 use the Unicode position. A cursor encoded by 0.2 can resolve to another offset (on `ab😀`, 4 in 0.2 and 2 in 1.0). Get new cursors from the upgraded document. @@ -139,12 +144,12 @@ can read differently after the upgrade: Documents also edited by `loro-crdt` peers already disagreed with Rust in these cases; 1.0 makes them agree. -How 1.0 reads data written by 0.2 (measured on the example above): +How 1.0 reads data written by 0.2 (measured on the examples above): -| 0.2 data | 1.0 reads | -| --------------------- | ----------------------------------------------------------------------- | -| Updates, JSON updates | Rust's reading: `béa9` | -| Full/shallow snapshot | The 0.2 text (`bé9a`) as current state, but the history reads like Rust | +| 0.2 data | 1.0 reads | +| --------------------- | ------------------------------------------------------------------------------- | +| Updates, JSON updates | Rust's reading: `béa9`, `bXcd` | +| Full/shallow snapshot | The 0.2 text (`bé9a`, `acXd`) as current state, but the history reads like Rust | A document loaded from a 0.2 snapshot therefore holds a current state that its own history disagrees with. Its current state is safe to read right after the import, diff --git a/loro-js/scripts/write-legacy-fixtures.mjs b/loro-js/scripts/write-legacy-fixtures.mjs index 6889f676d..71574b41f 100644 --- a/loro-js/scripts/write-legacy-fixtures.mjs +++ b/loro-js/scripts/write-legacy-fixtures.mjs @@ -62,4 +62,24 @@ function sync(from, to) { expected["cursor-emoji"] = owner.getCursorPos(cursor).offset; } +// Marks whose anchors 0.2 left out of text positions: with bold on "ab" in +// "abcd", inserting X at 3 and deleting 1 gave "acXd" in 0.2 (and Rust), but +// the ops 0.2 wrote mean "bXcd" to Rust. +{ + const owner = doc(1); + owner.configTextStyle({ bold: { expand: "after" } }); + const text = owner.getText("t"); + text.insert(0, "abcd"); + owner.commit(); + text.mark({ start: 0, end: 2 }, "bold", true); + owner.commit(); + text.insert(3, "X"); + text.delete(1, 1); + owner.commit(); + write("richtext-positions.update.blob", owner.export({ mode: "update" })); + write("richtext-positions.snapshot.blob", owner.export({ mode: "snapshot" })); + write("richtext-positions.json", JSON.stringify(owner.exportJsonUpdates(), null, 2)); + expected["richtext-positions"] = text.toDelta(); +} + write("expected.json", `${JSON.stringify(expected, null, 2)}\n`); diff --git a/loro-js/tests/fixtures/loro-js-0.2/expected.json b/loro-js/tests/fixtures/loro-js-0.2/expected.json index 79106f599..e40edbbea 100644 --- a/loro-js/tests/fixtures/loro-js-0.2/expected.json +++ b/loro-js/tests/fixtures/loro-js-0.2/expected.json @@ -1,4 +1,15 @@ { "text-concurrent": "bé9a", - "cursor-emoji": 4 + "cursor-emoji": 4, + "richtext-positions": [ + { + "insert": "a", + "attributes": { + "bold": true + } + }, + { + "insert": "cXd" + } + ] } diff --git a/loro-js/tests/fixtures/loro-js-0.2/richtext-positions.json b/loro-js/tests/fixtures/loro-js-0.2/richtext-positions.json new file mode 100644 index 000000000..d0b6e399d --- /dev/null +++ b/loro-js/tests/fixtures/loro-js-0.2/richtext-positions.json @@ -0,0 +1,65 @@ +{ + "schema_version": 1, + "start_version": {}, + "peers": [ + "1" + ], + "changes": [ + { + "id": "0@0", + "timestamp": 0, + "deps": [], + "lamport": 0, + "msg": null, + "ops": [ + { + "container": "cid:root-t:Text", + "content": { + "type": "insert", + "pos": 0, + "text": "abcd" + }, + "counter": 0 + }, + { + "container": "cid:root-t:Text", + "content": { + "type": "mark", + "start": 0, + "end": 2, + "style_key": "bold", + "style_value": true, + "info": 132 + }, + "counter": 4 + }, + { + "container": "cid:root-t:Text", + "content": { + "type": "mark_end" + }, + "counter": 5 + }, + { + "container": "cid:root-t:Text", + "content": { + "type": "insert", + "pos": 3, + "text": "X" + }, + "counter": 6 + }, + { + "container": "cid:root-t:Text", + "content": { + "type": "delete", + "pos": 1, + "len": 1, + "start_id": "1@0" + }, + "counter": 7 + } + ] + } + ] +} \ No newline at end of file diff --git a/loro-js/tests/fixtures/loro-js-0.2/richtext-positions.snapshot.blob b/loro-js/tests/fixtures/loro-js-0.2/richtext-positions.snapshot.blob new file mode 100644 index 0000000000000000000000000000000000000000..f2bb60d3dd65f2be8ab625d894a275dd299b6aaa GIT binary patch literal 304 zcmYjMI|{-;5Pffd`H6S}Yj5EN0s%p=vCsr93tSmS5d z+1Yu#o%g1!>I#1`o?jo(CqdF|k^vz?4#3^Jyx^Eg6rUd1!W new Uint8Array(readFileSync(fixtureUrl(name))); const json = (name: string): unknown => JSON.parse(readFileSync(fixtureUrl(name), "utf8")); -const legacy = json("expected.json") as Record; +const legacy = json("expected.json") as Record; interface Runtime { readonly LoroDoc: typeof LoroDoc; @@ -51,6 +51,35 @@ describe.each(runtimes)("documents written by loro.js 0.2, read by %s", (_, runt expect(history.getText("t").toString()).toBe("béa9"); }); + test("reads rich-text positions with Rust's anchors", () => { + expect(legacy["richtext-positions"]).toEqual([ + { insert: "a", attributes: { bold: true } }, + { insert: "cXd" }, + ]); + const styled = (load: (doc: LoroDoc) => void): LoroDoc => + read((doc) => { + doc.configTextStyle({ bold: { expand: "after" } }); + load(doc); + }); + // The ops 0.2 wrote left the mark's anchors out of their positions. + const rustReading = [{ insert: "bX", attributes: { bold: true } }, { insert: "cd" }]; + const fromUpdate = styled((doc) => + doc.import(fixture("richtext-positions.update.blob")), + ); + expect(fromUpdate.getText("t").toDelta()).toEqual(rustReading); + const fromJson = styled((doc) => + doc.importJsonUpdates(json("richtext-positions.json") as JsonSchema), + ); + expect(fromJson.getText("t").toDelta()).toEqual(rustReading); + // A snapshot keeps the 0.2 state, but its history reads like Rust. + const fromSnapshot = styled((doc) => + doc.import(fixture("richtext-positions.snapshot.blob")), + ); + expect(fromSnapshot.getText("t").toDelta()).toEqual(legacy["richtext-positions"]); + const history = styled((doc) => doc.import(fromSnapshot.export({ mode: "update" }))); + expect(history.getText("t").toDelta()).toEqual(rustReading); + }); + test("resolves a 0.2 cursor by its Unicode position", () => { expect(legacy["cursor-emoji"]).toBe(4); const owner = read((doc) => doc.import(fixture("cursor-emoji.snapshot.blob"))); From ccd06e02d8d0f70319be29c943f26d82889b2865 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:39:23 +0800 Subject: [PATCH 09/25] docs(loro.js): describe per-op checkouts in the differential suite Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-rust-differential.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/context/loro-js-rust-differential.md b/context/loro-js-rust-differential.md index c72addc6f..31372048a 100644 --- a/context/loro-js-rust-differential.md +++ b/context/loro-js-rust-differential.md @@ -33,7 +33,10 @@ per runtime with the same peer ID. Actions: - `sync` by binary or JSON updates, within a runtime or crossed (Rust imports the loro.js peer's updates and loro.js imports Rust's), so each side reads the other's positions; -- `checkout` to a recorded version and back; +- `checkout` to two versions in a row, then back to the latest: each is a + recorded commit version or the version right after a random op, which can + split a change or an op (a mark with only its start anchor, for example), so + transitions also run between historical and mid-change versions; - full and shallow snapshot export from either runtime, imported into both; - cursor creation, then cursor resolution after every later step; - `revertTo` and undo/redo. From b1e4bade6ee0144111b8654242603d54d0382a96 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:45:14 +0800 Subject: [PATCH 10/25] docs(loro.js): record Fugue subtree-end measurements Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 4c1ec6283..b2a78a3bb 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -111,7 +111,12 @@ JavaScript constant factor. Inside a physical ID run every element's origin-left is its predecessor, so the walk jumps between run starts (`SequenceIndex.nextPhysicalIdRunStart`, skipping single-run treap subtrees): O((runs in the subtree + 1) log n), and - O(log n) after a long typed run (`text-concurrent-insert-after-long-run`). + O(log n) after a long typed run. With a warm origin index, importing one + concurrent character after a 512k-character run takes 0.2–0.3 ms (0.2–0.45 ms + on `main`, which stopped at the origin-right bound and could misorder; 8.9–14.8 + ms for the per-scalar walk), and `text-concurrent-insert-after-long-run` stays + at 0.15–0.24 ms from 64k to 512k characters (2.4 → 12.8 ms per-scalar; + September 28, Node 22, 1-minute load 5–25, interleaved). - An imported Text delete is resolved by its position in the op's causal view, like Rust's tracker (`LoroText._deleteTargets`): O(log n + runs) through `visibleIdRuns` or the cached causal view. The recorded `start_id` is only a From b842514da920aa0319797364faca12a5a7e9e57f Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 09:45:27 +0800 Subject: [PATCH 11/25] docs(loro.js): record the accepted rich-text anchor costs Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 308ddae1d..f6c76afa7 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -302,6 +302,17 @@ build takes 7.1/19/362 ms to retreat 1k/4k/16k such marks, and building the 16k history takes 402 s in loro.js and 542 s in Rust. Every other `bench:complexity` entry stays flat from 1k to 8k. +These costs were reviewed and accepted (loro-dev/loro#1137). The review measured, +on Node 26 at 1-minute load 25–40, typing 1,000 characters inside a 64k bold +range at 2.9–4.4 ms (1.9 ms on `main`) and the 8k repeated-mark tail retreat and +restore at 8.0–8.1 and 11.6–13.2 ms (0.4–0.6 ms on `main`). Re-measured after the +review fixes (Node 22, interleaved with `main`, 1-minute load 5–25): the repeated- +mark tail retreat takes 0.51/1.18/2.25/6.11 ms and its restore 0.47/1.16/2.08/5.89 +ms at 1k/2k/4k/8k (0.08–0.41 ms on `main`); a 64k full-range mark applies in +0.11–0.15 ms (0.06–0.12), retreats in 0.22–0.27 ms (0.12–0.14), and restores in +0.11–0.14 ms (0.02–0.03); typing 1,000 characters inside it varies with load +(2.4–6.6 ms here, 2.6–4.3 ms on `main` in the same alternating runs). + A subscribed forward checkout that combines a full-range delete and mark takes 0.41 ms at 1k characters and 0.16 ms at 8k after warmup. Historical mark positions are converted directly to causal ID runs, and removed ID runs are From c14f3164ef3b979bfef2143ea7c8eb3a5e40c979 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 17:38:45 +0800 Subject: [PATCH 12/25] docs(loro.js): record Fugue measurements after adopting #1139 Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 803f04d04..af0588eb1 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -119,8 +119,11 @@ JavaScript constant factor. search takes 7.3 ms versus 11.1 ms, but on a deep chain of explicit links (two positions typed alternately, 64k elements) 60.6 ms versus 15.5 ms, because each of its O(log n) probes walks the chain. Realistic traces favor - the binary search. `text-concurrent-insert-after-long-run` covers the long - typed run. + the binary search. With a warm origin index, importing one concurrent + character after a 512k-character typed run takes 0.28–0.31 ms (0.29–0.30 ms + on `main`, which misorders other cases; a per-scalar walk took 9.7–10.7 ms), + and `text-concurrent-insert-after-long-run` stays at 0.18–0.30 ms from 64k to + 512k characters (September 28, Node 22, interleaved, 1-minute load 7–17). - An imported Text delete is resolved by its position in the op's causal view, like Rust's tracker (`LoroText._deleteTargets`): O(log n + runs) through `visibleIdRuns` or the cached causal view. The recorded `start_id` is only a From d0d652d9f624695c862581db5baebad59157ee1f Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Mon, 28 Sep 2026 17:38:54 +0800 Subject: [PATCH 13/25] docs(loro.js): add the rerun rich-text measurements Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 86b46540a..34eb7eb09 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -313,12 +313,13 @@ These costs were reviewed and accepted (loro-dev/loro#1137). The review measured on Node 26 at 1-minute load 25–40, typing 1,000 characters inside a 64k bold range at 2.9–4.4 ms (1.9 ms on `main`) and the 8k repeated-mark tail retreat and restore at 8.0–8.1 and 11.6–13.2 ms (0.4–0.6 ms on `main`). Re-measured after the -review fixes (Node 22, interleaved with `main`, 1-minute load 5–25): the repeated- -mark tail retreat takes 0.51/1.18/2.25/6.11 ms and its restore 0.47/1.16/2.08/5.89 -ms at 1k/2k/4k/8k (0.08–0.41 ms on `main`); a 64k full-range mark applies in -0.11–0.15 ms (0.06–0.12), retreats in 0.22–0.27 ms (0.12–0.14), and restores in -0.11–0.14 ms (0.02–0.03); typing 1,000 characters inside it varies with load -(2.4–6.6 ms here, 2.6–4.3 ms on `main` in the same alternating runs). +review fixes (two runs of two interleaved rounds with `main`, Node 22, 1-minute +load 5–25): the repeated-mark tail retreat takes 0.51–0.71/1.18–1.27/2.25–2.80/ +6.11–8.25 ms and its restore 0.47–0.53/1.16–1.35/2.08–2.57/5.89–7.75 ms at +1k/2k/4k/8k (0.08–0.43 ms on `main`); a 64k full-range mark applies in 0.11–0.17 +ms (0.06–0.12), retreats in 0.22–0.35 ms (0.11–0.14), and restores in 0.11–0.14 +ms (0.02–0.03); typing 1,000 characters inside it varies with load (2.4–6.6 ms, +2.6–4.3 ms on `main` in the same alternating runs). A subscribed forward checkout that combines a full-range delete and mark takes 0.41 ms at 1k characters and 0.16 ms at 8k after warmup. Historical mark From 10bc18dcb4709ff1f0abad48cbc0993b97f9e070 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 15:06:31 +0800 Subject: [PATCH 14/25] refactor(loro.js): drop the styled-text replay check that anchors made moot #1126 checked snapshot-hydrated styled Text before every full replay because loro.js did not count the style anchors, so a replay of Rust rich text could differ from its snapshot state. With the anchors in the sequence, styled Text replays like plain Text; the unreplayable fallback stays for snapshots whose state disagrees with their own history (context/loro-js-performance.md). Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 30 ++--- context/loro-js-richtext-anchors.md | 4 +- context/loro-js-rust-differential.md | 9 +- loro-js/README.md | 6 +- loro-js/src/runtime/document.ts | 109 ++++--------------- loro-js/tests/legacy-data.test.ts | 157 +++++++++++++++++---------- 6 files changed, 148 insertions(+), 167 deletions(-) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index eedfb9634..60f69e53a 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -178,12 +178,16 @@ JavaScript constant factor. copies every untouched entry and rewrites only the touched ones. A delete transition is still refused unless the deletion index recorded that delete, for example one imported while detached. -- loro.js does not count Rust rich-text style anchors in Text operation - positions, so a replay of Rust-created styled text can differ from its - snapshot state (for example, an insert right after a mark's end anchor). The - completion therefore compares the replay of a Text or List with the snapshot - state (visible ids, plus the delta when styled). When they differ, the replay - is discarded and the container becomes `unreplayable`: it keeps its snapshot +- The completion compares the replay of a Text or List with the snapshot state + (visible ids, plus the values). With style anchors in the sequence + (loro-dev/loro#1137) a replay of Rust or loro.js history, styled or not, + equals its snapshot state; before, loro.js did not count the anchors and a + replay of Rust-created styled text could differ. A replay now differs only + when the snapshot state disagrees with its own history: a snapshot written by + loro.js 0.2 (see `loro-js/README.md`, "Upgrading from 0.2"), or a loro.js + shallow snapshot with the retained-range gap described in + [loro-js-rust-differential.md](loro-js-rust-differential.md). When they + differ, the replay is discarded and the container becomes `unreplayable`: it keeps its snapshot state, encoded once, and is never given a replay. A transition that touches it runs without its operations and then moves it separately (`#planSnapshotStates`): when the installed state already has every forward @@ -237,13 +241,13 @@ JavaScript constant factor. the latest state and `revertTo` are right). This is loro.js's MovableList transition gap, left to loro-dev/loro#1132. - A full `#rebuildFromHistory` (the non-incremental fallback, shallow export, - `forkAt`) first checks the snapshot-hydrated Text containers that have or - had styles, including lazily encoded ones, and then rebuilds unreplayable - containers the same way. "Had" matters: when every marked character was - deleted, the snapshot state has no style, but the mark's anchors still shift - later Rust positions, so the replay differs (`#hasStyleHistory`: a mark - operation in the history, indexed as `#markedTexts`, or styles in the shallow - root state). Plain text is not checked, so it pays no extra replay. Only the root state of a shallow export uses the replay, since the + `forkAt`) rebuilds unreplayable containers the same way. It no longer checks + snapshot-hydrated styled Text first (`#checkSnapshotSequences`, removed in + loro-dev/loro#1137): that extra replay per styled Text existed because the + anchors shifted Rust positions, which the anchor model now counts. So styled + and plain Text behave alike: a hydrated container that no transition has + completed takes the replay of its history, which for a 0.2 snapshot is Rust's + reading. Only the root state of a shallow export uses the replay, since the snapshot state is later than the root. `forkAt` keeps a snapshot state only in a fork whose version includes that state's version; an older fork has none of the operations needed to undo later ones in that state, so it keeps diff --git a/context/loro-js-richtext-anchors.md b/context/loro-js-richtext-anchors.md index 375154cfe..64e326aea 100644 --- a/context/loro-js-richtext-anchors.md +++ b/context/loro-js-richtext-anchors.md @@ -148,7 +148,9 @@ op position must mean the same thing in both runtimes; documents shared with other positions); and a marker would need an encoding change that Rust does not have. The cost: in a document edited only with 0.2, every op after a mark can apply elsewhere, and a 0.2 snapshot keeps its 0.2 state while its history -reads like Rust (a checkout there and back switches to Rust's reading). The +reads like Rust (checkouts keep that state through the `unreplayable` fallback +of loro-dev/loro#1126 and show older versions approximately; a fork or shallow +export rebuilds it from history, which is Rust's reading). The package version is major; `loro-js/README.md` ("Upgrading from 0.2") gives the migration paths, and `loro-js/tests/legacy-data.test.ts` pins the readings of 0.2 fixtures in both runtimes. diff --git a/context/loro-js-rust-differential.md b/context/loro-js-rust-differential.md index 1f954c5ba..fd44bcc1f 100644 --- a/context/loro-js-rust-differential.md +++ b/context/loro-js-rust-differential.md @@ -80,8 +80,13 @@ tolerates them, so it also runs against older reference builds: can diverge from its own full-history import when they are wrong. The harness compares loro.js with the source document instead. -Snapshot-import checkout (`snapshotCheckout`) is off by default until loro.js -can check out a document imported from a snapshot (loro-dev/loro#1126). +Snapshot-import checkout (`snapshotCheckout`) is off by default. Since +loro-dev/loro#1126, checkout after importing a full snapshot matches Rust, but +checkout into the retained range of a shallow snapshot that loro.js exported +still diverges: about 20% of 120-step seeds, plain and rich text alike +(September 29). The replay of such a snapshot differs from its state, so the +text becomes `unreplayable` (see [loro-js-performance.md](loro-js-performance.md)); +the export gap itself is not fixed yet. ## Data written by loro.js 0.2 diff --git a/loro-js/README.md b/loro-js/README.md index f7ee96230..443947ee1 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -155,8 +155,10 @@ A document loaded from a 0.2 snapshot therefore holds a current state that its o history disagrees with. Its current state is safe to read right after the import, but do not keep using the document: exporting it as updates gives `béa9`, a replica loaded from updates holds other text and stays different after new edits -(`béQ9aZ` and `béQa9Z`), and checking out older versions and back can change the -content (Rust can then even show text that matches neither reading). +(`béQ9aZ` and `béQa9Z`), older versions it checks out are approximate (`acd` for +`abcd`), and a fork or a shallow-snapshot export can rebuild it from its history, which +switches it to Rust's reading (Rust can even show text that matches neither +reading after a checkout there and back). **Migrate every replica the same way.** Recommended paths: diff --git a/loro-js/src/runtime/document.ts b/loro-js/src/runtime/document.ts index 64479a5a9..ffa90c082 100644 --- a/loro-js/src/runtime/document.ts +++ b/loro-js/src/runtime/document.ts @@ -9,11 +9,7 @@ import { type DecodedOperationContent, } from "../codec/change-block-codec"; import type { ChangeLoroValue, ChangeValue } from "../codec/change-value"; -import { - containerTypeFromRawByte, - containerTypeToRawByte, - decodeContainerId, -} from "../codec/container-id"; +import { containerTypeFromRawByte, containerTypeToRawByte } from "../codec/container-id"; import { decodeDocument, decodeFastSnapshotBody, @@ -28,7 +24,6 @@ import { import { decodeChangeBlockKey, encodeChangeBlockKey } from "../codec/id"; import { decodeSstable, encodeSstable, type SstableEntry } from "../codec/sstable"; import { - decodeContainerStateWrapper, decodeLazyStateSnapshotStore, decodeStateSnapshotStore, encodeStateSnapshotStore, @@ -291,9 +286,6 @@ export class LoroDoc = Record>(); #movableMovePeers = new Map>(); #containersWithOperations = new Set(); - // Text containers with a mark operation in their history - // (#checkSnapshotSequences). - #markedTexts = new Set(); #containerKeys = new WeakMap(); #pendingHistory = new Map(); #deferredSnapshotHistory: DeferredSnapshotHistory | undefined; @@ -1192,7 +1184,7 @@ export class LoroDoc = Record = Record = Record = Record = Record(); fork.#recordTimestamp = this.#recordTimestamp; fork.#changeMergeInterval = this.#changeMergeInterval; @@ -2006,7 +1997,7 @@ export class LoroDoc = Record = Record = Record = Record 0; - } - /** * Readies the containers that `records` touch for a transition from * `current`. A lazily encoded container that no transition has touched still @@ -5266,9 +5197,11 @@ export class LoroDoc = Record = Record = Record = Record = Record(); - this.#rebuildFromHistory(rootVersion, restoreVersion, true); + this.#rebuildFromHistory(rootVersion, true); const rootRetained = this.#retainedContainerKeys(); const rootStore = this.#buildStateStore(startFrontiers, rootRetained, { redacted }); - this.#rebuildFromHistory(latestVersion, rootVersion); + this.#rebuildFromHistory(latestVersion); const latestRetained = this.#retainedContainerKeys(); for (const key of rootRetained) latestRetained.add(key); for (const [key, container] of this.#containers) { @@ -6420,7 +6349,7 @@ export class LoroDoc = Record { - const read = (load: (doc: LoroDoc) => void): LoroDoc => { - const doc = new runtime.LoroDoc(); - load(doc); - return doc; - }; +describe.each(runtimes)( + "documents written by loro.js 0.2, read by %s", + (name, runtime) => { + const read = (load: (doc: LoroDoc) => void): LoroDoc => { + const doc = new runtime.LoroDoc(); + load(doc); + return doc; + }; - test("orders concurrent inserts from history like Rust", () => { - expect(legacy["text-concurrent"]).toBe("bé9a"); - const fromUpdate = read((doc) => doc.import(fixture("text-concurrent.update.blob"))); - expect(fromUpdate.getText("t").toString()).toBe("béa9"); - const fromJson = read((doc) => - doc.importJsonUpdates(json("text-concurrent.json") as JsonSchema), - ); - expect(fromJson.getText("t").toString()).toBe("béa9"); - }); + test("orders concurrent inserts from history like Rust", () => { + expect(legacy["text-concurrent"]).toBe("bé9a"); + const fromUpdate = read((doc) => + doc.import(fixture("text-concurrent.update.blob")), + ); + expect(fromUpdate.getText("t").toString()).toBe("béa9"); + const fromJson = read((doc) => + doc.importJsonUpdates(json("text-concurrent.json") as JsonSchema), + ); + expect(fromJson.getText("t").toString()).toBe("béa9"); + }); - test("keeps a snapshot's current text but reads its history like Rust", () => { - const fromSnapshot = read((doc) => - doc.import(fixture("text-concurrent.snapshot.blob")), - ); - expect(fromSnapshot.getText("t").toString()).toBe("bé9a"); - const history = read((doc) => doc.import(fromSnapshot.export({ mode: "update" }))); - expect(history.getText("t").toString()).toBe("béa9"); - }); + test("keeps a snapshot's current text but reads its history like Rust", () => { + const fromSnapshot = read((doc) => + doc.import(fixture("text-concurrent.snapshot.blob")), + ); + expect(fromSnapshot.getText("t").toString()).toBe("bé9a"); + const history = read((doc) => doc.import(fromSnapshot.export({ mode: "update" }))); + expect(history.getText("t").toString()).toBe("béa9"); + // State and history disagree here, so the runtimes' forks differ: loro.js + // replays the history, Rust copies the state. + const fork = fromSnapshot.fork(); + expect(fork.getText("t").toString()).toBe(name === "loro.js" ? "béa9" : "bé9a"); + }); - test("reads rich-text positions with Rust's anchors", () => { - expect(legacy["richtext-positions"]).toEqual([ - { insert: "a", attributes: { bold: true } }, - { insert: "cXd" }, - ]); - const styled = (load: (doc: LoroDoc) => void): LoroDoc => - read((doc) => { - doc.configTextStyle({ bold: { expand: "after" } }); - load(doc); - }); - // The ops 0.2 wrote left the mark's anchors out of their positions. - const rustReading = [{ insert: "bX", attributes: { bold: true } }, { insert: "cd" }]; - const fromUpdate = styled((doc) => - doc.import(fixture("richtext-positions.update.blob")), - ); - expect(fromUpdate.getText("t").toDelta()).toEqual(rustReading); - const fromJson = styled((doc) => - doc.importJsonUpdates(json("richtext-positions.json") as JsonSchema), - ); - expect(fromJson.getText("t").toDelta()).toEqual(rustReading); - // A snapshot keeps the 0.2 state, but its history reads like Rust. - const fromSnapshot = styled((doc) => - doc.import(fixture("richtext-positions.snapshot.blob")), + test("reads rich-text positions with Rust's anchors", () => { + expect(legacy["richtext-positions"]).toEqual([ + { insert: "a", attributes: { bold: true } }, + { insert: "cXd" }, + ]); + const styled = (load: (doc: LoroDoc) => void): LoroDoc => + read((doc) => { + doc.configTextStyle({ bold: { expand: "after" } }); + load(doc); + }); + // The ops 0.2 wrote left the mark's anchors out of their positions. + const rustReading = [ + { insert: "bX", attributes: { bold: true } }, + { insert: "cd" }, + ]; + const fromUpdate = styled((doc) => + doc.import(fixture("richtext-positions.update.blob")), + ); + expect(fromUpdate.getText("t").toDelta()).toEqual(rustReading); + const fromJson = styled((doc) => + doc.importJsonUpdates(json("richtext-positions.json") as JsonSchema), + ); + expect(fromJson.getText("t").toDelta()).toEqual(rustReading); + // A snapshot keeps the 0.2 state, but its history reads like Rust. + const fromSnapshot = styled((doc) => + doc.import(fixture("richtext-positions.snapshot.blob")), + ); + expect(fromSnapshot.getText("t").toDelta()).toEqual(legacy["richtext-positions"]); + const history = styled((doc) => + doc.import(fromSnapshot.export({ mode: "update" })), + ); + expect(history.getText("t").toDelta()).toEqual(rustReading); + // State and history disagree, so the runtimes' forks differ: loro.js + // replays the history, Rust copies the state. + const fork = fromSnapshot.fork(); + fork.configTextStyle({ bold: { expand: "after" } }); + expect(fork.getText("t").toDelta()).toEqual( + name === "loro.js" ? rustReading : legacy["richtext-positions"], + ); + }); + + // Rust shows text that matches neither reading after this round trip. + test.skipIf(name !== "loro.js")( + "keeps a 0.2 snapshot's state through checkouts", + () => { + const doc = read((created) => { + created.configTextStyle({ bold: { expand: "after" } }); + created.import(fixture("richtext-positions.snapshot.blob")); + }); + // The checkout finds that the replay differs from the state, so the text + // keeps the state (#1126's unreplayable fallback) and shows older + // versions approximately. + const [first] = doc.frontiers(); + doc.checkout([{ peer: first!.peer, counter: 3 }]); + doc.checkoutToLatest(); + expect(doc.getText("t").toDelta()).toEqual(legacy["richtext-positions"]); + }, ); - expect(fromSnapshot.getText("t").toDelta()).toEqual(legacy["richtext-positions"]); - const history = styled((doc) => doc.import(fromSnapshot.export({ mode: "update" }))); - expect(history.getText("t").toDelta()).toEqual(rustReading); - }); - test("resolves a 0.2 cursor by its Unicode position", () => { - expect(legacy["cursor-emoji"]).toBe(4); - const owner = read((doc) => doc.import(fixture("cursor-emoji.snapshot.blob"))); - const cursor = runtime.Cursor.decode(fixture("cursor-emoji.cursor.blob")); - expect(owner.getCursorPos(cursor)?.offset).toBe(2); - const fresh = owner.getText("c").getCursor(4, 1)!; - expect(owner.getCursorPos(fresh)?.offset).toBe(4); - }); -}); + test("resolves a 0.2 cursor by its Unicode position", () => { + expect(legacy["cursor-emoji"]).toBe(4); + const owner = read((doc) => doc.import(fixture("cursor-emoji.snapshot.blob"))); + const cursor = runtime.Cursor.decode(fixture("cursor-emoji.cursor.blob")); + expect(owner.getCursorPos(cursor)?.offset).toBe(2); + const fresh = owner.getText("c").getCursor(4, 1)!; + expect(owner.getCursorPos(fresh)?.offset).toBe(4); + }); + }, +); From 4140169f068ddbd66bf3f3384fc3d053338a1dce Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 15:07:35 +0800 Subject: [PATCH 15/25] docs(loro.js): describe 0.2 snapshots after checkout, fork, and shallow export With #1126 on main, a checkout keeps a 0.2 snapshot's state while a fork or shallow export replays its history (Rust's reading); the test pins the fork difference from Rust. Co-Authored-By: Claude Opus 5.5 (1M context) --- loro-js/README.md | 6 ++- loro-js/tests/legacy-data.test.ts | 79 +++++++++++++++++-------------- 2 files changed, 48 insertions(+), 37 deletions(-) diff --git a/loro-js/README.md b/loro-js/README.md index b966de1f1..fc6ae0665 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -148,8 +148,10 @@ A document loaded from a 0.2 snapshot therefore holds a current state that its o history disagrees with. Its current state is safe to read right after the import, but do not keep using the document: exporting it as updates gives `béa9`, a replica loaded from updates holds other text and stays different after new edits -(`béQ9aZ` and `béQa9Z`), and checking out older versions and back can change the -content (Rust can then even show text that matches neither reading). +(`béQ9aZ` and `béQa9Z`), older versions it checks out can be approximate, and a +fork or a shallow-snapshot export can rebuild it from its history, which switches +it to Rust's reading (Rust can even show text that matches neither reading after a +checkout there and back). **Migrate every replica the same way.** Recommended paths: diff --git a/loro-js/tests/legacy-data.test.ts b/loro-js/tests/legacy-data.test.ts index e89f70a2b..9f17b0e01 100644 --- a/loro-js/tests/legacy-data.test.ts +++ b/loro-js/tests/legacy-data.test.ts @@ -25,38 +25,47 @@ const rust = loadRustReference(); const runtimes: [string, Runtime][] = [["loro.js", { LoroDoc, Cursor }]]; if (rust !== undefined) runtimes.push(["loro-crdt", rust]); -describe.each(runtimes)("documents written by loro.js 0.2, read by %s", (_, runtime) => { - const read = (load: (doc: LoroDoc) => void): LoroDoc => { - const doc = new runtime.LoroDoc(); - load(doc); - return doc; - }; - - test("orders concurrent inserts from history like Rust", () => { - expect(legacy["text-concurrent"]).toBe("bé9a"); - const fromUpdate = read((doc) => doc.import(fixture("text-concurrent.update.blob"))); - expect(fromUpdate.getText("t").toString()).toBe("béa9"); - const fromJson = read((doc) => - doc.importJsonUpdates(json("text-concurrent.json") as JsonSchema), - ); - expect(fromJson.getText("t").toString()).toBe("béa9"); - }); - - test("keeps a snapshot's current text but reads its history like Rust", () => { - const fromSnapshot = read((doc) => - doc.import(fixture("text-concurrent.snapshot.blob")), - ); - expect(fromSnapshot.getText("t").toString()).toBe("bé9a"); - const history = read((doc) => doc.import(fromSnapshot.export({ mode: "update" }))); - expect(history.getText("t").toString()).toBe("béa9"); - }); - - test("resolves a 0.2 cursor by its Unicode position", () => { - expect(legacy["cursor-emoji"]).toBe(4); - const owner = read((doc) => doc.import(fixture("cursor-emoji.snapshot.blob"))); - const cursor = runtime.Cursor.decode(fixture("cursor-emoji.cursor.blob")); - expect(owner.getCursorPos(cursor)?.offset).toBe(2); - const fresh = owner.getText("c").getCursor(4, 1)!; - expect(owner.getCursorPos(fresh)?.offset).toBe(4); - }); -}); +describe.each(runtimes)( + "documents written by loro.js 0.2, read by %s", + (name, runtime) => { + const read = (load: (doc: LoroDoc) => void): LoroDoc => { + const doc = new runtime.LoroDoc(); + load(doc); + return doc; + }; + + test("orders concurrent inserts from history like Rust", () => { + expect(legacy["text-concurrent"]).toBe("bé9a"); + const fromUpdate = read((doc) => + doc.import(fixture("text-concurrent.update.blob")), + ); + expect(fromUpdate.getText("t").toString()).toBe("béa9"); + const fromJson = read((doc) => + doc.importJsonUpdates(json("text-concurrent.json") as JsonSchema), + ); + expect(fromJson.getText("t").toString()).toBe("béa9"); + }); + + test("keeps a snapshot's current text but reads its history like Rust", () => { + const fromSnapshot = read((doc) => + doc.import(fixture("text-concurrent.snapshot.blob")), + ); + expect(fromSnapshot.getText("t").toString()).toBe("bé9a"); + const history = read((doc) => doc.import(fromSnapshot.export({ mode: "update" }))); + expect(history.getText("t").toString()).toBe("béa9"); + // State and history disagree here, so the runtimes' forks differ: loro.js + // replays the history, Rust copies the state. + const fork = fromSnapshot.fork(); + expect(fork.getText("t").toString()).toBe(name === "loro.js" ? "béa9" : "bé9a"); + }); + + test("resolves a 0.2 cursor by its Unicode position", () => { + expect(legacy["cursor-emoji"]).toBe(4); + const owner = read((doc) => doc.import(fixture("cursor-emoji.snapshot.blob"))); + const cursor = runtime.Cursor.decode(fixture("cursor-emoji.cursor.blob")); + expect(owner.getCursorPos(cursor)?.offset).toBe(2); + const fresh = owner.getText("c").getCursor(4, 1)!; + expect(owner.getCursorPos(fresh)?.offset).toBe(4); + }); + }, +); From 108b480f4f42e3d3fcfb407f7e746c0ad920545d Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 15:18:27 +0800 Subject: [PATCH 16/25] perf(loro.js): check a transition's sequence elements once per container #canTransitionRecords walked the whole sequence once per insert or mark op, so checking out across scattered edits was O(ops * elements): 668 ms at 8k characters with 2k edits on main, 5 ms now. Adds text-scattered-edits-checkout to bench:complexity. Co-Authored-By: Claude Opus 5.5 (1M context) --- loro-js/benchmarks/complexity.mjs | 21 +++++++++++ loro-js/src/runtime/document.ts | 54 +++++++++++++-------------- loro-js/src/runtime/sequence-index.ts | 2 +- 3 files changed, 49 insertions(+), 28 deletions(-) diff --git a/loro-js/benchmarks/complexity.mjs b/loro-js/benchmarks/complexity.mjs index 489164db0..a62447c66 100644 --- a/loro-js/benchmarks/complexity.mjs +++ b/loro-js/benchmarks/complexity.mjs @@ -527,6 +527,27 @@ for (const size of sizes) { return concurrentTarget.getText("text").length; }); + // Checking out across size / 4 scattered inserts: the transition checks each + // op's elements, once per container rather than one tree walk per op. + const scatteredDoc = new LoroDoc(); + scatteredDoc.setPeerId(4); + const scatteredText = scatteredDoc.getText("text"); + scatteredText.insert(0, "x".repeat(size)); + scatteredDoc.commit(); + const scatteredBase = scatteredDoc.frontiers(); + let scatteredSeed = 1; + for (let index = 0; index < size >>> 2; index += 1) { + scatteredSeed = (scatteredSeed * 1_103_515_245 + 12_345) >>> 0; + scatteredText.insert(scatteredSeed % scatteredText.length, "y"); + if (index % 16 === 0) scatteredDoc.commit(); + } + scatteredDoc.commit(); + measure("text-scattered-edits-checkout", size, () => { + scatteredDoc.checkout(scatteredBase); + scatteredDoc.checkoutToLatest(); + return scatteredText.length; + }); + const checkoutTextDoc = new LoroDoc(); checkoutTextDoc.setPeerId(3); checkoutTextDoc.getText("text").insert(0, "x".repeat(size)); diff --git a/loro-js/src/runtime/document.ts b/loro-js/src/runtime/document.ts index ffa90c082..47bc7da2e 100644 --- a/loro-js/src/runtime/document.ts +++ b/loro-js/src/runtime/document.ts @@ -4109,6 +4109,19 @@ export class LoroDoc = Record } >(); + // The inserted elements and style anchors each sequence must still hold, + // checked once per container below: one tree walk per operation made a + // checkout O(operations * elements). + const requiredRuns = new Map(); + const requireRun = ( + container: LoroList | LoroText, + start: CodecId, + length: number, + ): void => { + const runs = requiredRuns.get(container); + if (runs === undefined) requiredRuns.set(container, [{ start, length }]); + else runs.push({ start, length }); + }; for (const { change } of records) { const causalVersion = this.#causalVersionAt(change.dependencies); for (const operation of change.operations) { @@ -4174,40 +4187,24 @@ export class LoroDoc = Record = Record( } let count = countPhysicalIdsInRanges(node.left, targets); for (let offset = 0; offset < nodeLength(node); offset += 1) { - const id = nodeElement(node, offset).id; + const id = nodeId(node, offset); if (counterRangeRelation(targets, id.peer, id.counter, id.counter + 1) === 1) { count += 1; } From c95042a263304fd14b85d316665c2a8a9451b228 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 15:20:41 +0800 Subject: [PATCH 17/25] perf(loro.js): check a transition's sequence elements once per container #canTransitionRecords walked the whole sequence once per insert op, so checking out across scattered edits was O(ops * elements): 668 ms at 8k characters with 2k edits on main, 4 ms now. Adds text-scattered-edits-checkout to bench:complexity. Co-Authored-By: Claude Opus 5.5 (1M context) --- loro-js/benchmarks/complexity.mjs | 21 ++++++++++++ loro-js/src/runtime/document.ts | 46 +++++++++++++++------------ loro-js/src/runtime/sequence-index.ts | 2 +- 3 files changed, 48 insertions(+), 21 deletions(-) diff --git a/loro-js/benchmarks/complexity.mjs b/loro-js/benchmarks/complexity.mjs index 489164db0..a62447c66 100644 --- a/loro-js/benchmarks/complexity.mjs +++ b/loro-js/benchmarks/complexity.mjs @@ -527,6 +527,27 @@ for (const size of sizes) { return concurrentTarget.getText("text").length; }); + // Checking out across size / 4 scattered inserts: the transition checks each + // op's elements, once per container rather than one tree walk per op. + const scatteredDoc = new LoroDoc(); + scatteredDoc.setPeerId(4); + const scatteredText = scatteredDoc.getText("text"); + scatteredText.insert(0, "x".repeat(size)); + scatteredDoc.commit(); + const scatteredBase = scatteredDoc.frontiers(); + let scatteredSeed = 1; + for (let index = 0; index < size >>> 2; index += 1) { + scatteredSeed = (scatteredSeed * 1_103_515_245 + 12_345) >>> 0; + scatteredText.insert(scatteredSeed % scatteredText.length, "y"); + if (index % 16 === 0) scatteredDoc.commit(); + } + scatteredDoc.commit(); + measure("text-scattered-edits-checkout", size, () => { + scatteredDoc.checkout(scatteredBase); + scatteredDoc.checkoutToLatest(); + return scatteredText.length; + }); + const checkoutTextDoc = new LoroDoc(); checkoutTextDoc.setPeerId(3); checkoutTextDoc.getText("text").insert(0, "x".repeat(size)); diff --git a/loro-js/src/runtime/document.ts b/loro-js/src/runtime/document.ts index 8b4f093fb..247d4dd90 100644 --- a/loro-js/src/runtime/document.ts +++ b/loro-js/src/runtime/document.ts @@ -4095,6 +4095,19 @@ export class LoroDoc = Record } >(); + // The inserted elements each sequence must still hold, checked once per + // container below: one tree walk per operation made a checkout + // O(operations * elements). + const requiredRuns = new Map(); + const requireRun = ( + container: LoroList | LoroText, + start: CodecId, + length: number, + ): void => { + const runs = requiredRuns.get(container); + if (runs === undefined) requiredRuns.set(container, [{ start, length }]); + else runs.push({ start, length }); + }; for (const { change } of records) { const causalVersion = this.#causalVersionAt(change.dependencies); for (const operation of change.operations) { @@ -4177,31 +4190,21 @@ export class LoroDoc = Record = Record( } let count = countPhysicalIdsInRanges(node.left, targets); for (let offset = 0; offset < nodeLength(node); offset += 1) { - const id = nodeElement(node, offset).id; + const id = nodeId(node, offset); if (counterRangeRelation(targets, id.peer, id.counter, id.counter + 1) === 1) { count += 1; } From 981c12e666cfd0196ca52d357ce6a928195efa37 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 15:31:12 +0800 Subject: [PATCH 18/25] docs(loro.js): record the transition check measurements Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 17a1ffcf1..38450b831 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -253,6 +253,14 @@ JavaScript constant factor. that throws restores its previous version and state (`#transitionTo`, which also prepares inside its `try`), and `diff` restores the current state with a full rebuild when it or its move back throws. +- Before a transition, `#canTransitionRecords` checks that each sequence still + holds the elements that the crossed insert operations name. It collects the + runs per container and calls `containsIdRuns` once per container, reading IDs + without building element views: O(elements + runs log runs). One call per + operation made a checkout O(operations × elements): across 2k scattered + inserts in an 8k text it took 698 ms on `main` and takes 4.2 ms now + (`text-scattered-edits-checkout`, 13 → 698 ms from 1k to 8k before, 1.0 → + 4.2 ms now; September 29, Node 22, 1-minute load 6–9). - First checkout after importing a 262,144-operation single-peer Text snapshot takes about 57 ms (medians of 5 alternating runs on a loaded Apple M5 Pro), versus about 148 ms for the earlier whole-document replay; 65,536 operations From 6bb9f476860a5214d514384ec5d3be2218046419 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 15:31:32 +0800 Subject: [PATCH 19/25] docs(loro.js): record anchor-model constant costs and rich-text checkout Co-Authored-By: Claude Opus 5.5 (1M context) --- context/loro-js-performance.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/context/loro-js-performance.md b/context/loro-js-performance.md index 39ffa5e18..830ba979e 100644 --- a/context/loro-js-performance.md +++ b/context/loro-js-performance.md @@ -445,6 +445,21 @@ build takes 7.1/19/362 ms to retreat 1k/4k/16k such marks, and building the 16k history takes 402 s in loro.js and 542 s in Rust. Every other `bench:complexity` entry stays flat from 1k to 8k. +The zero-width counters are four more fields in every treap node, maintained +on every update even for text without anchors, so plain Text edits pay a +constant cost: 8,000 subscribed middle inserts take 15.4–15.8 ms against +14.3–14.4 ms without them, and `text-subscribed-batch`, `history-commit`, and +`history-update-batch-import` are 12–20% slower at 8k (September 29, Node 22, +alternating runs at 1-minute load 6–8). No entry grows with size because of +them. Moving them into a sidecar that only anchored sequences allocate, like +the line-break totals, would remove that cost. + +Reading a Rust-written styled Text from a snapshot (16k characters, 200 marks, +2k later edits) and checking out a middle version takes 24.7–25.0 ms the +first time, 4.1–4.3 ms back to the latest, and 11.8–12.2 ms for a fork; `main` +takes 35.9–37.2, 3.1–3.3, and 29.8–30.5 ms but shows wrong text, because it +marks the text unreplayable and toggles its snapshot state. + These costs were reviewed and accepted (loro-dev/loro#1137). The review measured, on Node 26 at 1-minute load 25–40, typing 1,000 characters inside a 64k bold range at 2.9–4.4 ms (1.9 ms on `main`) and the 8k repeated-mark tail retreat and From eabcddfd87c4c712a532e70512ce2b221aa78d9b Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 17:33:51 +0800 Subject: [PATCH 20/25] docs(loro.js): correct the 0.2 cursor and migration notes Cursors move for side 1 and end-of-text, not astral text; reading the final state with 1.0 needs the snapshot to be the final version; a shallow export switches a 0.2-snapshot document to Rust's reading. Co-Authored-By: Claude Opus 5.5 --- .changeset/loro-js-text-interop.md | 4 ++-- context/loro-js-rust-differential.md | 4 ++-- loro-js/README.md | 28 ++++++++++++++--------- loro-js/scripts/write-legacy-fixtures.mjs | 2 +- loro-js/tests/legacy-data.test.ts | 2 +- 5 files changed, 23 insertions(+), 17 deletions(-) diff --git a/.changeset/loro-js-text-interop.md b/.changeset/loro-js-text-interop.md index 0f17142ab..521f1c6c7 100644 --- a/.changeset/loro-js-text-interop.md +++ b/.changeset/loro-js-text-interop.md @@ -5,9 +5,9 @@ Text now follows Rust (`loro-crdt`) everywhere, including when reading documents written by earlier loro.js versions. This breaks documents edited only with loro.js that have concurrent text edits or stored cursors: read "Upgrading from 0.2" in the README before upgrading. - **Breaking:** concurrent text inserts are ordered like Rust (the fix is shared with loro-dev/loro#1139). Earlier versions could place an insert after a sibling's subtree behind later, unrelated concurrent text, so a history with such inserts can now read differently (for example `béa9` instead of `bé9a`). Earlier loro.js replicas that merged such edits in different orders, and documents shared with `loro-crdt` peers, had already diverged; they now agree with Rust. -- **Breaking:** text cursors follow Rust. An encoded cursor stores the Unicode position of its target, not the UTF-16 position, so a cursor encoded by an earlier version can resolve to another offset. A cursor at the end or in empty text has no target, `getCursorPos` returns the target's own offset for every side, and a deleted target reports the length of the text before it with side `-1`. Get new cursors after upgrading. +- **Breaking:** text cursors follow Rust: a cursor reports its target's own offset, so a cursor encoded by an earlier version with side 1, or at the end of the text, resolves one character earlier (the end cursor of `abc` at 2 instead of 3). Other cursors resolve as before. The encoded origin is now the Unicode position, as in Rust. A cursor at the end or in empty text has no target, `getCursorPos` returns the target's own offset for every side, and a deleted target reports the length of the text before it with side `-1`. Get new cursors after upgrading. - An imported delete is applied by its position, as Rust does, instead of by its recorded `start_id` (Rust's WASM build up to 1.16.3 can record a wrong one around astral characters, loro-dev/loro#1149). - `checkout`, `diff`, and `revertTo` no longer restore a character twice when two concurrent deletes removed it. - Consecutive inserts in one transaction are merged into one op, and a delete that spans several ID runs writes the last run first, as Rust does. -Upgrading: updates and JSON updates written by an earlier version are read like Rust. A snapshot written by an earlier version keeps its current text, but its history reads like Rust, so a document loaded from it must not stay in use (exported updates, replicas loaded from updates, and checkouts can show other content). Migrate every replica the same way: read the final state (with the old version, or right after importing an old snapshot) and build a new document from it, or import the old updates everywhere and accept Rust's reading. See "Upgrading from 0.2" in the README. +Upgrading: updates and JSON updates written by an earlier version are read like Rust. A snapshot written by an earlier version keeps its current text, but its history reads like Rust, so a document loaded from it must not stay in use (exported updates, replicas loaded from updates, and checkouts can show other content). Exporting a shallow snapshot from such a document switches the document itself to Rust's reading. Migrate every replica the same way: read the final state with the old version and build a new document from it (reading it with 1.0 right after importing an old snapshot only works when that snapshot is the final version, with no updates stored after it), or import the old updates everywhere and accept Rust's reading. See "Upgrading from 0.2" in the README. diff --git a/context/loro-js-rust-differential.md b/context/loro-js-rust-differential.md index 1f954c5ba..c417d8c68 100644 --- a/context/loro-js-rust-differential.md +++ b/context/loro-js-rust-differential.md @@ -86,8 +86,8 @@ can check out a document imported from a snapshot (loro-dev/loro#1126). ## Data written by loro.js 0.2 loro.js 0.2 ordered some concurrent text inserts differently from Rust (a -concurrent insert after a sibling's subtree, fixed by loro-dev/loro#1139) and encoded -text cursors by UTF-16 position. Decision (2026-09-28): later versions read all +concurrent insert after a sibling's subtree, fixed by loro-dev/loro#1139) and resolved +side-1 and end-of-text cursors one character after their target. Decision (2026-09-28): later versions read all data with Rust's semantics and add no version marker. The same bytes must mean the same thing in both runtimes, documents shared with `loro-crdt` peers had already diverged, and a marker would need a format change that Rust does not diff --git a/loro-js/README.md b/loro-js/README.md index fc6ae0665..65b80487d 100644 --- a/loro-js/README.md +++ b/loro-js/README.md @@ -130,9 +130,11 @@ can read differently after the upgrade: - **Concurrent text inserts.** 0.2 could order an insert after a sibling's subtree differently from Rust. 1.0 orders it like Rust, so such a history can give other text (`bé9a` in 0.2, `béa9` in 1.0 and Rust). -- **Cursors.** 0.2 encoded the UTF-16 position of a cursor's target; Rust and 1.0 use - the Unicode position. A cursor encoded by 0.2 can resolve to another offset (on - `ab😀`, 4 in 0.2 and 2 in 1.0). Get new cursors from the upgraded document. +- **Cursors.** 1.0 resolves cursors like Rust: a cursor reports its target's own + offset. A cursor encoded by 0.2 with side 1, or at the end of the text, therefore + resolves one character earlier, whatever the characters are (on `abc`, the end + cursor is at 3 in 0.2 and 2 in 1.0; a side-1 cursor on `b` at 2 and 1). Other + cursors resolve as before. Get new cursors from the upgraded document. Documents also edited by `loro-crdt` peers already disagreed with Rust in these cases; 1.0 makes them agree. @@ -149,17 +151,21 @@ history disagrees with. Its current state is safe to read right after the import but do not keep using the document: exporting it as updates gives `béa9`, a replica loaded from updates holds other text and stays different after new edits (`béQ9aZ` and `béQa9Z`), older versions it checks out can be approximate, and a -fork or a shallow-snapshot export can rebuild it from its history, which switches -it to Rust's reading (Rust can even show text that matches neither reading after a -checkout there and back). +fork can rebuild it from its history, which switches it to Rust's reading. +Exporting a shallow snapshot from it (`export({ mode: "shallow-snapshot" })`) does +the same to the document itself: its current state becomes Rust's reading (`bé9a` +turns into `béa9`). Rust can even show text that matches neither reading after a +checkout there and back. **Migrate every replica the same way.** Recommended paths: -1. **Keep the content, drop the history (safest).** Read the final state, either - with 0.2 after checking out the latest version, or with 1.0 right after importing - a 0.2 snapshot: `doc.toJSON()`, or `text.toDelta()` for rich text. Build one new - document from it in 1.0 and share that document with every replica. Do not - import old updates or snapshots into it. +1. **Keep the content, drop the history (safest).** Read the final state with 0.2, + after checking out the latest version: `doc.toJSON()`, or `text.toDelta()` for + rich text. Build one new document from it in 1.0 and share that document with + every replica. Do not import old updates or snapshots into it. Reading the state + with 1.0 right after importing a 0.2 snapshot gives the same result only when + that snapshot is the final version: if updates were stored after it, 1.0 (like + Rust) reads those updates the Rust way. 2. **Keep the history, accept Rust's reading.** Import the 0.2 updates (not snapshots) on every replica, and check the content, because it can differ from what 0.2 showed. diff --git a/loro-js/scripts/write-legacy-fixtures.mjs b/loro-js/scripts/write-legacy-fixtures.mjs index 6889f676d..1b807cdde 100644 --- a/loro-js/scripts/write-legacy-fixtures.mjs +++ b/loro-js/scripts/write-legacy-fixtures.mjs @@ -50,7 +50,7 @@ function sync(from, to) { expected["text-concurrent"] = peer2.getText("t").toString(); } -// A cursor on the end of "ab😀": 0.2 encoded its UTF-16 position. +// The end cursor of "ab😀": 0.2 resolved it after its target, Rust and 1.0 at it. { const owner = doc(1); const text = owner.getText("c"); diff --git a/loro-js/tests/legacy-data.test.ts b/loro-js/tests/legacy-data.test.ts index 9f17b0e01..dccc17c97 100644 --- a/loro-js/tests/legacy-data.test.ts +++ b/loro-js/tests/legacy-data.test.ts @@ -59,7 +59,7 @@ describe.each(runtimes)( expect(fork.getText("t").toString()).toBe(name === "loro.js" ? "béa9" : "bé9a"); }); - test("resolves a 0.2 cursor by its Unicode position", () => { + test("resolves a 0.2 end cursor one character earlier, like Rust", () => { expect(legacy["cursor-emoji"]).toBe(4); const owner = read((doc) => doc.import(fixture("cursor-emoji.snapshot.blob"))); const cursor = runtime.Cursor.decode(fixture("cursor-emoji.cursor.blob")); From 1856d8cf2743cc691dead6f5009bb7e533245896 Mon Sep 17 00:00:00 2001 From: Zixuan Chen Date: Tue, 29 Sep 2026 17:47:49 +0800 Subject: [PATCH 21/25] fix(loro.js): don't re-insert hidden style anchors when a checkout replays After a snapshot import and a checkout back past a mark, attach, checkout, and diff threw `duplicate sequence id` when the forward records could not move incrementally (a movable-list set, move, or delete): the replay inserted the mark's anchors again although the state already held them. `hasMaterializedSequenceInsertions` now counts anchor ops, and `diff` rebuilds its `from` state in that case too. The differential harness now edits a movable list, checks out three versions of every imported full snapshot, diffs, and attaches; seed 19 fails without this fix. It also derived its second checkout roll from bits that repeated the first. Details: context/loro-js-richtext-anchors.md and context/loro-js-rust-differential.md. Co-Authored-By: Claude Opus 5.5 --- context/loro-js-richtext-anchors.md | 10 +- context/loro-js-rust-differential.md | 31 +++- loro-js/src/runtime/document.ts | 11 +- loro-js/tests/richtext-anchors.test.ts | 54 +++++++ .../tests/support/richtext-differential.ts | 139 ++++++++++++++---- 5 files changed, 203 insertions(+), 42 deletions(-) diff --git a/context/loro-js-richtext-anchors.md b/context/loro-js-richtext-anchors.md index 64e326aea..bdcb4cd07 100644 --- a/context/loro-js-richtext-anchors.md +++ b/context/loro-js-richtext-anchors.md @@ -1,6 +1,6 @@ # loro-js Rich-Text Anchors -Verified against code 2026-09-28. +Verified against code 2026-09-29. loro.js keeps rich-text style anchors in the text sequence, as Rust does, so a Text op position means the same thing in both runtimes. This article covers the @@ -103,6 +103,14 @@ Remote ops: event baseline is started when an anchor op is collected; without it, a version with a start anchor but no end anchor was diffed against an empty text and reported the whole text as inserted. +- Anchors are sequence elements, so a checkout that leaves them hidden in the + state must not insert them again. When a transition cannot run incrementally + (a movable-list set, for example) and would apply the forward records + directly, `hasMaterializedSequenceInsertions` in `document.ts` counts + `text-mark`/`text-mark-end` ops with the inserts and sends the checkout to a + rebuild; `diff` rebuilds its `from` state in the same case. Missing that + threw `duplicate sequence id` on `attach` after a snapshot import and two + checkouts. - Undoing a mark gives back, wherever it changed the value, the value from just before its op (`LoroText._undoStyle`). That matches Rust for local histories; loro.js undo still does not transform against remote edits the way Rust's diff --git a/context/loro-js-rust-differential.md b/context/loro-js-rust-differential.md index 96c651d7d..d9c1e8c7b 100644 --- a/context/loro-js-rust-differential.md +++ b/context/loro-js-rust-differential.md @@ -1,6 +1,6 @@ # loro-js Differential Tests Against Rust -Verified against code 2026-09-28. +Verified against code 2026-09-29. `loro-js/tests/richtext-differential.test.ts` runs random multi-peer Text scenarios in loro.js and in Rust (`loro-crdt`, nodejs WASM build) side by side @@ -38,6 +38,16 @@ per runtime with the same peer ID. Actions: split a change or an op (a mark with only its start anchor, for example), so transitions also run between historical and mid-change versions; - full and shallow snapshot export from either runtime, imported into both; + a full snapshot's copies then check out three versions in a row (picked like + `checkout`'s), diff the last two, and attach, so an imported state moves back + and forward over elements it already holds; +- movable-list inserts and sets, written by Rust and imported by loro.js. The + list is not compared; its set ops keep checkouts off the incremental path, so + they replay the text's records (the path of the `duplicate sequence id` + regression that seed 19 catches). Moves and deletes are left out: loro.js and + Rust disagree on such lists (loro-dev/loro#1132), and Rust panics + (`movable_list_state.rs` "consistency check failed") when it checks out a + snapshot loro.js wrote with the differing state; - cursor creation, then cursor resolution after every later step; - `revertTo` and undo/redo. @@ -80,13 +90,18 @@ tolerates them, so it also runs against older reference builds: can diverge from its own full-history import when they are wrong. The harness compares loro.js with the source document instead. -Snapshot-import checkout (`snapshotCheckout`) is off by default. Since -loro-dev/loro#1126, checkout after importing a full snapshot matches Rust, but -checkout into the retained range of a shallow snapshot that loro.js exported -still diverges: about 20% of 120-step seeds, plain and rich text alike -(September 29). The replay of such a snapshot differs from its state, so the -text becomes `unreplayable` (see [loro-js-performance.md](loro-js-performance.md)); -the export gap itself is not fixed yet. +Checkout after a full snapshot import (`snapshotCheckout`) is on. Checkout into +the retained range of a shallow snapshot (`shallowCheckout`) is off: loro.js +still diverges there when it wrote the snapshot. `#calculateShallowStart` in +`loro-js/src/runtime/document.ts` uses the requested frontiers as the root, +while Rust moves the root back to the nearest critical version, and retained +ops concurrent with a non-critical root do not replay from its state. The +harness visits only a few versions per export and fails 6 of 40 rich and 5 of +20 plain 120-step seeds (September 29). An exhaustive check of every retained +version from every commit root (second review of loro-dev/loro#1136/#1137) +finds most multi-peer histories affected: over half of the non-critical-root +versions diverge, and none with a critical root. The fix is to choose the root +like Rust; it is not done yet. ## Data written by loro.js 0.2 diff --git a/loro-js/src/runtime/document.ts b/loro-js/src/runtime/document.ts index 47bc7da2e..c30fe62e1 100644 --- a/loro-js/src/runtime/document.ts +++ b/loro-js/src/runtime/document.ts @@ -1168,7 +1168,13 @@ export class LoroDoc = Record