fix(loro.js): reject invalid MovableList move/set refs (port Rust #1125)

.changeset/loro-js-movable-list-invalid-ops.md

@@ -0,0 +1,5 @@
+---
+"loro.js": patch
+---
+
+Importing a MovableList move or set that names an unknown element, an element of another list, an element outside the op's causal history, or (on a shallow document) an element deleted before the shallow root now fails and rolls back the whole import, as in Rust (loro-dev/loro#1125). A move or set of a deleted element is applied like a concurrent one on every import path.

context/loro-js-movable-list.md

@@ -8,8 +8,8 @@ reproduce, the loro.js model that implements them, and how the differential
 suite checks the two against each other. Complexity rules still come from
 [loro-js-performance.md](loro-js-performance.md).
 
-Status: the model below and atomic imports are implemented. Import validation
-("Validation and import atomicity") lands in a follow-up change.
+Status: the model below, atomic imports, and import validation are
+implemented.
 
 ## Rust semantics (the contract)
 
@@ -61,9 +61,9 @@ A MovableList is two CRDTs glued together
   section 8) lists every alive position in order, dead ones included, plus
   element IDs and last-set IDs when they differ from the position/element ID.
 
-Invalid `Move`/`Set` handling (Rust #1125, open when this was written) is in
-[movable-list-op-validation.md](movable-list-op-validation.md) once merged; see
-the "Validation" section below for the loro.js side.
+Invalid `Move`/`Set` handling (Rust loro-dev/loro#1125) is in
+[movable-list-op-validation.md](movable-list-op-validation.md); see the
+"Validation" section below for the loro.js side.
 
 ## Why the old loro.js model diverged
 
@@ -211,17 +211,46 @@ Rust's rule for `from_move` inserts that transforms leave behind
 
 ## Validation and import atomicity
 
-Atomicity is implemented ([import-batch-atomicity.md](import-batch-atomicity.md),
-"loro.js"). Validation is planned. It follows Rust #1125. A `Move`/`Set` whose element is unknown,
-lives in another container, is outside the op's causal history, or (on a
-shallow doc) was deleted before the root is rejected with an error. So is a
-`from`/`to` outside the op-index range of its causal view. A `Move`/`Set` of a
-deleted element is applied with the concurrent-op semantics above.
-`import`/`importBatch` stage the document state (history indexes, containers,
-pending changes, version, subscribers' events, undo stacks) and restore it
-when any blob fails, so a rejected import is invisible
-([import-batch-atomicity.md](import-batch-atomicity.md) states the Rust
-contract).
+Validation follows Rust loro-dev/loro#1125 (`validate_movable_list_elem_refs_since`;
+[movable-list-op-validation.md](movable-list-op-validation.md)).
+`#validateMovableListRefs` in `document.ts` runs on every imported change after
+integration: FastUpdates, non-initializing snapshots, and `importBatch`
+(except the snapshot that seeds an empty document, which Rust does not
+validate op by op either). A `Move`/`Set` whose element is unknown, is not a
+`movable-list-insert` into the same list, is outside the op's causal history,
+or (on a shallow doc) is older than the root but missing from the root state is
+rejected with an error. `MovableListState.applyMove` rejects a `from`/`to`
+outside the op-index range of the op's causal view before mutating anything. A
+`Move`/`Set` of a deleted element is valid and is applied with the
+concurrent-op semantics above, the same on every import path.
+
+The element lookup maps `(peer, lamport)` to an op ID by binary search over
+the peer's indexed changes (lamports grow with counters). While snapshot
+history is deferred only the overlay is indexed; a miss there means the element
+comes from the snapshot base, and it is accepted without decoding history when
+the list's hydrated state holds it and the op has seen the whole base.
+Otherwise history is materialized once. The shallow-root element set is built
+once per root store and list.
+
+A rejection rolls back the whole import or batch. `import`/`importBatch` stage
+the document state (history indexes, containers, pending changes, version,
+subscribers' events, undo stacks) and restore it when any blob fails, so a
+rejected import is invisible ([import-batch-atomicity.md](import-batch-atomicity.md),
+"loro.js", states both contracts). Rust reports an out-of-range index as
+"movable list diff retains N items but state only has M"; loro.js says "out of
+range". `loro-js/tests/movable-list-invalid-ops.test.ts` ports
+`crates/loro/tests/movable_list_invalid_ops.rs`; run against a WASM build of
+`main` with #1125 merged, all of it but the loro.js-only undecodable-snapshot
+test passes, error text aside.
+
+Rust rules loro.js does not copy (edge cases, documented rather than ported):
+- Rust rejects any List/MovableList/Text op position at or past 1,073,741,822
+  when it decodes ops (`InnerListOp::check_positions`), on every import path.
+  loro.js rejects such a move only when it applies it, so a detached import
+  records it until `attach` fails.
+- Rust bounds-checks the composed diff of an import, so an out-of-bounds move
+  that a later op of the same import cancels is accepted. loro.js checks each
+  op when it applies it and rejects the import.
 
 ## Differential suite
 

context/loro-js-performance.md

@@ -533,7 +533,10 @@ The remaining differences are representation and JavaScript constant factors:
   concurrently with the snapshot, or names a MovableList element it lacks,
   rebuilds only that container from its own history (`#prepareSnapshotImport`);
   so does the first version transition that touches a hydrated MovableList.
-  Later imports and transitions are incremental again.
+  Later imports and transitions are incremental again. Import validation of
+  MovableList moves and sets costs O(log changes) per op and keeps deferred
+  snapshot history deferred when the target element is in the hydrated state
+  ([loro-js-movable-list.md](loro-js-movable-list.md), "Validation").
 - The million-operation C1.1 concurrent-text trace still exposes a large
   constant-factor and retained-memory gap. Its local edit phase is about 4x the
   WASM adapter, and parsing the 6.5 MB snapshot takes 3.62 seconds versus 43 ms.

loro-js/src/runtime/document.ts

@@ -720,6 +720,7 @@ export class LoroDoc<T extends Record<string, Container> = Record<string, Contai
       }
       this.#assertImportsNotOutdated(imported);
       integration = this.#integrateHistory(imported);
+      this.#validateMovableListRefs(integration.added);
       const { added } = integration;
       if (added.length === 0 && !this.#detached) {
         this.#materializePendingRoots(imported, integration.pending);
@@ -914,6 +915,9 @@ export class LoroDoc<T extends Record<string, Container> = Record<string, Contai
         installStagedShallowRoot();
         if (!initializeFromSnapshot) this.#assertImportsNotOutdated(imported);
         integration = this.#integrateHistory(imported);
+        // Like Rust, a snapshot that initializes an empty document is not
+        // validated op by op.
+        if (!initializeFromSnapshot) this.#validateMovableListRefs(integration.added);
         if (!this.#detached && initializeFromSnapshot) {
           if (this.#hasEventSubscribers()) {
             beforeValues = this.#captureEventValues(integration.added);
@@ -1026,6 +1030,12 @@ export class LoroDoc<T extends Record<string, Container> = Record<string, Contai
       if (decoded !== snapshotSeed) this.#assertImportsNotOutdated(decoded.records);
     }
     const integration = this.#integrateHistory(ordered.flatMap(({ records }) => records));
+    const seedChanges = new Set(
+      snapshotSeed?.records.map(({ change }) => changeKey(change.id)) ?? [],
+    );
+    this.#validateMovableListRefs(
+      integration.added.filter(({ change }) => !seedChanges.has(changeKey(change.id))),
+    );
     let beforeValues = new Map<string, unknown>();
     let preparedDiffs = new Map<string, Diff>();
     if (integration.added.length > 0 && !this.#detached) {
@@ -8005,6 +8015,155 @@ export class LoroDoc<T extends Record<string, Container> = Record<string, Contai
     return version;
   }
 
+  /**
+   * Rust #1125 (`OpLog::validate_movable_list_elem_refs_since`): a MovableList
+   * move or set must name an element inserted into the same list within the
+   * op's causal history. On a shallow document, an element from before the
+   * root must be in the root state; no op after the root can see one that was
+   * deleted before it. An index out of range fails while the op is applied.
+   * Either way the whole import is rolled back (`#atomically`). A move or set
+   * of a deleted element is valid: it has the meaning of a concurrent one. See
+   * context/loro-js-movable-list.md.
+   */
+  #validateMovableListRefs(records: readonly HistoryRecord[]): void {
+    for (const { change } of records) {
+      let causalVersion: CausalVersion | undefined;
+      for (const operation of change.operations) {
+        const content = operation.content;
+        if (content.type !== "movable-list-move" && content.type !== "movable-list-set") {
+          continue;
+        }
+        causalVersion ??= this.#causalVersionAt(change.dependencies);
+        const opId = { peer: change.id.peer, counter: operation.counter };
+        const lamport = change.lamport + (operation.counter - change.id.counter);
+        if (
+          !this.#isMovableElementVisible(
+            operation.container,
+            opId,
+            lamport,
+            content.elementId,
+            causalVersion,
+          )
+        ) {
+          throw new Error(
+            `Movable list op ${opId.counter}@${opId.peer} targets element ` +
+              `L${content.elementId.lamport}@${content.elementId.peer}, which is not in the ` +
+              "list's causal history",
+          );
+        }
+      }
+    }
+  }
+
+  #isMovableElementVisible(
+    container: CodecContainerId,
+    opId: CodecId,
+    opLamport: number,
+    elementId: { readonly peer: bigint; readonly lamport: number },
+    causalVersion: CausalVersion,
+  ): boolean {
+    if (elementId.lamport >= opLamport) return false;
+    let target = this.#idAtLamport(elementId.peer, elementId.lamport);
+    const deferred = this.#deferredSnapshotHistory;
+    if (target === undefined && deferred !== undefined) {
+      // Only the overlay is indexed, so the element comes from the snapshot
+      // base. Snapshot state only holds elements inserted into that list, all
+      // of them seen by an op that has seen the whole base.
+      if (
+        this.#snapshotStateHasElement(container, elementId) &&
+        deferred.endVersion
+          ._codecEntriesUnsorted()
+          .every(({ peer, counter }) => counter <= (causalVersion.get(peer) ?? 0))
+      ) {
+        return true;
+      }
+      this.#materializeDeferredHistory();
+      target = this.#idAtLamport(elementId.peer, elementId.lamport);
+    }
+    if (target === undefined) return this.#shallowRootHasElement(container, elementId);
+    const causal =
+      (target.peer === opId.peer && target.counter < opId.counter) ||
+      target.counter < (causalVersion.get(target.peer) ?? 0);
+    if (!causal) return false;
+    const operation = this.#operationAt(target);
+    if (operation === undefined) return this.#shallowRootHasElement(container, elementId);
+    return (
+      operation.content.type === "movable-list-insert" &&
+      containerIdsEqual(operation.container, container)
+    );
+  }
+
+  /**
+   * The op ID of `peer`'s op with `lamport`, or undefined when indexed history
+   * lacks it. Deferred snapshot history is not indexed.
+   */
+  #idAtLamport(peer: bigint, lamport: number): CodecId | undefined {
+    const records = this.#historyByPeer.get(peer);
+    if (records === undefined) return undefined;
+    let low = 0;
+    let high = records.length;
+    while (low < high) {
+      const middle = (low + high) >>> 1;
+      if (records[middle]!.change.lamport <= lamport) low = middle + 1;
+      else high = middle;
+    }
+    const record = records[low - 1];
+    if (record === undefined) return undefined;
+    const offset = lamport - record.change.lamport;
+    return offset < changeLength(record.change)
+      ? { peer, counter: record.change.id.counter + offset }
+      : undefined;
+  }
+
+  #snapshotStateHasElement(
+    id: CodecContainerId,
+    elementId: { readonly peer: bigint; readonly lamport: number },
+  ): boolean {
+    const container = this.#containers.get(this.#containerKey(id));
+    if (!(container instanceof LoroMovableList)) return false;
+    this._ensureContainerHydrated(container);
+    return container._state.element(elementId.peer, elementId.lamport) !== undefined;
+  }
+
+  /** Rust's `shallow_root_has_movable_list_elem`. */
+  #shallowRootHasElement(
+    container: CodecContainerId,
+    elementId: { readonly peer: bigint; readonly lamport: number },
+  ): boolean {
+    const store = this.#shallowRootStore;
+    if (store?.kind !== "sstable") return false;
+    const key = formatContainerId(container);
+    let elements = shallowRootElements.get(store)?.get(key);
+    if (elements === undefined) {
+      elements = new Set();
+      const entry = store.containers.find(({ id }) => formatContainerId(id) === key);
+      const state = entry?.wrapper.state;
+      if (state?.kind === CodecContainerType.MovableList) {
+        let listIndex = Number(state.items[0]?.invisibleListItems ?? 0n);
+        let elementIndex = 0;
+        for (let index = 1; index < state.items.length; index += 1) {
+          const item = state.items[index]!;
+          const listItem = state.listItemIds[listIndex]!;
+          const id = item.positionIdEqualsElementId
+            ? {
+                peerIndex: listItem.peerIndex,
+                lamport: listItem.counter + listItem.lamportSub,
+              }
+            : state.elementIds[elementIndex++]!;
+          elements.add(`${state.peers[Number(id.peerIndex)]!}:${id.lamport}`);
+          listIndex += 1 + Number(item.invisibleListItems);
+        }
+      }
+      let byContainer = shallowRootElements.get(store);
+      if (byContainer === undefined) {
+        byContainer = new Map();
+        shallowRootElements.set(store, byContainer);
+      }
+      byContainer.set(key, elements);
+    }
+    return elements.has(`${elementId.peer}:${elementId.lamport}`);
+  }
+
   #operationAt(id: CodecId): DecodedOperation | undefined {
     const record = this.#recordContaining(id);
     if (record === undefined) return undefined;
@@ -9055,6 +9214,9 @@ function transformListDelta(
   return output;
 }
 
+/** Element IDs of each MovableList in a shallow root store, built on first use. */
+const shallowRootElements = new WeakMap<object, Map<string, Set<string>>>();
+
 function hasMaterializedSequenceInsertions(
   records: readonly HistoryRecord[],
   containers: ReadonlyMap<string, LoroContainer>,