Text deletes written by loro-crdt (WASM) ≤ 1.16.3 can record a wrong start_id around astral characters #1149

Summary

In the WASM build (loro-crdt up to 1.16.3), a text delete that spans text from separate inserts can record a start_id that names the wrong characters when an astral character (emoji, 𝒳, …) sits before a run boundary inside the deleted range. #1135 fixes this for new ops (get_text_entity_ranges compared ID contiguity using the UTF-16 length instead of the Unicode length). Deletes already written into existing histories keep the wrong start_id, and several read paths trust it:

• A full import applies the delete by position, so its text is right.
• A document imported from a shallow snapshot whose root is before the delete gets different text: the deleted text is only known as placeholders there, and the placeholders take their real IDs from the delete's start_id (CrdtRope::delete, real_id).
• A cursor on a character that was deleted this way no longer resolves in the full document (getCursorPos returns undefined), because no delete op names it.

Native Rust builds do not write such ops (their event length is the Unicode length), but they read existing ones the same way.

Repro (published loro-crdt@1.16.3)

import { LoroDoc, Cursor } from "loro-crdt";

const writer = new LoroDoc();
writer.setPeerId(7);
const text = writer.getText("t");
text.insert(0, "a😀b");
writer.commit();
text.insert(3, "x"); // "a😀xb"
writer.commit();
const cursorX = text.getCursor(3); // on "x"
const shallowRoot = writer.oplogFrontiers();
text.delete(0, 4); // deletes "a😀x"
writer.commit();
text.insert(1, "!");
writer.commit();

const full = new LoroDoc();
full.import(writer.export({ mode: "update" }));
const shallow = new LoroDoc();
shallow.import(full.export({ mode: "shallow-snapshot", frontiers: shallowRoot }));
const pos = (doc) => doc.getCursorPos(Cursor.decode(cursorX.encode()))?.offset;

console.log(writer.exportJsonUpdates().changes.flatMap((c) => c.ops).map((op) => op.content).filter((c) => c.type === "delete"));
console.log(full.getText("t").toString(), shallow.getText("t").toString(), pos(full), pos(shallow));

Output:

[ { type: 'delete', pos: 0, len: 3, start_id: '0@0' } ]   // names a😀b (IDs 0..=2), but a😀x (IDs 0, 1, 3) was deleted
b! x! undefined 0

Expected (and what the same edits without the emoji give, e.g. "aeb"): b! b! 0 0. A build with #1135 writes [{pos: 2, len: 1, start_id: 3@…}, {pos: 0, len: 2, start_id: 0@…}] for these edits, and full and shallow imports agree; with #1135, importing the bytes written by 1.16.3 still gives b! x! undefined 0.

Scope

Affected: histories edited through loro-crdt (WASM) where one delete removed text from more than one insert run and an astral character ended a run inside the deleted range, and the next run's first ID happened to equal that run's start ID plus its UTF-16 length (as in the repro: a later insert by the same peer).

Options

• Accept and document: full imports stay right; only shallow snapshots rooted before such a delete, and cursors on the deleted characters, are affected.
• Make the readers independent of start_id where the position is available: e.g. derive the deleted IDs from the position during replay (as the full import does) instead of labeling placeholders with start_id, or check that the recorded IDs match the replayed position and fall back to the position when they do not.
• A repair step that rewrites the affected deletes (would change op contents, so it only works for documents that can be re-exported).

Found by the loro.js ↔ Rust differential suite (#1136) and the #1135 review.