fix: reject creating unknown-type containers instead of panicking

[zxch3n]

Summary

This version can import, export, check out, move and delete a container whose type it doesn't know (ContainerType::Unknown, written by a newer Loro). It must never create one, because the content is opaque to it. On main, several paths could be driven into creating one and panicked. The first report was JS applyDiff with "🦜:cid:0@1:Unknown(9)" hitting unreachable!() in Handler::new_unattached (found by the author of #1138).

apply_diff / revert_to / undo

• Pre-check: LoroDoc::check_apply_diff_creates_no_unknown_container runs before _apply_diff touches the doc. It returns LoroError::ArgErr when the batch would create an unknown container. Handler::new_unattached returns the same error as a backstop.
• Why it has to predict the apply loop: local ops have no rollback (apply_diff applies part of a batch before returning an error (e.g. OutOfBound) #1154), and the loop changes its own inputs as it runs. So the check looks at the whole batch first and predicts:
  • fresh: Map/List container values, the metas of created or moved tree nodes, and the mergeable children of those get new ids. Their entries apply to new, empty containers and are never skipped.
  • revived: a mergeable marker that is written makes the child and its descendants reachable again. Ancestors come from arena parents, because a hidden mergeable container has no visible path.
  • The loop writes the marker of mergeable(parent, key) under the parent as it resolves it. When that is not the value's own id, the child is checked as new and empty. This covers a parent the undo manager already remapped, or a value under another key.
  • removed: a deleted child and its descendants are skipped by later entries.
• Direction of errors: every prediction errs towards checking an entry, never towards skipping one.
• Contract checked in debug builds: the check returns the entries holding unknown containers that it predicted to be skipped. In debug builds the loop asserts three things: it skips those entries; it never applies unknown values from an entry the plan checked differently; and no entry of an accepted batch fails on creating an unknown container.
• MovableList moves are decided by unknown_container_created_by_delta, which reuses fix: apply multi-child MovableList diffs as moves without losing elements #1138's plan_moves.
• Entries that target an unknown container are a no-op, as before, and are not checked.
• Undo/redo: a step that would recreate an unknown container fails as a whole with that ArgErr.
  • The doc is unchanged and the step is dropped; its edits stay in the doc.
  • The rebase of the manager's other steps over the undo now waits until the step is accepted.
  • The rejected step's own changes are composed into the remote diff of its row, so the steps before it are rebased over them like over remote changes.
  • The next undo therefore undoes the step before, with the same limits as after a remote edit. If the rejected step moved an element an earlier step inserted, undoing that earlier step finds nothing to do and falls through, exactly as after the same move by a remote peer (tested).
  • With a merge interval, the rejected step is every edit merged into it (loro-crdt's default is 1 s).
  • This is documented on UndoManager::undo/redo in Rust and JS.

Other paths

• Container copies: insert_container / set_container / push_container reject an unknown container, or a detached or attached container holding one, before any op is written. Before this, Rust LoroUnknown silently became an empty copy.
• Import / checkout: importing or checking out a change that creates an unknown container and edits it in the same range panicked in UnknownState::apply_diff. That is the forward-compatibility path itself.
• Cursors: get_cursor_pos on Map, Tree, Counter or unknown containers now returns not found (undefined in JS).
• parent(): a child whose parent is an unknown container, as stored in a newer snapshot, now returns Handler::Unknown. JS throws the readable "unknown container" error.
• Undo cursors: undo/redo with an onPush cursor on a missing container no longer panics.
• Unknown diffs: Diff::compose_ref composes two Diff::Unknown as a no-op.
• JS doc.diff(): since fix: restore deleted mergeable children once on revert/apply_diff/undo #1134 the diff can hold an entry for an unknown container, which made the conversion throw "Unknown diff". Those entries have no content and applying them is a no-op, so they are left out.
• JS events (loro-crdt: subscribers get no events at all when an event batch touches an unknown container type #1151): an event batch that touches an unknown container used to be dropped entirely, known containers included. Now only the container events that can't be converted are left out and logged.
• Error message: all three creation paths now use one helper and one message: Cannot create a container of type Unknown(9): the type is unknown to this version of Loro.

Rationale and test-forging recipes: context/unknown-containers.md.

Full-state batches (#1134)

A batch marked full_state (from LoroDoc::diff, or JS applyDiff(diff, { fullState: true })) aligns a re-activated mergeable child with the state this doc kept for it. Keeping an unknown container that is already there creates nothing, so the check must see the aligned edits, not the raw full states.

• The plan is the single source of alignment: when the batch holds an unknown container value, plan_full_state_batch aligns the whole batch once, before anything is applied. The check runs on the planned edits, and the loop applies those same edits instead of aligning again.
• No side effects: align_full_state never mutates now; a counter gets an increment diff.
• Other batches: batches without unknown values skip the plan, and the loop aligns each entry itself as on main.
• Skipped entries: the plan doesn't count marker writes from an entry whose target is unreachable before the batch, because the loop skips it. JS applyDiff accepts any order, so the entry that revives it may come later.
• Where the plan and the loop can differ: the loop keeps its own full_state_targets, because only it follows container_remap as containers are recreated. It uses a planned edit only when it aligns the same container the plan aligned. The one expected difference is a mergeable child of a container the batch recreates. The plan marks it fresh and checks it as new and empty; the loop aligns it against the new child.
• Still rejected: a child that alignment really has to create. For example, the doc kept no state for it, a plain list recreates it, or its parent is recreated by the batch.

Review follow-ups

Round 1 (review of 9bd96f3):

• Partial apply: fixed with the fresh/revived/removed model.
• False rejections: removed-child prediction; unknown targets skipped.
• Other items: duplicate U; undo drops only the rejected step; get_parent test; compose_ref arm; one error message; performance.

Round 2 (#1134 review round 4):

• Full-state false rejection: fixed by checking the aligned edits.
• JS doc.diff(): no longer throws on unknown entries.
• Issues opened: loro-crdt: subscribers get no events at all when an event batch touches an unknown container type #1151 and apply_diff applies part of a batch before returning an error (e.g. OutOfBound) #1154.

Round 3 (review of b550599):

• B1: the planner resolved ids with the initial remap. The mergeable child of a container recreated by the batch was aligned against its old hidden state and accepted, then failed half way. Fixed: recreated containers and their mergeable children are fresh in the plan and in the check (also covers the N4 crafted is_move case).
• B2: a rejected undo step had already rebased the other steps over itself, so later undos removed the wrong edits. Fixed: the rebase is deferred, and the rejected step's changes are composed into its row's remote diff.
• B3: the skip prediction used the visible path, which is empty below a hidden mergeable container. Undo returned Ok(true) with U silently lost. Fixed: ancestors come from arena parents.
• Structure: the plan is now the single source of alignment and removes the 2× alignment cost. Prediction mistakes are asserted in debug builds.
• Twin oracle: the reviewer's random test is in the repo (below).

Round 4 (review of df94c67):

• R3-1 (blocking): undo into a host the undo manager had already recreated (list element, tree meta, nested) returned Ok(true) half applied in release; a debug build panicked and the JS dev build trapped. The check looked into the old hidden child while the loop writes to mergeable(new host, key). Fixed as above.
• R3-2: a mergeable value under another key, in a crafted batch. Fixed by the same change.
• R3-3: a reordered full-state batch. Fixed by the skip-aware plan above.
• _apply_diff assertion: it now also asserts, in debug builds, that an accepted batch never fails on creating an unknown container.
• Undo docs: the rebase limits are documented.
• Oracle: the reviewer's collaboration oracle (remote peer, interleaved undo/redo, reordered batches, 3-level mergeable chains, pinned fork peer ids) is in the repo.
• The fixes are the reviewer's validated prototype.

Validation

Rust. cargo test -p loro -p loro-internal: 1300 passed, 0 failed, 23 ignored. That is on main 5c7552d, which includes #1125, #1152 and #1153. pnpm check is clean. cargo fmt was applied to the touched files only.

Tests:

• crates/loro/tests/unknown_container_atomicity.rs:
  • reproductions from all rounds, including B1 under a recreated map, list element and tree node (plus the crafted is_move variant);
  • B3 for a mergeable map and a mergeable tree;
  • B2 undo, with and without remote edits, and redo symmetry.
  • The 5 round-3 tests fail on b550599.
  • Round 4 adds R3-1 (list element, tree meta and nested; with and without the remote fill), R3-2 and R3-3; the 3 new tests fail on df94c67.
  • A test pins that undo after a rejected move behaves like undo after the same remote move.
• crates/loro/tests/unknown_container_twin_oracle.rs:
  • random edits on a forged doc and on a twin holding Counters in their place;
  • every apply_diff, full-state and incremental, every revert_to and every undo/redo must be rejected without changes, or give the same state as the twin;
  • an undo oracle: undoing everything leaves exactly the edits of the rejected steps, and redo restores all.
  • Default 60 seeds, about 7 s in debug; LORO_UNKNOWN_TWIN_SEEDS runs more.
  • Seeds 0–2999 pass in release with debug assertions on: 29,576 matching results, 39,321 clean rejections, no partial apply, no mismatch, no assertion.
  • On b550599 it fails at seed 3 with a partial apply.
  • The collaboration variant adds a remote peer, undo/redo interleaved with edits and syncs, natural, reversed and shuffled batch order, and 3-level mergeable chains. Seeds 0–2999 pass in release with debug assertions: 22,737 matching results, 6,895 clean rejections, nothing else.
• Shared forging helpers: crates/loro/tests/unknown_container_support/.

WASM. pnpm -C crates/loro-wasm build-dev plus package tests: vitest 434 passed / 33 files, plus tsc, the CommonJS smoke test, deno and bun. The JS reproductions of B1, B2, B3 and R3-1 use Rust-generated fixtures. R3-1 used to trap the dev build.

Performance (measured at df94c67; round 4 only adds checks to batches that hold unknown containers, and the reviewer measured the same numbers). Release build, median of 7 runs per test, main 5c7552d vs this PR (df94c67).

• Plain paths: 3 rounds in the order M,P,P,M,M,P, 1-minute loadavg 6.5–7.8.
• Full-state paths: 6 alternating rounds, loadavg 5.3–5.9.
• Values are the median across rounds.

Scenario	main	this PR	PR/main
apply_diff list 100k ints → fresh doc	26.6 ms	26.2 ms	0.99
apply_diff map 100k keys → fresh doc	26.4 ms	26.5 ms	1.00
apply_diff 10k child containers → fresh doc	20.0 ms	20.5 ms	1.03
apply_diff 100k plain values into a MovableList	35.4 ms	33.5 ms	0.95
revert_to empty from 100k list	137.8 ms	137.7 ms	1.00
undo 100k list insert	122.7 ms	124.0 ms	1.01
unknown element move in 100k MovableList: apply_diff / revert_to / undo	0.35 / 2.61 / 2.62 ms	0.33 / 2.66 / 2.64 ms	≈1.0
full-state revive 100k mergeable list, no unknown / 1 unknown	5.62 / 5.83 ms	5.85 / 5.97 ms	1.04 / 1.02
full-state revive mergeable map 100k keys, no unknown / 1 unknown	13.50 / 13.60 ms	13.43 / 13.49 ms	0.99 / 0.99
full-state revive 2000 mergeable maps × 50 keys, no unknown	14.41 ms	14.77 ms	1.02
full-state revive 2000 mergeable maps × 50 keys, 1 unknown each	15.74 ms	17.84 ms	1.13
undo hiding 2000 mergeable maps, no unknown / 1 unknown each	16.56 / 17.22 ms	17.15 / 17.77 ms	1.04 / 1.03

The review measured 1.7–1.8× on batches holding unknown values; that came from aligning twice and is gone. The remaining +13% on 2000 small targets that each hold an unknown container is the plan's own bookkeeping and the check. It only runs for batches that hold unknown values.

Known leftovers (not changed here)

• loro-crdt: subscribers get no events at all when an event batch touches an unknown container type #1151: an event whose container holds an unknown child value is still left out; representing that child in JS is an API question.
• apply_diff applies part of a batch before returning an error (e.g. OutOfBound) #1154: apply_diff still applies part of a batch when it fails for other reasons, such as OutOfBound.
• JS diff() omits unknown entries: lossless in this version. A newer version that knows the type reads a missing full-state entry as "empty", so applying such a diff() there with fullState clears the hidden state of a kept container of that type (review NB1; documented in the context doc).
• JSON updates can't round-trip mergeable markers (Binary becomes a list of numbers).
• Other undo errors: an accepted undo step whose _apply_diff fails for another reason is still only logged, as on main.

🤖 Generated with Claude Code

[github-actions]

WASM Size Report

• Original size: 3272.76 KB
• Gzipped size: 1087.19 KB
• Brotli size: 760.88 KB

[zxch3n]

Addressed the adversarial review and merged main (#1138 and #1135; the movable_list_apply_delta.rs conflict is resolved by taking #1138's version plus ? on new_unattached). Head: 18447e5.

Blocking 1, partial application: the pre-check now predicts what the apply loop does to recreated (fresh), revived (mergeable marker) and removed (deleted child) containers. Every prediction errs towards checking a diff. With this change:

• apply_diff(doc.diff(v1, v0)) and revertTo on the m.k = [U, 1] reproduction fail with no change. The same holds for tree meta and mergeable list, in Rust and JS.
• A new 200-seed random test asserts that every apply_diff(diff(latest, v)), revert_to and undo/redo is atomic.
• Turning the new predictions off makes it fail again.

Non-blocking items:

• 2 (fix: restore deleted mergeable children once on revert/apply_diff/undo #1134): handled on the fix: restore deleted mergeable children once on revert/apply_diff/undo #1134 side, since it lands after this PR. Details are in the description.
• 4: duplicate U is covered by reusing fix: apply multi-child MovableList diffs as moves without losing elements #1138's plan_moves.
• 5: both false rejections are fixed (removed-child prediction; unknown targets are skipped).
• 6, performance: back to main's numbers. The full table is in the description.
• 7, undo: a step that would recreate U fails as a whole with ArgErr, the doc is unchanged, and the step is dropped. The stack is not cleared and the next undo undoes exactly one earlier step. Documented on Rust and JS undo/redo.
  • I did not apply the rest of such a step. That would drop U's insert from the diff, and a wrongly predicted MovableList move would then lose an existing U.
• 8: get_parent test added, forged through the shallow-snapshot KV header.
• 9: compose_ref has a no-op arm for (Unknown, Unknown).
• 10: one error message everywhere.

JS events: opened #1151 and fixed the simple part here. A batch touching an unknown container now keeps its other events. Representing an unknown child value inside a parent's event is left to #1151.

Rust: 1209 passed, 0 failed; the combination with #1125 and #1134 passes 1251. pnpm check is clean. WASM vitest: 397 passed and 1 flaky ephemeral timeout, which passes 3/3 alone; tsc, deno and bun pass.

[zxch3n]

Merged main with #1134 (1eb14b3) and fixed the combination blocker from #1134 review round 4 (B1). Head: b550599.

B1, full-state revival falsely rejected: re-activating a mergeable Map, List, MovableList or tree meta that still holds an unknown container was rejected as creating it.

• Fix: the check now plans the full-state alignment first. plan_full_state_batch replays the loop's full_state_targets bookkeeping, and align_full_state gains a dry_run flag that skips the counter increment. The check then runs on the planned edits, with the aligned targets treated as never skipped.
• Still rejected: creating U (no hidden state, or a kept hidden child whose aligned edit recreates U), before anything is written.
• Default incremental path: unchanged.

Tests:

• Rust and WASM cover the 4 holders × full-state, both the "keeps U" and "must create U" cases, plus the kept-hidden-child case. Each fails without the corresponding part of the fix.
• The 200-seed random check now applies every diff() both as a full state and incrementally, and includes a mergeable map with a regular child.
• The WASM mergeable fixtures are forged in Rust, because JSON updates lose the binary markers. A drift test guards them.

JS doc.diff(): since #1134 it could throw "Unknown diff" for an unknown container's entry. Those entries are now left out, which is lossless because applying one is a no-op.

Issues opened: #1154 (apply_diff partially applies on other errors, with a minimal repro). #1151 was opened earlier for events.

Results:

• Rust: 1249 passed, 0 failed.
• Combined with fix: reject malformed movable list move/set ops on import instead of aborting #1125 at a45d957: 1271 passed, 0 failed.
• pnpm check is clean.
• WASM: vitest 416 passed, plus tsc, CommonJS, deno and bun.
• Performance, alternating with main at loadavg 6.7–7.9: all scenarios match main within noise, except reviving a 100k mergeable list that holds an unknown element (5.9 → 10.0 ms). That is the planner's second alignment, and it only runs for batches with unknown values. The table is in the description.

[zxch3n]

Addressed the second adversarial review (of b550599) and merged main (5c7552d: #1125, #1152, #1153). Head: df94c67.

• B1 (partial apply under a recreated parent): the plan and the check treat containers the batch may recreate, and their mergeable children, as new and empty, erring towards "recreated". This also covers the crafted is_move N4 case.
• B2 (undo after a rejected step undid the wrong edits):
  • the step's rebase of the other steps now waits until the step is accepted;
  • on rejection, the step's own changes are composed into its row's remote diff.
  • U1, U2, redo symmetry and the B2 oracle now give the expected results.
• B3 (undo returned Ok(true) and lost U): the skip prediction follows arena parents instead of the visible path.
• Structure:
  • The plan is the single source of alignment. For batches holding unknown values it aligns once, the check runs on the planned edits, and the loop applies those edits instead of aligning again.
  • align_full_state has no side effects.
  • The loop uses a planned edit only for the container the plan aligned. Otherwise it aligns at runtime, which only happens for fresh children or batches without unknown values.
  • Debug builds assert that the loop never applies an entry the check predicted as skipped, or that it checked differently.
• Tests:
  • Your reproductions are in unknown_container_atomicity.rs (5 new tests; all fail on b550599) and in JS, using Rust-generated fixtures.
  • The twin oracle and the undo oracle are in unknown_container_twin_oracle.rs, 60 seeds by default, about 7 s in debug.
  • Seeds 0–2999 in release with debug assertions: no partial apply, no mismatch, no assertion.
• Results:
  • Rust: 1295 passed, 0 failed.
  • pnpm check is clean.
  • WASM: vitest 433 passed, plus tsc, CommonJS, deno and bun.
• Performance (alternating with main, loadavg 5.3–7.8):
  • Plain paths match main.
  • Batches without unknown values skip the plan.
  • The 1.7× cost on batches holding unknown values is gone.
  • What's left is +13% when reviving 2000 small mergeable maps that each hold an unknown container. The table is in the description.

NB1 (diff() omission read as "empty" by a newer version) is documented in context/unknown-containers.md and listed as a leftover. NB3 (ArgErr as the rejection signal) is documented there too.

[zxch3n]

Addressed the third adversarial review (of df94c67). Head: fc16330; CI passes. Main hasn't moved (5c7552d), so no merge was needed.

• R3-1 (blocking) and R3-2: in the check's Map pass, a mergeable value is checked as new and empty when the loop writes it under mergeable(parent, key) for a different id. That is either a parent the undo manager already remapped, or a value under another key. This is your validated prototype.
• R3-3: the full-state plan no longer counts marker writes from an entry whose target is unreachable before the batch, because the loop skips it. Also your prototype.
• _apply_diff assertion: it now asserts in debug builds that no entry of an accepted batch fails on creating an unknown container. It matches that exact error rather than any ArgErr; the undo wrapper does the same.
• B2 limit: documented in the Rust and JS undo() docs, the changeset and the context doc. A rejected step's edits are rebased over like a remote peer's, including the move case you found. undo_after_a_rejected_move_matches_a_remote_move pins that undo after the rejected move matches undo after the same remote move.
• Tests:
  • R3-1 (list element, tree meta and nested; with and without the remote fill), R3-2 and R3-3 are in unknown_container_atomicity.rs. All three fail on df94c67, two of them via the debug assertions.
  • R3-1 is also covered in JS; it used to trap the dev build.
  • Your collaboration oracle (remote peer, interleaved undo/redo, reordered batches, 3-level mergeable chains, pinned fork peer ids) is in unknown_container_twin_oracle.rs, 60 seeds by default.
  • Seeds 0–2999 in release with debug assertions: collaboration 22,737 matching results and 6,895 clean rejections; original twin 29,576 / 39,321; undo oracle clean. No partial apply, no mismatch, no assertion.
• Results:
  • Rust: 1300 passed, 0 failed.
  • pnpm check is clean.
  • WASM: vitest 434 passed, plus tsc, CommonJS, deno and bun.