diff --git a/.changeset/known-history-json-lossy.md b/.changeset/known-history-json-lossy.md new file mode 100644 index 000000000..351da4e03 --- /dev/null +++ b/.changeset/known-history-json-lossy.md @@ -0,0 +1,9 @@ +--- +"loro-crdt": patch +--- + +Fix a 1.16.4 regression where history relayed through JSON was rejected with `UsedOpID`, without letting binary import treat those conversions as identical. + +Binary `import`, `import_with`, fast updates, fast snapshots, and `import_batch` stay exact: overlapping op ids must be structurally identical, including finite doubles, large integers, binary versus a list, non-finite doubles versus null, marker strings versus containers, and unknown payloads. JSON import, and `import_with_history_mode` / `import_batch_with_history_mode` with `ImportHistoryMode::JsonLossy`, tolerate the conversions JSON is known to introduce: I64/I64 or I64/Double that match as f64 when at least one integer is outside `[-2^53, 2^53]`; binary versus a same-length list of byte numbers; non-finite doubles versus null; a container-marker string versus any Container; unknown payloads skipped. Finite doubles still use `==`. The receiver keeps its existing prefix values. + +Finite doubles round-trip because `serde_json`'s `float_roundtrip` feature is enabled on the normal dependency. Counter deltas are integer-encoded only when the fraction is exactly 0, and a nonzero delta is applied on import even when it is smaller than `f64::EPSILON`. The JSON format is otherwise unchanged, and this does not restore values a JSON importer already dropped. Callers must opt in before a binary blob that carries those degraded values will import. diff --git a/Cargo.lock b/Cargo.lock index 26f9a190f..fee6c1aae 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1489,6 +1489,14 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "json-float-consumer" +version = "0.0.0" +dependencies = [ + "loro 1.16.2", + "serde_json", +] + [[package]] name = "lazy_static" version = "1.5.0" diff --git a/Cargo.toml b/Cargo.toml index 5dcc92dda..d6e0fa725 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -14,6 +14,7 @@ members = [ "crates/kv-store", "crates/loro-wasm-tools", "crates/generic-btree", + "crates/json-float-consumer", ] resolver = "2" diff --git a/context/import-peer-id-reuse.md b/context/import-peer-id-reuse.md index 1ded5c612..2c7203d07 100644 --- a/context/import-peer-id-reuse.md +++ b/context/import-peer-id-reuse.md @@ -61,14 +61,90 @@ is per atom range, not per change: (`unicode_start` is an arena offset), and a one-atom delete ignores the sign of `signed_len`, because slicing a reversed delete down to one atom yields `-1` where a forward one yields `1`. -- **JSON imports** use `ImportedValues::Lossy`: JSON text does not round-trip every - value (`NaN` becomes `null`, binary may come back as a list), so value payloads - are not compared there. Containers, positions, text, keys, ids and deps still are. +- **Value payloads** use the caller's `ImportedValues` mode. `Exact` is the + default for binary `import`, `import_with`, fast updates, fast snapshots, and + `import_batch`, and it is always available (not test-only). It is structural + equality: finite doubles use `==` (a 1 ULP gap is rejected), large integers + stay distinct, binary is not a list, non-finite doubles are not null, a marker + string is not a Container, and unknown payloads are compared. `JsonLossy` is + used by `import_json_updates` and by `import_with_history_mode` / + `import_batch_with_history_mode` when the caller opts in, because a later + binary export can carry values degraded by an earlier JSON hop. It accepts, + symmetrically and recursively inside lists, maps, and rich-text styles: + `Binary` versus a list of the same u8 numbers, non-finite doubles (`NaN`, + positive or negative infinity) versus `Null`, and `Double(d)` versus `I64(i)` + when `d == i as f64`. JS numbers erase the distinction between an integral + Double and I64; JSON text written by JS also drops the `.0` + (`crates/loro-wasm/src/convert.rs`, `js_json_schema_to_loro_json_schema`; + `LoroValueVisitor` in `loro-common/src/value.rs`). Two I64s also match when + they convert to the same f64 and at least one has `|i| > 2^53`: a JS-number + relay can round `2^60 + 1` to `2^60`. Inside `[-2^53, 2^53]`, distinct + integers still differ and Double/I64 equality is exact numerically. Finite + Double/Double stays `==` in both modes. `serde_json`'s `float_roundtrip` + feature is on the normal dependency, so a true JSON text round trip of a + finite double keeps its bits and passes `Exact`. +- **Large-integer JSON paths.** Native JSON text preserves an I64, but parsing it + into JS numbers can lose precision. The WASM object importer uses + `serde_wasm_bindgen::Deserializer::deserialize_any`, which reads numbers outside + JS's safe integer range (`|n| > 2^53 - 1`) as Double. The generic + `js_value_to_loro_value` also returns Double for `|n| > 2^53`, but returns I64 + at the boundary; a JS-written JSON string can decode a rounded I64 instead. + Both representations are tolerated, including JS's shortest decimal spelling + (for example `JSON.stringify(2 ** 60)` writes `1152921504606847000`, which converts + to the same f64). The direct WASM schema exporter currently uses + `serde_wasm_bindgen::Serializer::serialize_i64`, which rejects unsafe I64s + rather than rounding them. The tolerance covers history relayed through JS + parsing native JSON; it does not change that exporter's existing limitation. +- **Ambiguous payload kinds.** A valid `🦜:cid:` string becomes a Container in + JSON (`LoroValueVisitor::visit_str`). JSON peer compression can then reinterpret + the string's peer as an index, so a marker-string versus Container pair is + accepted without comparing the encoded id. Two Containers or two strings still + must match. Unknown-op payloads (`FutureInnerContent::Unknown`) are not + compared in `JsonLossy`: they may carry nested LoroValues or arena references + whose meaning this version cannot establish. The op's prop and container still + must match. `Exact` compares the payload. This retains the prior JSON bypass + for that opaque kind; a later binary relay of it is accepted only when the + caller opts into `JsonLossy`. +- **Counter ops** carry tagged `OwnedValue::F64`/`I64` values in JSON + (`encoding/json_schema.rs::decode_op`), both decoded to `Counter(f64)`. + Finite increments therefore remain comparable numerically, including integral + increments. NaN equals NaN. A nonzero counter is not treated as zero: integer + encoding is used only when `fract() == 0.0` (and the existing magnitude bound + holds), so `±1e-17` is not stored as `I64(0)`. Applying that delta does not + treat a magnitude below `f64::EPSILON` as an empty diff. Already-corrupted old history + cannot be recovered. A non-finite counter increment serialized as JSON text is + already invalid at decode because its tagged f64 becomes null; this is not a + new known-history rejection. +- **Other values** are compared exactly. Unlike main's blanket JSON value bypass, + JSON imports now reject genuinely different representable payloads. Text, + op container ids, positions, keys, element ids, style metadata and deps still + must match on every import path. + The known prefix is then trimmed, so the receiver keeps its own values. + This fixes the 1.16.4 rejection of JSON-relayed history without changing the JSON + format: a JSON importer still gets a list instead of `Binary` (including mergeable + container markers), `Null` instead of a non-finite double, an I64 instead of an + integral Double, a rounded large integer, or a Container instead of a + marker-looking string. It does not repair that importer's lost values or make + its state identical to the binary receiver's. - History below the shallow root, or any local change `get_change` cannot find, is not compared. The import then behaves as it did before this check. ## What it does not catch +- **Conflicts indistinguishable from JSON loss, only under `JsonLossy`.** Two + actual writers using the same op id can store Binary versus a list of the same + bytes, NaN/positive or negative infinity versus null, or an integral Double + versus the matching I64. Different large integers (or a large integer versus a + Double) also pass when they convert to the same f64 and at least one is + outside `[-2^53, 2^53]`, such as `2^60 + 1` versus `2^60`; distinct integers + inside that range remain checked exactly. This ambiguity exists even when both + writers stored I64s directly. Default binary import rejects it (`UsedOpID`). + It passes only for `import_json_updates` or an explicit `JsonLossy` binary + import. A valid marker-string versus Container pair and differing unknown-op + payloads also pass under those bypasses. The receiver keeps its own prefix; + the conflicting value (or its type) remains different, while unrelated matching + values stay the same. `Exact` must not leave the documents with equal version + vectors and different values. - **Delta sync.** If the conflicting prefix is not in the import (the sender exported from the receiver's version vector), nothing can be compared: the receiver's vv already claims those ids. A tail position past the end of a @@ -120,9 +196,27 @@ universal ratio. and re-imports that must still succeed (piecewise vs merged change stores, every op kind including `NaN` and one-atom reversed deletes, shallow docs). - `crates/loro-wasm/tests/import_reused_peer_id.test.ts`. - -- `oplog::known_history::tests`: cross-arena container identity and Unicode atom - comparisons, and a dependency mismatch inside a merged import's cold prefix. +- `crates/loro/tests/import_json_relay.rs`: JSON-to-binary relays through full + updates, snapshots and batch import using `ImportHistoryMode::JsonLossy`; + binary values, mergeable text/counter, non-finite/integral doubles, rounded + large integers as I64 and Double (including the `2^53` boundary, both signs + and i64 extrema), marker strings with peer compression, tagged counter values + and unknown payloads, nested/sequence/style values, both comparison directions + and JSON re-imports, later relay edits, retained prefix history, genuine + text/value conflicts that must still be rejected, and an exact-path rejection + of `2^60` versus `2^60 + 1`. +- `oplog::known_history::json_lossy_value_tests`: the precise value equivalences + and rejection boundaries, including nested values, byte-list contents, numeric + representations, 1 ULP finite doubles, marker-string ambiguity and unknown-op + prop/container checks. +- `oplog::known_history::two_arena_tests`: cross-arena container identity and + Unicode atom comparisons, and a dependency mismatch inside a merged import's + cold prefix. +- `crates/json-float-consumer`: a normal `loro` dependency, without the `loro` + crate's dev-only `float_roundtrip`, checks finite-double JSON text then an + exact binary reimport, and the same `2^60` conflict under both modes. +- `crates/loro/tests/contracts/counter.rs`: encoding `1e-17` and importing the + binary update yields `1e-17`, not `0`. - `oplog::change_store::test`: identical blocks stay lazy, dirty caches shadow old KV bytes, repeated snapshots allocate only new text/list values, and `merged_cold_text_overlap_is_accepted_without_parsing_the_known_block` proves diff --git a/crates/json-float-consumer/Cargo.toml b/crates/json-float-consumer/Cargo.toml new file mode 100644 index 000000000..f1d6b2749 --- /dev/null +++ b/crates/json-float-consumer/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "json-float-consumer" +version = "0.0.0" +edition = "2021" +publish = false +description = "External loro consumer without serde_json float_roundtrip" + +[dependencies] +loro = { path = "../loro" } +# Deliberately not the workspace dependency and not float_roundtrip. Feature +# unification must come from loro-internal's normal dependency, not from a +# dev-dependency of the loro crate. +serde_json = "1.0" diff --git a/crates/json-float-consumer/src/lib.rs b/crates/json-float-consumer/src/lib.rs new file mode 100644 index 000000000..6a7c05bb8 --- /dev/null +++ b/crates/json-float-consumer/src/lib.rs @@ -0,0 +1,2 @@ +//! Workspace member whose normal `loro` dependency must round-trip finite +//! doubles through JSON text without the `loro` crate's dev-dependency features. diff --git a/crates/json-float-consumer/tests/json_float.rs b/crates/json-float-consumer/tests/json_float.rs new file mode 100644 index 000000000..f5f7ed7d6 --- /dev/null +++ b/crates/json-float-consumer/tests/json_float.rs @@ -0,0 +1,50 @@ +use loro::{ExportMode, ImportHistoryMode, LoroDoc, LoroError, VersionVector}; + +#[test] +fn finite_doubles_survive_json_text_then_exact_binary_import() { + for number in [ + 51.248178375505404, + -93.31137037688033, + 2.0030397744267762e-253, + ] { + let a = LoroDoc::new(); + a.get_map("m").insert("n", number).unwrap(); + let json = a + .export_json_updates(&VersionVector::default(), &a.oplog_vv()) + .unwrap(); + let text = serde_json::to_string(&json).unwrap(); + let b = LoroDoc::new(); + b.import_json_updates(text).unwrap(); + b.get_map("m").insert("extra", 1).unwrap(); + let updates = b.export(ExportMode::all_updates()).unwrap(); + a.import(&updates).unwrap(); + assert_eq!(a.get_deep_value(), b.get_deep_value()); + } +} + +#[test] +fn exact_import_rejects_large_integers_one_apart() { + let local = 1i64 << 60; + let remote = local + 1; + let a = LoroDoc::new(); + a.set_peer_id(1).unwrap(); + a.get_map("m").insert("n", local).unwrap(); + let b = LoroDoc::new(); + b.set_peer_id(1).unwrap(); + b.get_map("m").insert("n", remote).unwrap(); + b.get_map("m").insert("extra", 1).unwrap(); + let bytes = b.export(ExportMode::all_updates()).unwrap(); + + let before = a.get_deep_value(); + let vv = a.oplog_vv(); + let err = a.import(&bytes).unwrap_err(); + assert!(matches!(err, LoroError::UsedOpID { .. })); + assert_eq!(a.get_deep_value(), before); + assert_eq!(a.oplog_vv(), vv); + assert!(a.oplog_vv() != b.oplog_vv() || a.get_deep_value() == b.get_deep_value()); + + a.import_with_history_mode(&bytes, "", ImportHistoryMode::JsonLossy) + .unwrap(); + assert_eq!(a.oplog_vv(), b.oplog_vv()); + assert_ne!(a.get_deep_value(), b.get_deep_value()); +} diff --git a/crates/loro-internal/Cargo.toml b/crates/loro-internal/Cargo.toml index ca6bfd102..0d94e0fa3 100644 --- a/crates/loro-internal/Cargo.toml +++ b/crates/loro-internal/Cargo.toml @@ -27,7 +27,7 @@ rustc-hash = { workspace = true } serde = { workspace = true } thiserror = { workspace = true } enum-as-inner = { workspace = true } -serde_json = { workspace = true } +serde_json = { workspace = true, features = ["float_roundtrip"] } bytes = { workspace = true } xxhash-rust = { workspace = true } serde_columnar = { workspace = true } diff --git a/crates/loro-internal/src/arena.rs b/crates/loro-internal/src/arena.rs index 871faf6f2..6a987d1f3 100644 --- a/crates/loro-internal/src/arena.rs +++ b/crates/loro-internal/src/arena.rs @@ -667,11 +667,18 @@ impl SharedArena { (self.inner.values.lock()[range]).to_vec() } - /// Whether the values in two ranges are equal, without cloning them. - pub(crate) fn value_slices_eq(&self, a: Range, b: Range) -> bool { + /// Compare values in two ranges without cloning them or taking the lock twice. + pub(crate) fn value_slices_eq( + &self, + a: Range, + b: Range, + eq: impl Fn(&LoroValue, &LoroValue) -> bool, + ) -> bool { let values = self.inner.values.lock(); match (values.get(a), values.get(b)) { - (Some(a), Some(b)) => a == b, + (Some(a), Some(b)) => { + a.len() == b.len() && a.iter().zip(b.iter()).all(|(a, b)| eq(a, b)) + } _ => false, } } diff --git a/crates/loro-internal/src/encoding.rs b/crates/loro-internal/src/encoding.rs index 390eae05f..8d10807cd 100644 --- a/crates/loro-internal/src/encoding.rs +++ b/crates/loro-internal/src/encoding.rs @@ -12,6 +12,7 @@ use outdated_encode_reordered::{import_changes_to_oplog, ImportChangesResult}; pub(crate) use value::OwnedValue; use crate::change::Change; +use crate::oplog::ImportedValues; use crate::version::{Frontiers, VersionRange}; use crate::LoroDoc; use crate::{oplog::OpLog, LoroError, VersionVector}; @@ -233,8 +234,12 @@ pub struct ImportStatus { pub(crate) fn decode_oplog( oplog: &mut OpLog, parsed: ParsedHeaderAndBody, + values: ImportedValues, ) -> Result { - let changes = decode_oplog_changes(oplog, parsed)?; + let changes = decode_oplog_changes(oplog, parsed, values)?; + // Outdated decoders do not go through `import_changes_and_apply`. Compare + // here so a restored outdated import still honors the caller's mode. + let changes = oplog.check_and_trim_known_part_of_changes(changes, values)?; let result = apply_decoded_changes_to_oplog(oplog, changes)?; if result.has_deps_before_shallow_root { return Err(LoroError::ImportUpdatesThatDependsOnOutdatedVersion); @@ -246,13 +251,14 @@ pub(crate) fn decode_oplog( pub(crate) fn decode_oplog_changes( oplog: &mut OpLog, parsed: ParsedHeaderAndBody, + values: ImportedValues, ) -> Result, LoroError> { let ParsedHeaderAndBody { mode, body, .. } = parsed; match mode { EncodeMode::OutdatedRle | EncodeMode::OutdatedSnapshot => { Err(LoroError::ImportUnsupportedEncodingMode) } - EncodeMode::FastSnapshot => fast_snapshot::decode_oplog(oplog, body), + EncodeMode::FastSnapshot => fast_snapshot::decode_oplog(oplog, body, values), EncodeMode::FastUpdates => fast_snapshot::decode_updates(oplog, body.to_vec().into()), EncodeMode::Auto => unreachable!(), } diff --git a/crates/loro-internal/src/encoding/fast_snapshot.rs b/crates/loro-internal/src/encoding/fast_snapshot.rs index 0bf0f5130..53fa01cf9 100644 --- a/crates/loro-internal/src/encoding/fast_snapshot.rs +++ b/crates/loro-internal/src/encoding/fast_snapshot.rs @@ -346,7 +346,11 @@ fn encode_snapshot_inner_with( snapshot } -pub(crate) fn decode_oplog(oplog: &mut OpLog, bytes: &[u8]) -> Result, LoroError> { +pub(crate) fn decode_oplog( + oplog: &mut OpLog, + bytes: &[u8], + values: crate::oplog::ImportedValues, +) -> Result, LoroError> { let oplog_len = bytes .get(0..4) .ok_or_else(|| LoroError::DecodeError("decode_oplog: missing length prefix".into()))?; @@ -358,7 +362,8 @@ pub(crate) fn decode_oplog(oplog: &mut OpLog, bytes: &[u8]) -> Result { let c_abs = c.abs(); - if c_abs.fract() < f64::EPSILON && (c_abs as i64) < (2 << 26) { + if c_abs.fract() == 0.0 && (c_abs as i64) < (2 << 26) { Value::I64(*c as i64) } else { Value::F64(*c) diff --git a/crates/loro-internal/src/event.rs b/crates/loro-internal/src/event.rs index b58cd1084..9bb5f1014 100644 --- a/crates/loro-internal/src/event.rs +++ b/crates/loro-internal/src/event.rs @@ -404,8 +404,9 @@ impl InternalDiff { InternalDiff::Map(m) => m.updated.is_empty(), InternalDiff::Tree(t) => t.is_empty(), InternalDiff::MovableList(t) => t.is_empty(), + // 0 is empty. A smaller nonzero delta such as 1e-17 is a real increment. #[cfg(feature = "counter")] - InternalDiff::Counter(c) => c.abs() < f64::EPSILON, + InternalDiff::Counter(c) => *c == 0.0, InternalDiff::Unknown => true, } } @@ -493,8 +494,9 @@ impl Diff { Diff::Text(t) => t.is_empty(), Diff::Map(m) => m.updated.is_empty(), Diff::Tree(t) => t.diff.is_empty(), + // Same rule as `InternalDiff`: only an exact 0 is an empty counter diff. #[cfg(feature = "counter")] - Diff::Counter(c) => c.abs() < f64::EPSILON, + Diff::Counter(c) => *c == 0.0, Diff::Unknown => true, } } diff --git a/crates/loro-internal/src/loro.rs b/crates/loro-internal/src/loro.rs index 917494531..764e3f639 100644 --- a/crates/loro-internal/src/loro.rs +++ b/crates/loro-internal/src/loro.rs @@ -612,7 +612,23 @@ impl LoroDoc { bytes: &[u8], origin: InternalString, ) -> Result { - self.with_barrier(|| self._import_with(bytes, origin)) + self.import_with_history_mode(bytes, origin, false) + } + + /// Import binary bytes. `json_lossy` accepts only the equivalences a JSON hop + /// can introduce; the default binary path passes `false`. + pub fn import_with_history_mode( + &self, + bytes: &[u8], + origin: InternalString, + json_lossy: bool, + ) -> Result { + let values = if json_lossy { + ImportedValues::JsonLossy + } else { + ImportedValues::Exact + }; + self.with_barrier(|| self._import_with(bytes, origin, values)) } #[tracing::instrument(skip_all)] @@ -620,6 +636,7 @@ impl LoroDoc { &self, bytes: &[u8], origin: InternalString, + values: ImportedValues, ) -> Result { ensure_cov::notify_cov("loro_internal::import"); let parsed = parse_header_and_body(bytes, true)?; @@ -637,7 +654,7 @@ impl LoroDoc { ); let _e = s.enter(); self.update_oplog_and_apply_delta_to_state_if_needed( - |oplog| oplog.decode(parsed), + |oplog| oplog.decode(parsed, values), origin, ) } @@ -647,7 +664,7 @@ impl LoroDoc { decode_snapshot(self, parsed.mode, parsed.body, origin) } else { self.update_oplog_and_apply_delta_to_state_if_needed( - |oplog| oplog.decode(parsed), + |oplog| oplog.decode(parsed, values), origin, ) } @@ -659,8 +676,8 @@ impl LoroDoc { decode_snapshot(self, parsed.mode, parsed.body, origin) } else { self.import_changes_and_apply_delta_to_state_if_needed( - |oplog| encoding::decode_oplog_changes(oplog, parsed), - ImportedValues::Exact, + |oplog| encoding::decode_oplog_changes(oplog, parsed, values), + values, origin, ) @@ -671,8 +688,8 @@ impl LoroDoc { } } EncodeMode::FastUpdates => self.import_changes_and_apply_delta_to_state_if_needed( - |oplog| encoding::decode_oplog_changes(oplog, parsed), - ImportedValues::Exact, + |oplog| encoding::decode_oplog_changes(oplog, parsed, values), + values, origin, ), EncodeMode::Auto => { @@ -1018,7 +1035,7 @@ impl LoroDoc { self.with_barrier(|| { let result = self.import_changes_and_apply_delta_to_state_if_needed( |oplog| crate::encoding::json_schema::decode_json_changes(json, &oplog.arena), - ImportedValues::Lossy, + ImportedValues::JsonLossy, Default::default(), ); self.emit_events(); @@ -2333,12 +2350,27 @@ impl LoroDoc { // PERF: opt #[tracing::instrument(skip_all)] pub fn import_batch(&self, bytes: &[Vec]) -> LoroResult { + self.import_batch_with_history_mode(bytes, false) + } + + /// Like [`Self::import_batch`]. `json_lossy` is the same opt-in as + /// [`Self::import_with_history_mode`]. + pub fn import_batch_with_history_mode( + &self, + bytes: &[Vec], + json_lossy: bool, + ) -> LoroResult { + let values = if json_lossy { + ImportedValues::JsonLossy + } else { + ImportedValues::Exact + }; if bytes.is_empty() { return Ok(ImportStatus::default()); } if bytes.len() == 1 { - return self.import(&bytes[0]); + return self.import_with_history_mode(&bytes[0], Default::default(), json_lossy); } let mut success = VersionRange::default(); @@ -2416,7 +2448,7 @@ impl LoroDoc { for (_meta, data) in meta_arr { #[cfg(test)] take_panic_at_batch_import_blob_for_test(); - match guard.doc._import_with(data, Default::default()) { + match guard.doc._import_with(data, Default::default(), values) { Ok(s) => { for (peer, (start, end)) in s.success.iter() { match success.0.entry(*peer) { diff --git a/crates/loro-internal/src/oplog.rs b/crates/loro-internal/src/oplog.rs index 5e58fd875..0f3064517 100644 --- a/crates/loro-internal/src/oplog.rs +++ b/crates/loro-internal/src/oplog.rs @@ -824,8 +824,12 @@ impl OpLog { } #[inline(always)] - pub(crate) fn decode(&mut self, data: ParsedHeaderAndBody) -> Result { - decode_oplog(self, data) + pub(crate) fn decode( + &mut self, + data: ParsedHeaderAndBody, + values: ImportedValues, + ) -> Result { + decode_oplog(self, data, values) } /// Containers that have at least one op inside `spans`. diff --git a/crates/loro-internal/src/oplog/change_store.rs b/crates/loro-internal/src/oplog/change_store.rs index cbf6ffcf5..f76ea5ced 100644 --- a/crates/loro-internal/src/oplog/change_store.rs +++ b/crates/loro-internal/src/oplog/change_store.rs @@ -488,6 +488,7 @@ impl ChangeStore { pub(crate) fn decode_snapshot_for_updates( bytes: Bytes, oplog: &crate::OpLog, + values: super::ImportedValues, ) -> Result, LoroError> { let arena = SharedArena::new(); let store = ChangeStore::new_mem(&arena, Arc::new(AtomicI64::new(0))); @@ -521,11 +522,7 @@ impl ChangeStore { drop(inner); drop(external); changes.sort_unstable_by_key(|c| c.lamport); - let changes = oplog.check_and_trim_known_part_in_arena( - changes, - super::ImportedValues::Exact, - &arena, - )?; + let changes = oplog.check_and_trim_known_part_in_arena(changes, values, &arena)?; Ok(changes .into_iter() .map(|mut change| { diff --git a/crates/loro-internal/src/oplog/known_history.rs b/crates/loro-internal/src/oplog/known_history.rs index 583fee19b..9e71c3955 100644 --- a/crates/loro-internal/src/oplog/known_history.rs +++ b/crates/loro-internal/src/oplog/known_history.rs @@ -7,7 +7,7 @@ //! the doc) or silently scrambled it (loro-dev/loro#1118). See //! `context/import-peer-id-reuse.md`. -use loro_common::{Counter, HasCounterSpan, LoroError, LoroResult, PeerID, ID}; +use loro_common::{Counter, HasCounterSpan, LoroError, LoroResult, LoroValue, PeerID, ID}; use rle::{HasLength, Sliceable}; use rustc_hash::FxHashMap; @@ -23,15 +23,69 @@ use crate::{ #[cfg(feature = "counter")] use crate::op::FutureInnerContent; -/// How faithfully the import format carries op values. +/// How to compare imported op values with known history. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum ImportedValues { - /// Binary encodings round-trip every value. + /// Structural equality. Default for binary import. Exact, - /// JSON updates may have gone through JSON text, which does not round-trip - /// every value (`NaN` becomes `null`, binary may come back as a list). Values - /// are then not compared; everything else still is. - Lossy, + /// Equivalences JSON export/import is known to introduce. Callers opt in. + JsonLossy, +} + +impl ImportedValues { + fn eq(self, a: &LoroValue, b: &LoroValue) -> bool { + if self == Self::Exact { + return a == b; + } + + match (a, b) { + (LoroValue::I64(a), LoroValue::I64(b)) => { + // JS numbers can round distinct large integers to the same f64. + // Keep exact comparison in the range where every integer fits. + const MAX_EXACT_INTEGER: i64 = 1 << 53; + a == b + || ((a.unsigned_abs() > MAX_EXACT_INTEGER as u64 + || b.unsigned_abs() > MAX_EXACT_INTEGER as u64) + && *a as f64 == *b as f64) + } + (LoroValue::Double(d), LoroValue::I64(i)) + | (LoroValue::I64(i), LoroValue::Double(d)) => { + // Inside +/-2^53 this is exact numeric equality; outside it also + // accepts the rounded Double produced by a JS-number relay. + *d == *i as f64 + } + (LoroValue::String(string), LoroValue::Container(_)) + | (LoroValue::Container(_), LoroValue::String(string)) => { + // A marker-looking string also becomes a container reference. + // JSON peer compression can reinterpret its peer as an index, + // so the decoded id cannot safely be compared with that string. + loro_common::ContainerID::try_from_loro_value_string(string).is_some() + } + (LoroValue::Binary(bytes), LoroValue::List(list)) + | (LoroValue::List(list), LoroValue::Binary(bytes)) => { + bytes.len() == list.len() + && bytes + .iter() + .zip(list.iter()) + .all(|(&byte, value)| match value { + LoroValue::I64(number) => *number == i64::from(byte), + LoroValue::Double(number) => *number == f64::from(byte), + _ => false, + }) + } + (LoroValue::Double(number), LoroValue::Null) + | (LoroValue::Null, LoroValue::Double(number)) => !number.is_finite(), + (LoroValue::List(a), LoroValue::List(b)) => { + a.len() == b.len() && a.iter().zip(b.iter()).all(|(a, b)| self.eq(a, b)) + } + (LoroValue::Map(a), LoroValue::Map(b)) => { + a.len() == b.len() + && a.iter() + .all(|(key, a)| b.get(key).is_some_and(|b| self.eq(a, b))) + } + _ => a == b, + } + } } impl OpLog { @@ -409,20 +463,23 @@ fn op_eq( return false; } - let exact = values == ImportedValues::Exact; - match (&a.content, &b.content) { - (InnerContent::List(a), InnerContent::List(b)) => list_op_eq(a_arena, b_arena, exact, a, b), + (InnerContent::List(a), InnerContent::List(b)) => { + list_op_eq(a_arena, b_arena, values, a, b) + } (InnerContent::Map(a), InnerContent::Map(b)) => { a.key == b.key - && a.value.is_some() == b.value.is_some() - && (!exact || a.value == b.value) + && match (&a.value, &b.value) { + (Some(a), Some(b)) => values.eq(a, b), + (None, None) => true, + _ => false, + } } (InnerContent::Tree(a), InnerContent::Tree(b)) => a == b, (InnerContent::Future(a), InnerContent::Future(b)) => match (a, b) { #[cfg(feature = "counter")] (FutureInnerContent::Counter(a), FutureInnerContent::Counter(b)) => { - !exact || a == b || (a.is_nan() && b.is_nan()) + a == b || (a.is_nan() && b.is_nan()) } ( crate::op::FutureInnerContent::Unknown { @@ -433,7 +490,16 @@ fn op_eq( prop: b_prop, value: b_value, }, - ) => a_prop == b_prop && (!exact || a_value == b_value), + ) => { + // A newer container's payload may contain LoroValues or arena + // indices whose meaning this version cannot interpret. JSON + // imports skip that payload; binary imports compare it. + a_prop == b_prop + && match values { + ImportedValues::JsonLossy => true, + ImportedValues::Exact => a_value == b_value, + } + } #[allow(unreachable_patterns)] _ => false, }, @@ -441,10 +507,33 @@ fn op_eq( } } +fn value_ranges_eq( + a_arena: &SharedArena, + b_arena: &SharedArena, + values: ImportedValues, + a: std::ops::Range, + b: std::ops::Range, +) -> bool { + let eq = |left: &LoroValue, right: &LoroValue| values.eq(left, right); + if std::ptr::eq(a_arena, b_arena) { + a_arena.value_slices_eq(a, b, eq) + } else { + a_arena.with_values(a, |a_values| { + b_arena.with_values(b, |b_values| { + a_values.len() == b_values.len() + && a_values + .iter() + .zip(b_values.iter()) + .all(|(left, right)| eq(left, right)) + }) + }) + } +} + fn list_op_eq( a_arena: &SharedArena, b_arena: &SharedArena, - exact: bool, + values: ImportedValues, a: &InnerListOp, b: &InnerListOp, ) -> bool { @@ -460,14 +549,13 @@ fn list_op_eq( }, ) => { a_pos == b_pos - && (!exact - || if std::ptr::eq(a_arena, b_arena) { - a_arena.value_slices_eq(a_slice.to_range(), b_slice.to_range()) - } else { - a_arena.with_values(a_slice.to_range(), |a| { - b_arena.with_values(b_slice.to_range(), |b| a == b) - }) - }) + && value_ranges_eq( + a_arena, + b_arena, + values, + a_slice.to_range(), + b_slice.to_range(), + ) } ( InnerListOp::InsertText { @@ -512,7 +600,7 @@ fn list_op_eq( elem_id: b_elem, value: b_value, }, - ) => a_elem == b_elem && (!exact || a_value == b_value), + ) => a_elem == b_elem && values.eq(a_value, b_value), ( InnerListOp::StyleStart { start: a_start, @@ -532,7 +620,7 @@ fn list_op_eq( a_start == b_start && a_end == b_end && a_key == b_key - && (!exact || a_value == b_value) + && values.eq(a_value, b_value) && a_info == b_info } (InnerListOp::StyleEnd, InnerListOp::StyleEnd) => true, @@ -541,7 +629,300 @@ fn list_op_eq( } #[cfg(test)] -mod tests { +mod json_lossy_value_tests { + use super::ImportedValues; + use loro_common::LoroValue; + + fn assert_lossy_pair(a: &LoroValue, b: &LoroValue, equal: bool) { + assert_eq!(ImportedValues::JsonLossy.eq(a, b), equal, "{a:?} vs {b:?}"); + assert_eq!(ImportedValues::JsonLossy.eq(b, a), equal, "{b:?} vs {a:?}"); + } + + #[test] + fn byte_lists_must_match_every_byte() { + let binary = LoroValue::from(vec![0u8, 1, 255]); + for list in [ + LoroValue::from(vec![0, 1, 255]), + LoroValue::from(vec![0.0, 1.0, 255.0]), + ] { + assert_lossy_pair(&binary, &list, true); + assert!(!ImportedValues::Exact.eq(&binary, &list)); + } + assert_lossy_pair( + &LoroValue::from(Vec::::new()), + &LoroValue::from(Vec::::new()), + true, + ); + for list in [ + LoroValue::from(vec![0, 1]), + LoroValue::from(vec![0, 1, 255, 0]), + LoroValue::from(vec![0, 1, 254]), + LoroValue::from(vec![0, -1, 255]), + LoroValue::from(vec![0, 1, 256]), + LoroValue::from(vec![0.0, 1.5, 255.0]), + LoroValue::from(vec![0.0, f64::NAN, 255.0]), + LoroValue::from(vec![ + LoroValue::I64(0), + LoroValue::Bool(true), + LoroValue::I64(255), + ]), + ] { + assert_lossy_pair(&binary, &list, false); + } + assert_lossy_pair(&binary, &LoroValue::from(vec![0u8, 1, 254]), false); + } + + #[test] + fn only_non_finite_numbers_match_null() { + for number in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] { + let number = LoroValue::Double(number); + assert_lossy_pair(&number, &LoroValue::Null, true); + assert!(!ImportedValues::Exact.eq(&number, &LoroValue::Null)); + assert!(ImportedValues::Exact.eq(&number, &number)); + } + for number in [0.0, -0.0, 1.5, f64::MAX] { + assert_lossy_pair(&LoroValue::Double(number), &LoroValue::Null, false); + } + assert_lossy_pair(&LoroValue::I64(0), &LoroValue::Null, false); + assert_lossy_pair( + &LoroValue::Double(f64::INFINITY), + &LoroValue::Double(f64::NEG_INFINITY), + false, + ); + } + + #[test] + fn nested_values_allow_only_json_representations() { + let nested = + |value| LoroValue::Map(vec![("key".into(), LoroValue::from(vec![value]))].into()); + let original = nested(LoroValue::from(vec![ + LoroValue::from(vec![1u8, 2]), + LoroValue::Double(f64::NAN), + ])); + let json: LoroValue = + serde_json::from_str(&serde_json::to_string(&original).unwrap()).unwrap(); + assert_lossy_pair(&original, &json, true); + assert!(!ImportedValues::Exact.eq(&original, &json)); + assert_lossy_pair( + &original, + &nested(LoroValue::from(vec![ + LoroValue::from(vec![1, 3]), + LoroValue::Null, + ])), + false, + ); + assert_lossy_pair( + &original, + &LoroValue::Map(vec![("other".into(), original["key"].clone())].into()), + false, + ); + assert_lossy_pair(&original, &LoroValue::Map(Default::default()), false); + assert_lossy_pair( + &LoroValue::from(vec![1, 2]), + &LoroValue::from(vec![1, 2, 3]), + false, + ); + } + + #[test] + fn finite_doubles_one_ulp_apart_stay_different() { + let a = 1.0_f64; + let b = f64::from_bits(a.to_bits() + 1); + let (a, b) = (LoroValue::Double(a), LoroValue::Double(b)); + assert_lossy_pair(&a, &b, false); + assert!(!ImportedValues::Exact.eq(&a, &b)); + } + + #[test] + fn ordinary_values_still_require_exact_equality() { + let values = [ + LoroValue::Null, + LoroValue::Bool(false), + LoroValue::Bool(true), + LoroValue::I64(1), + LoroValue::I64(2), + LoroValue::Double(1.5), + LoroValue::from("one"), + LoroValue::from("two"), + LoroValue::from(loro_common::ContainerID::new_root( + "a", + loro_common::ContainerType::Map, + )), + LoroValue::from(loro_common::ContainerID::new_root( + "b", + loro_common::ContainerType::Map, + )), + ]; + for a in &values { + for b in &values { + assert_lossy_pair(a, b, a == b); + } + } + } + + #[test] + fn doubles_match_the_integer_as_f64() { + for integer in [0, 2, -2, 9_007_199_254_740_991, i64::MIN, i64::MAX] { + let double = LoroValue::Double(integer as f64); + let integer = LoroValue::I64(integer); + assert_lossy_pair(&double, &integer, true); + assert!(!ImportedValues::Exact.eq(&double, &integer)); + } + assert_lossy_pair(&LoroValue::Double(-0.0), &LoroValue::I64(0), true); + for number in [2.5, 3.0, f64::NAN, f64::INFINITY] { + assert_lossy_pair(&LoroValue::Double(number), &LoroValue::I64(2), false); + } + } + + #[test] + fn large_integers_match_only_the_same_js_number() { + const BOUND: i64 = 1 << 53; + const LARGE: i64 = 1 << 60; + for (original, rounded) in [ + (BOUND + 1, BOUND), + (-BOUND - 1, -BOUND), + (LARGE + 1, LARGE), + (LARGE + 1, LARGE + 24), // JSON.stringify spells 2^60 as ...7000. + (-LARGE - 1, -LARGE), + (-LARGE - 1, -LARGE - 24), + (i64::MAX, i64::MAX - 1), + (i64::MIN + 1, i64::MIN), + ] { + let original = LoroValue::I64(original); + for rounded in [LoroValue::I64(rounded), LoroValue::Double(rounded as f64)] { + assert_lossy_pair(&original, &rounded, true); + assert!(!ImportedValues::Exact.eq(&original, &rounded)); + } + } + for (a, b) in [ + (BOUND - 1, BOUND), + (-BOUND + 1, -BOUND), + (BOUND, BOUND + 2), + (-BOUND, -BOUND - 2), + (LARGE, LARGE + 256), + (-LARGE, -LARGE - 256), + (i64::MAX, i64::MAX - 1024), + (i64::MIN, i64::MIN + 1024), + ] { + assert_lossy_pair(&LoroValue::I64(a), &LoroValue::I64(b), false); + assert_lossy_pair(&LoroValue::I64(a), &LoroValue::Double(b as f64), false); + } + } + + #[test] + fn container_marker_strings_are_ambiguous_after_peer_compression() { + use loro_common::{ContainerID, ContainerType}; + let id = ContainerID::new_root("target", ContainerType::Text); + let marker = LoroValue::from(id.to_loro_value_string()); + let container = LoroValue::Container(id.clone()); + assert_lossy_pair(&marker, &container, true); + assert!(!ImportedValues::Exact.eq(&marker, &container)); + for string in ["plain text", "🦜:invalid"] { + assert_lossy_pair(&LoroValue::from(string), &container, false); + } + assert_lossy_pair( + &LoroValue::from("🦜:cid:0@0:Text"), + &LoroValue::Container(ContainerID::new_normal( + loro_common::ID::new(1, 0), + ContainerType::Text, + )), + true, + ); + assert_lossy_pair(&marker, &LoroValue::from("🦜:cid:root-other:Text"), false); + assert_lossy_pair( + &container, + &LoroValue::Container(ContainerID::new_root("other", ContainerType::Text)), + false, + ); + } + + #[test] + fn unknown_payload_bypass_still_compares_props_and_containers() { + use crate::{ + arena::SharedArena, + encoding::OwnedValue, + op::{FutureInnerContent, InnerContent, Op}, + }; + use loro_common::{ContainerID, ContainerType}; + let arena = SharedArena::new(); + let container = + arena.register_container(&ContainerID::new_root("future", ContainerType::Unknown(9))); + let make_op = |prop, value| Op { + counter: 0, + container, + content: InnerContent::Future(FutureInnerContent::Unknown { + prop, + value: Box::new(value), + }), + }; + let a = make_op(1, OwnedValue::LoroValue(LoroValue::Double(2.0))); + let b = make_op(1, OwnedValue::LoroValue(LoroValue::I64(2))); + assert!(super::op_eq( + &arena, + &arena, + ImportedValues::JsonLossy, + &a, + &b + )); + assert!(!super::op_eq(&arena, &arena, ImportedValues::Exact, &a, &b)); + let different_prop = make_op(2, OwnedValue::LoroValue(LoroValue::I64(2))); + assert!(!super::op_eq( + &arena, + &arena, + ImportedValues::JsonLossy, + &a, + &different_prop + )); + let mut different_container = b; + different_container.container = + arena.register_container(&ContainerID::new_root("other", ContainerType::Unknown(9))); + assert!(!super::op_eq( + &arena, + &arena, + ImportedValues::JsonLossy, + &a, + &different_container + )); + } + + #[cfg(feature = "counter")] + #[test] + fn counter_values_keep_numeric_equality_in_lossy_mode() { + use crate::{ + arena::SharedArena, + op::{FutureInnerContent, InnerContent, Op}, + }; + use loro_common::{ContainerID, ContainerType}; + let arena = SharedArena::new(); + let container = + arena.register_container(&ContainerID::new_root("counter", ContainerType::Counter)); + let make_op = |value| Op { + counter: 0, + container, + content: InnerContent::Future(FutureInnerContent::Counter(value)), + }; + for (a, b, equal) in [ + (2.0, 2.0, true), + (2.0, 3.0, false), + (-0.0, 0.0, true), + (f64::NAN, f64::NAN, true), + (f64::INFINITY, f64::INFINITY, true), + (f64::NAN, f64::INFINITY, false), + (1e-17, 0.0, false), + (-1e-17, 0.0, false), + ] { + for values in [ImportedValues::JsonLossy, ImportedValues::Exact] { + assert_eq!( + super::op_eq(&arena, &arena, values, &make_op(a), &make_op(b)), + equal + ); + } + } + } +} + +#[cfg(test)] +mod two_arena_tests { use super::*; use crate::{ container::list::list_op::ListOp, diff --git a/crates/loro/src/lib.rs b/crates/loro/src/lib.rs index 7d7f57355..ecb87b72b 100644 --- a/crates/loro/src/lib.rs +++ b/crates/loro/src/lib.rs @@ -91,6 +91,23 @@ mod counter; #[cfg(feature = "counter")] pub use counter::LoroCounter; +/// How overlapping op ids are compared during import. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +#[non_exhaustive] +pub enum ImportHistoryMode { + /// Reject op-id reuse unless the overlapping ops are identical. + #[default] + Exact, + /// Accept only the equivalences JSON export/import is known to introduce: + /// I64/I64 or I64/Double that match as f64 when at least one integer is outside ±2^53; + /// Binary vs a same-length list of I64/Double bytes; + /// non-finite Double vs Null; + /// a string that parses as a Loro container marker vs any Container (peer compression can rewrite the peer, so the id is not compared); + /// unknown payloads skipped. + /// Finite doubles still use ==. Distinct ops that merely look like these conversions are accepted; callers must opt in. + JsonLossy, +} + /// `LoroDoc` is the entry for the whole document. /// When it's dropped, all the associated [`Container`]s will be invalidated. /// @@ -431,6 +448,19 @@ impl LoroDoc { self.doc.import_batch(bytes) } + /// Like [`import_batch`], with the same history mode as [`import_with_history_mode`]. + /// + /// The existing [`import_batch`] signature stays exact. + #[inline] + pub fn import_batch_with_history_mode( + &self, + bytes: &[Vec], + mode: ImportHistoryMode, + ) -> LoroResult { + self.doc + .import_batch_with_history_mode(bytes, mode == ImportHistoryMode::JsonLossy) + } + /// Get a [Container] by container id. #[inline] pub fn get_container(&self, id: ContainerID) -> Option { @@ -722,11 +752,32 @@ impl LoroDoc { /// for telemetry or filtering. /// Pitfalls: /// - Same as [`import`]: verify `ImportStatus.pending` and fetch dependencies if needed. + /// - Known-history comparison is [`ImportHistoryMode::Exact`]. Bytes that already + /// passed through JSON need [`import_with_history_mode`]. #[inline] pub fn import_with(&self, bytes: &[u8], origin: &str) -> Result { self.doc.import_with(bytes, origin.into()) } + /// Import binary bytes, choosing how overlapping op ids are compared. + /// + /// [`import`] and [`import_with`] stay [`ImportHistoryMode::Exact`]. + /// [`ImportHistoryMode::JsonLossy`] is for bytes whose values may have been + /// degraded by an earlier JSON hop. [`import_json_updates`] already uses that mode. + #[inline] + pub fn import_with_history_mode( + &self, + bytes: &[u8], + origin: &str, + mode: ImportHistoryMode, + ) -> Result { + self.doc.import_with_history_mode( + bytes, + origin.into(), + mode == ImportHistoryMode::JsonLossy, + ) + } + /// Import the json schema updates. /// /// # Example diff --git a/crates/loro/tests/contracts/counter.rs b/crates/loro/tests/contracts/counter.rs index b428773b9..df8d2af50 100644 --- a/crates/loro/tests/contracts/counter.rs +++ b/crates/loro/tests/contracts/counter.rs @@ -6,6 +6,17 @@ use loro::{ }; use serde_json::json; +#[test] +fn tiny_counter_increment_round_trips_through_binary_updates() -> LoroResult<()> { + let a = LoroDoc::new(); + a.get_counter("c").increment(1e-17)?; + let bytes = a.export(ExportMode::all_updates())?; + let b = LoroDoc::new(); + b.import(&bytes)?; + assert_eq!(b.get_counter("c").get_value(), 1e-17); + Ok(()) +} + #[test] fn detached_counter_attaches_with_value_and_keeps_identity() -> LoroResult<()> { let doc = LoroDoc::new(); diff --git a/crates/loro/tests/import_json_relay.rs b/crates/loro/tests/import_json_relay.rs new file mode 100644 index 000000000..36ed7448b --- /dev/null +++ b/crates/loro/tests/import_json_relay.rs @@ -0,0 +1,609 @@ +//! History relayed through JSON keeps its lossy values even after binary export. +//! Re-importing those bytes requires `ImportHistoryMode::JsonLossy`. Default +//! binary import stays exact and does not replace the receiver's original values. + +use loro::{ + ContainerID, ContainerType, ExpandType, ExportMode, ImportHistoryMode, JsonFutureOp, + JsonOpContent, LoroDoc, LoroError, LoroValue, StyleConfig, ValueOrContainer, ID, +}; + +fn source() -> LoroDoc { + let doc = LoroDoc::new(); + doc.set_peer_id(1).unwrap(); + doc.config_default_text_style(Some(StyleConfig { + expand: ExpandType::After, + })); + doc +} + +fn js_json(doc: &LoroDoc) -> String { + js_json_numbers(doc, false) +} + +fn js_json_numbers(doc: &LoroDoc, large_as_double: bool) -> String { + let json = doc + .export_json_updates(&Default::default(), &doc.oplog_vv()) + .unwrap(); + let mut json = serde_json::to_value(&json).unwrap(); + // Model JS-number precision and integral-number normalization. Large I64s + // passed through JSON.parse lose bits. In loro-wasm/src/convert.rs, + // js_json_schema_to_loro_json_schema uses serde_wasm_bindgen: unsafe JS + // integers become Double. A JSON string can instead decode a rounded I64; + // use its exact integral representative here (JS's decimal spelling varies). + // The direct WASM schema exporter currently rejects unsafe I64s, so these + // tests model relays parsing native JSON, rather than that exporter. + fn canonicalize_numbers(value: &mut serde_json::Value, large_as_double: bool) { + match value { + serde_json::Value::Number(number) => { + const BOUND: i64 = 1 << 53; + let n = number.as_f64().unwrap(); + let large = number + .as_i64() + .is_some_and(|i| i.unsigned_abs() > BOUND as u64) + || (number.is_f64() && n.abs() >= BOUND as f64); + if large { + *value = if large_as_double || n >= i64::MAX as f64 || n < i64::MIN as f64 { + serde_json::Value::from(n) + } else { + serde_json::Value::from(n as i64) + }; + } else if number.is_f64() && n.fract() == 0.0 { + *value = serde_json::Value::from(n as i64); + } + } + serde_json::Value::Array(values) => values + .iter_mut() + .for_each(|value| canonicalize_numbers(value, large_as_double)), + serde_json::Value::Object(values) => values + .values_mut() + .for_each(|value| canonicalize_numbers(value, large_as_double)), + _ => {} + } + } + canonicalize_numbers(&mut json, large_as_double); + serde_json::to_string(&json).unwrap() +} + +fn json_copy(doc: &LoroDoc) -> LoroDoc { + json_copy_with(doc, js_json) +} + +fn json_copy_with(doc: &LoroDoc, to_json: impl Fn(&LoroDoc) -> String) -> LoroDoc { + let json = to_json(doc); + let copy = LoroDoc::new(); + copy.set_peer_id(2).unwrap(); + copy.import_json_updates(json.as_str()).unwrap(); + copy +} + +fn map_value(doc: &LoroDoc, key: &str) -> LoroValue { + match doc.get_map("m").get(key).unwrap() { + ValueOrContainer::Value(value) => value, + other => panic!("expected a value, got {other:?}"), + } +} + +fn style_delta(doc: &LoroDoc) -> Option> { + doc.get_deep_value() + .as_map() + .unwrap() + .contains_key("text") + .then(|| doc.get_text("text").to_delta()) +} + +fn assert_relay_keeps_prefix(a: &LoroDoc, check_loss: impl Fn(&LoroDoc)) { + assert_relay_keeps_prefix_with_json(a, js_json, check_loss) +} + +fn assert_relay_keeps_prefix_with_json( + a: &LoroDoc, + to_json: impl Fn(&LoroDoc) -> String, + check_loss: impl Fn(&LoroDoc), +) { + a.commit(); + let prefix = a.export(ExportMode::all_updates()).unwrap(); + let b = json_copy_with(a, &to_json); + check_loss(&b); + + // Full exports must keep working after later relay edits too. + for tail in 1..=2 { + b.get_map("relay").insert("extra", tail).unwrap(); + b.commit(); + for mode in [ + Some(ExportMode::all_updates()), + Some(ExportMode::Snapshot), + None, + ] { + let blob = mode.map(|mode| b.export(mode).unwrap()); + for batch in [false, true] { + if batch && blob.is_none() { + continue; + } + let c = LoroDoc::new(); + c.set_peer_id(3).unwrap(); + c.import(&prefix).unwrap(); + let expected = c.get_deep_value().into_map().unwrap(); + let expected_style = style_delta(&c); + let prefix_vv = c.oplog_vv(); + let expected_history = serde_json::to_string( + &c.export_json_updates_without_peer_compression( + &Default::default(), + &prefix_vv, + ) + .unwrap(), + ) + .unwrap(); + match &blob { + Some(blob) if batch => { + c.import_batch_with_history_mode( + &[blob.clone()], + ImportHistoryMode::JsonLossy, + ) + .unwrap(); + } + Some(blob) => { + c.import_with_history_mode(blob, "", ImportHistoryMode::JsonLossy) + .unwrap(); + } + None => { + c.import_json_updates(to_json(&b).as_str()).unwrap(); + } + } + + let actual = c.get_deep_value().into_map().unwrap(); + for (key, value) in expected.iter() { + assert_eq!(actual.get(key), Some(value), "prefix root {key}"); + } + assert_eq!(actual.len(), expected.len() + 1); + assert_eq!(actual["relay"]["extra"], LoroValue::from(tail)); + assert_eq!(style_delta(&c), expected_style); + assert_eq!( + serde_json::to_string( + &c.export_json_updates_without_peer_compression( + &Default::default(), + &prefix_vv, + ) + .unwrap(), + ) + .unwrap(), + expected_history + ); + assert_eq!(c.oplog_vv(), b.oplog_vv()); + } + } + } + + // Compare the same representations in the opposite direction: the receiver + // owns the JSON prefix and the binary sender owns the original values. + let c = json_copy_with(a, &to_json); + let expected = c.get_deep_value().into_map().unwrap(); + let expected_style = style_delta(&c); + let prefix_vv = c.oplog_vv(); + let expected_history = serde_json::to_string( + &c.export_json_updates_without_peer_compression(&Default::default(), &prefix_vv) + .unwrap(), + ) + .unwrap(); + a.get_map("relay").insert("extra", 3).unwrap(); + a.commit(); + for mode in [ + Some(ExportMode::all_updates()), + Some(ExportMode::Snapshot), + None, + ] { + let receiver = LoroDoc::new(); + receiver + .import(&c.export(ExportMode::all_updates()).unwrap()) + .unwrap(); + if let Some(mode) = mode { + receiver + .import_with_history_mode( + &a.export(mode).unwrap(), + "", + ImportHistoryMode::JsonLossy, + ) + .unwrap(); + } else { + receiver.import_json_updates(to_json(a).as_str()).unwrap(); + } + let actual = receiver.get_deep_value().into_map().unwrap(); + for (key, value) in expected.iter() { + assert_eq!(actual.get(key), Some(value), "JSON prefix root {key}"); + } + assert_eq!(style_delta(&receiver), expected_style); + assert_eq!( + serde_json::to_string( + &receiver + .export_json_updates_without_peer_compression(&Default::default(), &prefix_vv) + .unwrap(), + ) + .unwrap(), + expected_history + ); + assert_eq!(actual.len(), expected.len() + 1); + assert_eq!(actual["relay"]["extra"], LoroValue::I64(3)); + assert_eq!(receiver.oplog_vv(), a.oplog_vv()); + } +} + +#[test] +fn binary_map_value_survives_json_then_binary_relay() { + let a = source(); + a.get_map("m").insert("bin", vec![0u8, 1, 255]).unwrap(); + assert_relay_keeps_prefix(&a, |b| { + assert_eq!(map_value(b, "bin"), LoroValue::from(vec![0, 1, 255])); + }); +} + +#[test] +fn mergeable_text_survives_json_then_binary_relay() { + let a = source(); + a.get_map("m") + .ensure_mergeable_text("mt") + .unwrap() + .insert(0, "hi") + .unwrap(); + assert_relay_keeps_prefix(&a, |b| assert!(map_value(b, "mt").is_list())); +} + +#[test] +fn mergeable_counter_survives_json_then_binary_relay() { + let a = source(); + a.get_map("m") + .ensure_mergeable_counter("mc") + .unwrap() + .increment(2.0) + .unwrap(); + assert_relay_keeps_prefix(&a, |b| assert!(map_value(b, "mc").is_list())); +} + +#[test] +fn non_finite_map_values_survive_json_then_binary_relay() { + for value in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] { + let a = source(); + a.get_map("m").insert("number", value).unwrap(); + assert_relay_keeps_prefix(&a, |b| assert_eq!(map_value(b, "number"), LoroValue::Null)); + } +} + +#[test] +fn integral_double_survives_json_and_binary_relays() { + for number in [2.0, -2.0, -0.0] { + let a = source(); + a.get_map("m") + .insert("number", LoroValue::Double(number)) + .unwrap(); + assert_relay_keeps_prefix(&a, |b| { + assert_eq!(map_value(b, "number"), LoroValue::I64(number as i64)); + }); + } +} + +#[test] +fn integral_double_json_reimport_into_original_writer_succeeds() { + let a = source(); + a.get_map("m") + .insert("number", LoroValue::Double(2.0)) + .unwrap(); + a.commit(); + let b = json_copy(&a); + assert_eq!(map_value(&b, "number"), LoroValue::I64(2)); + b.get_map("relay").insert("extra", 1).unwrap(); + b.commit(); + a.import_json_updates(js_json(&b).as_str()).unwrap(); + assert_eq!(map_value(&a, "number"), LoroValue::Double(2.0)); + assert_eq!(a.get_map("relay").get_value()["extra"], LoroValue::I64(1)); + assert_eq!(a.oplog_vv(), b.oplog_vv()); +} + +#[test] +fn large_integer_survives_json_and_binary_relays_as_i64() { + for number in [ + (1i64 << 60) + 1, + -(1i64 << 60) - 1, + (1i64 << 53) + 1, + -(1i64 << 53) - 1, + ] { + let a = source(); + a.get_map("m").insert("number", number).unwrap(); + assert_relay_keeps_prefix(&a, |b| { + assert_eq!(map_value(b, "number"), LoroValue::I64(number as f64 as i64)); + assert_ne!(map_value(b, "number"), LoroValue::I64(number)); + }); + } +} + +#[test] +fn large_integer_survives_json_and_binary_relays_as_double() { + for number in [ + (1i64 << 60) + 1, + -(1i64 << 60) - 1, + (1i64 << 53) + 1, + -(1i64 << 53) - 1, + i64::MAX, + i64::MIN + 1, + ] { + let a = source(); + let nested = LoroValue::Map( + vec![( + "nested".into(), + LoroValue::from(vec![LoroValue::I64(number)]), + )] + .into(), + ); + a.get_map("m").insert("number", number).unwrap(); + a.get_map("m").insert("nested", nested.clone()).unwrap(); + a.get_list("list").push(nested.clone()).unwrap(); + a.get_text("text").insert(0, "styled").unwrap(); + a.get_text("text").mark(0..6, "custom", nested).unwrap(); + let expected_style = a.get_text("text").to_delta(); + assert_relay_keeps_prefix_with_json( + &a, + |doc| js_json_numbers(doc, true), + |b| { + assert_eq!(map_value(b, "number"), LoroValue::Double(number as f64)); + assert_eq!( + map_value(b, "nested")["nested"][0], + LoroValue::Double(number as f64) + ); + assert_ne!(b.get_text("text").to_delta(), expected_style); + }, + ); + } +} + +#[test] +fn large_integer_json_reimport_into_original_writer_succeeds() { + for large_as_double in [false, true] { + for number in [(1i64 << 60) + 1, -(1i64 << 60) - 1] { + let a = source(); + a.get_map("m").insert("number", number).unwrap(); + a.commit(); + let b = json_copy_with(&a, |doc| js_json_numbers(doc, large_as_double)); + assert_ne!(map_value(&b, "number"), LoroValue::I64(number)); + b.get_map("relay").insert("extra", 1).unwrap(); + b.commit(); + a.import_json_updates(js_json_numbers(&b, large_as_double).as_str()) + .unwrap(); + assert_eq!(map_value(&a, "number"), LoroValue::I64(number)); + assert_eq!(a.get_map("relay").get_value()["extra"], LoroValue::I64(1)); + assert_eq!(a.oplog_vv(), b.oplog_vv()); + } + } +} + +#[test] +fn container_marker_string_survives_json_and_binary_relays() { + let a = source(); + // The string uses compressed peer index 0, which the JSON importer maps to + // actual peer 1. Its counter matches the map op, so this is accepted JSON. + a.get_map("m").insert("cid", "🦜:cid:0@0:Text").unwrap(); + assert_relay_keeps_prefix(&a, |b| { + assert!(matches!( + b.get_map("m").get("cid"), + Some(ValueOrContainer::Container(_)) + )); + assert_eq!( + b.get_map("m").get_value()["cid"], + LoroValue::Container(ContainerID::new_normal(ID::new(1, 0), ContainerType::Text)) + ); + }); +} + +#[test] +fn tagged_counter_values_keep_float_semantics_through_json() { + let a = source(); + for increment in [2.0, -0.25, 0.5] { + a.get_counter("counter").increment(increment).unwrap(); + a.commit(); + } + assert_relay_keeps_prefix(&a, |b| { + assert_eq!(b.get_counter("counter").get_value(), 2.25) + }); +} + +#[test] +fn non_finite_counter_json_remains_invalid_before_history_comparison() { + for increment in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] { + let a = source(); + a.get_counter("counter").increment(increment).unwrap(); + a.commit(); + let b = source(); + let before = b.oplog_vv(); + assert!(matches!( + b.import_json_updates(js_json(&a).as_str()), + Err(LoroError::InvalidJsonSchema) + )); + assert_eq!(b.oplog_vv(), before); + } +} + +#[test] +fn unknown_op_payload_survives_json_and_binary_relays() { + let template = source(); + template.get_counter("unknown").increment(1.0).unwrap(); + template.commit(); + let mut json = template + .export_json_updates_without_peer_compression(&Default::default(), &template.oplog_vv()) + .unwrap(); + let op = &mut json.changes[0].ops[0]; + op.container = ContainerID::new_root("unknown", ContainerType::Unknown(9)); + let JsonOpContent::Future(future) = &mut op.content else { + unreachable!() + }; + // A newer container can carry arbitrary nested LoroValues in its opaque op. + // The typed schema preserves Double(2.0); a JS relay turns it into I64(2). + future.value = JsonFutureOp::Unknown( + serde_json::from_str(r#"{"value_type":"loro_value","value":{"nested":[2.0]}}"#).unwrap(), + ); + let a = source(); + a.import_json_updates(json).unwrap(); + let original = js_json(&a); + assert_relay_keeps_prefix(&a, |b| assert_eq!(js_json(b), original)); +} + +#[test] +fn style_values_survive_json_then_binary_relay() { + for value in [LoroValue::from(vec![0u8, 255]), LoroValue::Double(f64::NAN)] { + let a = source(); + let text = a.get_text("text"); + text.insert(0, "styled").unwrap(); + text.mark(0..6, "custom", value).unwrap(); + let expected = text.to_delta(); + assert_relay_keeps_prefix(&a, |b| assert_ne!(b.get_text("text").to_delta(), expected)); + } +} + +#[test] +fn nested_and_sequence_values_survive_json_then_binary_relay() { + let a = source(); + let value = LoroValue::Map( + vec![( + "nested".into(), + LoroValue::List( + vec![ + LoroValue::from(vec![1u8, 2]), + LoroValue::Double(f64::INFINITY), + LoroValue::Double(2.0), + ] + .into(), + ), + )] + .into(), + ); + a.get_map("m").insert("nested", value.clone()).unwrap(); + a.get_list("list").push(value.clone()).unwrap(); + let movable = a.get_movable_list("movable"); + movable.push(value.clone()).unwrap(); + movable.push(0).unwrap(); + movable.set(1, value.clone()).unwrap(); + a.get_text("text").insert(0, "nested style").unwrap(); + a.get_text("text") + .mark(0..6, "custom", value.clone()) + .unwrap(); + assert_relay_keeps_prefix(&a, |b| assert_ne!(map_value(b, "nested"), value)); +} + +#[test] +fn real_text_conflict_is_rejected_after_json_then_binary_relay() { + let a = source(); + a.get_text("text").insert(0, "hello").unwrap(); + a.commit(); + let conflict = source(); + conflict.get_text("text").insert(0, "hellO").unwrap(); + conflict.commit(); + let b = json_copy(&conflict); + b.get_map("relay").insert("extra", 1).unwrap(); + b.commit(); + + for mode in [ExportMode::all_updates(), ExportMode::Snapshot] { + let c = LoroDoc::new(); + c.import(&a.export(ExportMode::all_updates()).unwrap()) + .unwrap(); + let before = c.get_deep_value(); + let vv = c.oplog_vv(); + let err = c.import(&b.export(mode).unwrap()).unwrap_err(); + assert!(matches!(err, LoroError::UsedOpID { id } if id == ID::new(1, 4))); + assert_eq!(c.get_deep_value(), before); + assert_eq!(c.oplog_vv(), vv); + } +} + +#[test] +fn exact_import_rejects_large_integers_one_apart() { + let local = 1i64 << 60; + let remote = local + 1; + let a = source(); + a.get_map("m").insert("n", local).unwrap(); + a.commit(); + let b = source(); + b.get_map("m").insert("n", remote).unwrap(); + b.commit(); + b.get_map("m").insert("extra", 1).unwrap(); + b.commit(); + let bytes = b.export(ExportMode::all_updates()).unwrap(); + let before = a.get_deep_value(); + let vv = a.oplog_vv(); + let err = a.import(&bytes).unwrap_err(); + assert!(matches!(err, LoroError::UsedOpID { .. })); + assert_eq!(a.get_deep_value(), before); + assert_eq!(a.oplog_vv(), vv); + assert!(a.oplog_vv() != b.oplog_vv() || a.get_deep_value() == b.get_deep_value()); + + a.import_with_history_mode(&bytes, "", ImportHistoryMode::JsonLossy) + .unwrap(); + assert_eq!(a.oplog_vv(), b.oplog_vv()); + assert_eq!(map_value(&a, "n"), LoroValue::I64(local)); + assert_ne!(map_value(&a, "n"), map_value(&b, "n")); +} + +#[test] +fn genuine_value_conflicts_are_rejected_on_json_and_binary_imports() { + let nested = |byte| LoroValue::Map(vec![("nested".into(), LoroValue::from(vec![byte]))].into()); + for (local, conflicting) in [ + (LoroValue::from(vec![1u8, 2]), LoroValue::from(vec![1, 3])), + (LoroValue::Double(f64::NAN), LoroValue::I64(0)), + (LoroValue::Double(2.0), LoroValue::I64(3)), + (LoroValue::I64((1 << 53) - 1), LoroValue::I64(1 << 53)), + (LoroValue::I64(1 << 53), LoroValue::I64((1 << 53) + 2)), + (LoroValue::I64(1 << 60), LoroValue::I64((1 << 60) + 256)), + (LoroValue::I64(-(1 << 60)), LoroValue::I64(-(1 << 60) - 256)), + (nested(1), nested(2)), + ] { + // Check every operation that carries a value, including arena-backed inserts. + for kind in ["map", "list", "movable-set", "style"] { + let write = |doc: &LoroDoc, value: LoroValue| match kind { + "map" => doc.get_map("m").insert("value", value).unwrap(), + "list" => doc.get_list("list").push(value).unwrap(), + "movable-set" => { + doc.get_movable_list("movable").push(0).unwrap(); + doc.get_movable_list("movable").set(0, value).unwrap(); + } + "style" => { + doc.get_text("text").insert(0, "style").unwrap(); + doc.get_text("text").mark(0..5, "custom", value).unwrap(); + } + _ => unreachable!(), + }; + let a = source(); + write(&a, local.clone()); + a.commit(); + let conflict = source(); + write(&conflict, conflicting.clone()); + conflict.commit(); + let b = json_copy(&conflict); + b.get_map("relay").insert("extra", 1).unwrap(); + b.commit(); + let json = serde_json::to_string( + &b.export_json_updates(&Default::default(), &b.oplog_vv()) + .unwrap(), + ) + .unwrap(); + for import_kind in ["json", "updates", "snapshot"] { + let c = LoroDoc::new(); + c.import(&a.export(ExportMode::all_updates()).unwrap()) + .unwrap(); + let before = c.get_deep_value(); + let before_style = style_delta(&c); + let vv = c.oplog_vv(); + let err = match import_kind { + "json" => c.import_json_updates(json.as_str()).unwrap_err(), + "updates" => c + .import(&b.export(ExportMode::all_updates()).unwrap()) + .unwrap_err(), + "snapshot" => c + .import(&b.export(ExportMode::Snapshot).unwrap()) + .unwrap_err(), + _ => unreachable!(), + }; + assert!( + matches!(err, LoroError::UsedOpID { .. }), + "{kind}/{import_kind}: {err:?}" + ); + assert_eq!(c.get_deep_value(), before); + assert_eq!(style_delta(&c), before_style); + assert_eq!(c.oplog_vv(), vv); + } + } + } +} diff --git a/docs/encoding.md b/docs/encoding.md index 138b013df..e1f2cc060 100644 --- a/docs/encoding.md +++ b/docs/encoding.md @@ -907,9 +907,9 @@ the operation `len`. Length definitions: [`DeleteSpan::content_len`](../crates/loro-internal/src/container/list/list_op.rs#L124-L130). For a Counter delta `d`, the writer computes `a = abs(d)` and chooses tag 3 -I64 exactly when `a.fract() < f64::EPSILON && (a as i64) < (2 << 26)`; it then -writes `d as i64`. Otherwise it chooses tag 4 F64. This is a one-sided -fractional-part test, not a symmetric distance-to-nearest-integer test. +I64 exactly when `a.fract() == 0.0 && (a as i64) < (2 << 26)`; it then +writes `d as i64`. Otherwise it chooses tag 4 F64. A nonzero fraction such as +`1e-17` stays F64. This is an exact fractional-part test, not a tolerance. The decoder retains support for tag 13 `TreeMove`, but the mode-4 writer uses tag 16 `RawTreeMove`. In the current change-block decoder, tag 13 cannot be