Skip to content

fix: accept JSON-relayed history in binary imports - #1175

Merged
zxch3n merged 5 commits into
mainfrom
fix/known-history-json-lossy
Oct 16, 2026
Merged

zxch3n merged 5 commits into
mainfrom
fix/known-history-json-lossy

Conversation

@zxch3n

@zxch3n zxch3n commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Summary

Fix a 1.16.4 regression where full binary updates or snapshots from a JSON relay fail with UsedOpID at a receiver that already has the original prefix. For example, Rust's Double(2.0) can become I64(2) through JS/JSON; the receiver now accepts the relay's new operations while retaining its original Double.

  • Reuse ImportedValues::Lossy for binary and JSON imports. Accept Binary versus matching u8 lists, non-finite doubles versus Null, and Double versus I64 when d == i as f64, in either direction and recursively inside nested and rich-text style values. Also accept I64 versus I64 when both convert to the same f64 and at least one has |i| > 2^53; distinct integers inside [-2^53, 2^53] still require exact equality.
  • Retain a value-comparison bypass for valid Container-marker strings versus Containers and for opaque unknown-op payloads. Marker strings can be reinterpreted through a compressed JSON peer table; this version cannot safely establish unknown container value semantics. Unknown-op props and op container ids still must match.
  • Compare other value payloads instead of main's blanket JSON value bypass. Keep text, op positions, op container ids, keys, element ids, style metadata and dependencies checked, and trim the known prefix so the receiver retains its own values and history. ImportedValues::Exact now exists only under cfg(test) as the strict test baseline.
  • Add native regression and rejection tests, update context/import-peer-id-reuse.md, and add a loro-crdt patch changeset describing both the relay fix and the changed JSON conflict checks.

Rationale

Binary encoding preserves the values it is given, but that history may already have passed through JSON. Treating every binary import as exact therefore rejects the representations produced by the existing JSON format. The comparison permits known JSON conversions and bypasses the ambiguous payload kinds described above.

Counter JSON operations carry tagged numeric values, both decoded to Counter(f64), so finite increments retain their numeric meaning. Non-finite counter increments serialized as JSON text remain invalid at decode, as before.

Large I64s in native JSON can lose precision when a relay parses them into JS numbers. For example, 2^60 + 1 becomes the same f64 as 2^60; JS's JSON text spells that number 1152921504606847000, which can decode as a rounded I64. WASM's object schema importer uses serde_wasm_bindgen::Deserializer::deserialize_any, which reads unsafe JS numbers as Double. The generic js_value_to_loro_value also returns Double above 2^53, and I64 at that boundary. Both rounded representations now pass the overlap check. The direct WASM schema exporter currently rejects unsafe I64s through serde_wasm_bindgen::Serializer::serialize_i64; this PR leaves that existing export limitation unchanged.

The JSON format is unchanged. Binary values still come back from JSON as lists, including mergeable container markers; non-finite doubles become Null; integral Doubles can become I64; large integers can round through JS; and marker-looking strings can become Containers. This fixes the relay's sync failure without restoring lost values or removing existing differences between peers. JSON import behavior does change for genuine conflicts: representable payloads are now compared where main previously skipped all payload comparisons. Conversely, real same-id conflicts that resemble the allowed conversions, including different large integers with the same f64 representation, or use the bypassed payload kinds, can still pass and leave the receiver's own value intact. Both limits are documented in the context article and changeset.

Validation

Native regressions exercise actual JSON strings and explicitly model JS's integral-number normalization and large-integer rounding to I64 or Double. They cover both signs, the 2^53 boundary, i64 extrema, nested/list/style values, both comparison directions, and repeated full updates/snapshot/batch/JSON imports. They assert that new operations arrive while the receiver retains its existing prefix values and history, and that genuine text/value conflicts remain rejected, including distinct safe integers and large numbers with different f64 representations. A Node probe verified the rounding and JSON decimal spelling; the WASM conversion behavior was checked against its source and the configured serde-wasm-bindgen 0.6.5 serializer/deserializer.

  • cargo test -p loro --test import_json_relay --test import_reused_peer_id: 17 relay tests and 9 peer-id reuse tests passed, 0 failed. Includes integral Doubles and rounded large integers on JSON re-import into the original writer and JSON-to-binary relay paths, compressed Container-marker strings, nested unknown-op payloads, tagged counter increments and pre-existing rejection of non-finite counter JSON.
  • cargo test -p loro-internal --lib known_history: 8 passed, 0 failed under default features. Counter-specific unit coverage also passed in the feature-enabled full suite.
  • pnpm test: nextest 1770 passed, 37 skipped; doctests 62 passed, 0 failed.
  • pnpm check: passed (cargo clippy --all-features -- -Dwarnings).
  • Rust formatting checks on changed files and git diff --check: passed.

See context/import-peer-id-reuse.md for the JSON relay comparison rules.

Co-Authored-By: GPT-6 <noreply@openai.com>
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

WASM Size Report

  • Original size: 3379.58 KB
  • Gzipped size: 1132.19 KB
  • Brotli size: 793.89 KB

zxch3n and others added 4 commits October 1, 2026 05:26
See context/import-peer-id-reuse.md for comparison rules and ambiguous payload limits.

Co-Authored-By: GPT-6 <noreply@openai.com>
See context/import-peer-id-reuse.md for numeric equivalence and WASM conversion limits.

Co-Authored-By: GPT-6 <noreply@openai.com>
JSON relays were rejected as reused op ids, and treating every import as
those lossy equivalences also accepted real same-id conflicts. Binary
import stays structurally exact. JSON import and an explicit history mode
opt into the conversions JSON introduces. Finite doubles round-trip because
serde_json float_roundtrip is a normal dependency, and counter deltas are
integer-encoded only when the fraction is exactly zero.
#1177 made apply_decoded_changes_to_oplog return Result. The outdated decode path still compares with the caller's ImportedValues and now propagates that error. JSON export call sites follow the new LoroResult.
@zxch3n
zxch3n merged commit 3827945 into main Oct 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant