Repository navigation
fix: accept JSON-relayed history in binary imports - #1175
Merged
Merged
Conversation
See context/import-peer-id-reuse.md for the JSON relay comparison rules. Co-Authored-By: GPT-6 <noreply@openai.com>
Contributor
WASM Size Report
|
See context/import-peer-id-reuse.md for comparison rules and ambiguous payload limits. Co-Authored-By: GPT-6 <noreply@openai.com>
See context/import-peer-id-reuse.md for numeric equivalence and WASM conversion limits. Co-Authored-By: GPT-6 <noreply@openai.com>
JSON relays were rejected as reused op ids, and treating every import as those lossy equivalences also accepted real same-id conflicts. Binary import stays structurally exact. JSON import and an explicit history mode opt into the conversions JSON introduces. Finite doubles round-trip because serde_json float_roundtrip is a normal dependency, and counter deltas are integer-encoded only when the fraction is exactly zero.
#1177 made apply_decoded_changes_to_oplog return Result. The outdated decode path still compares with the caller's ImportedValues and now propagates that error. JSON export call sites follow the new LoroResult.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix a 1.16.4 regression where full binary updates or snapshots from a JSON relay fail with
UsedOpIDat a receiver that already has the original prefix. For example, Rust'sDouble(2.0)can becomeI64(2)through JS/JSON; the receiver now accepts the relay's new operations while retaining its original Double.ImportedValues::Lossyfor binary and JSON imports. Accept Binary versus matching u8 lists, non-finite doubles versus Null, and Double versus I64 whend == i as f64, in either direction and recursively inside nested and rich-text style values. Also accept I64 versus I64 when both convert to the same f64 and at least one has|i| > 2^53; distinct integers inside[-2^53, 2^53]still require exact equality.ImportedValues::Exactnow exists only undercfg(test)as the strict test baseline.context/import-peer-id-reuse.md, and add aloro-crdtpatch changeset describing both the relay fix and the changed JSON conflict checks.Rationale
Binary encoding preserves the values it is given, but that history may already have passed through JSON. Treating every binary import as exact therefore rejects the representations produced by the existing JSON format. The comparison permits known JSON conversions and bypasses the ambiguous payload kinds described above.
Counter JSON operations carry tagged numeric values, both decoded to
Counter(f64), so finite increments retain their numeric meaning. Non-finite counter increments serialized as JSON text remain invalid at decode, as before.Large I64s in native JSON can lose precision when a relay parses them into JS numbers. For example,
2^60 + 1becomes the same f64 as2^60; JS's JSON text spells that number1152921504606847000, which can decode as a rounded I64. WASM's object schema importer usesserde_wasm_bindgen::Deserializer::deserialize_any, which reads unsafe JS numbers as Double. The genericjs_value_to_loro_valuealso returns Double above2^53, and I64 at that boundary. Both rounded representations now pass the overlap check. The direct WASM schema exporter currently rejects unsafe I64s throughserde_wasm_bindgen::Serializer::serialize_i64; this PR leaves that existing export limitation unchanged.The JSON format is unchanged. Binary values still come back from JSON as lists, including mergeable container markers; non-finite doubles become Null; integral Doubles can become I64; large integers can round through JS; and marker-looking strings can become Containers. This fixes the relay's sync failure without restoring lost values or removing existing differences between peers. JSON import behavior does change for genuine conflicts: representable payloads are now compared where main previously skipped all payload comparisons. Conversely, real same-id conflicts that resemble the allowed conversions, including different large integers with the same f64 representation, or use the bypassed payload kinds, can still pass and leave the receiver's own value intact. Both limits are documented in the context article and changeset.
Validation
Native regressions exercise actual JSON strings and explicitly model JS's integral-number normalization and large-integer rounding to I64 or Double. They cover both signs, the
2^53boundary, i64 extrema, nested/list/style values, both comparison directions, and repeated full updates/snapshot/batch/JSON imports. They assert that new operations arrive while the receiver retains its existing prefix values and history, and that genuine text/value conflicts remain rejected, including distinct safe integers and large numbers with different f64 representations. A Node probe verified the rounding and JSON decimal spelling; the WASM conversion behavior was checked against its source and the configured serde-wasm-bindgen 0.6.5 serializer/deserializer.cargo test -p loro --test import_json_relay --test import_reused_peer_id: 17 relay tests and 9 peer-id reuse tests passed, 0 failed. Includes integral Doubles and rounded large integers on JSON re-import into the original writer and JSON-to-binary relay paths, compressed Container-marker strings, nested unknown-op payloads, tagged counter increments and pre-existing rejection of non-finite counter JSON.cargo test -p loro-internal --lib known_history: 8 passed, 0 failed under default features. Counter-specific unit coverage also passed in the feature-enabled full suite.pnpm test: nextest 1770 passed, 37 skipped; doctests 62 passed, 0 failed.pnpm check: passed (cargo clippy --all-features -- -Dwarnings).git diff --check: passed.