Repository navigation
Avoid full-space head requests for placeholder root counts - #806
Closed
Kymorphius wants to merge 1 commit into
Closed
Kymorphius wants to merge 1 commit into
Kymorphius wants to merge 1 commit into
Conversation
Treat a hash-only root Count=1 as unknown for large local ranges and retain normal recursive comparison. Validated package tests, race tests and original-algorithm regression overlay.
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
Author
|
recheck |
Member
|
Thank you so much for this, @Kymorphius! 🙏 Your investigation and the live numbers (71k heads / 9.2 MB down to 468 heads / 70 KB) pinned the problem down precisely, and pointing at Digging in, we found the placeholder count wasn't the whole story, so we fixed it at the source in #807 instead of special-casing
On a local network, a 50k-object space with 7 missing ids goes from 50,004 heads / 6.45 MB to 368 heads / 58 KB. Thanks again — this was a really valuable report. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a large local space differs from a node's cached root hash, ldiff can request every remote head even if only a few objects differ.
any-sync-node'stryNodeHeadSyncreturns the real space hash with a placeholderCount: 1for a single hash-only root query. The client currently treats that count as an accurate small range and follows it with a rootElements: truerequest. For a large space, this defeats recursive range comparison.This change treats that specific hash-only root summary as an unknown count when the local range exceeds the comparison threshold, and follows the normal child ranges instead. Equal hashes still terminate immediately; small and empty local spaces retain their existing behavior. Genuine one-element remote spaces still produce correct additions/removals. The wire format, hashes, and element comparison rules are unchanged.
An anonymized live comparison using any-sync v0.13.5 went from 71,292 returned heads / 9,196,793 protobuf response bytes to 468 heads / 70,491 bytes (99.23% fewer bytes) per differing-space comparison. This measures HeadSync response payloads, not total network traffic; the compatibility fix is applied here to current main and covered by deterministic tests.
Tests cover large spaces with seven new IDs, equal hashes, empty/small local spaces, the threshold boundary, a genuinely single-element remote space, and CompareDiff head ordering. Replacing only the changed algorithm file with the original main version via a Go overlay makes the new regression tests fail on whole-root element requests.
Validation:
go test ./app/ldiff ./commonspace/headsync ./commonspace/spacesyncprotogo test -race ./app/ldiff ./commonspace/headsync ./commonspace/spacesyncprotogofmtandgit diff --checkRepeated comparisons caused by cached empty object trees in anytype-heart are a separate recovery issue. This PR addresses discovery traffic only and does not change object-tree recovery.
Related object-recovery fix: anyproto/anytype-heart#3322