Skip to content

Avoid full-space head requests for placeholder root counts - #806

Closed
Kymorphius wants to merge 1 commit into
anyproto:mainfrom
Kymorphius:fix/headsync-placeholder-count
Closed

Kymorphius wants to merge 1 commit into
anyproto:mainfrom
Kymorphius:fix/headsync-placeholder-count

Conversation

@Kymorphius

@Kymorphius Kymorphius commented Oct 11, 2026 •

Copy link
Copy Markdown

When a large local space differs from a node's cached root hash, ldiff can request every remote head even if only a few objects differ.

any-sync-node's tryNodeHeadSync returns the real space hash with a placeholder Count: 1 for a single hash-only root query. The client currently treats that count as an accurate small range and follows it with a root Elements: true request. For a large space, this defeats recursive range comparison.

This change treats that specific hash-only root summary as an unknown count when the local range exceeds the comparison threshold, and follows the normal child ranges instead. Equal hashes still terminate immediately; small and empty local spaces retain their existing behavior. Genuine one-element remote spaces still produce correct additions/removals. The wire format, hashes, and element comparison rules are unchanged.

An anonymized live comparison using any-sync v0.13.5 went from 71,292 returned heads / 9,196,793 protobuf response bytes to 468 heads / 70,491 bytes (99.23% fewer bytes) per differing-space comparison. This measures HeadSync response payloads, not total network traffic; the compatibility fix is applied here to current main and covered by deterministic tests.

Tests cover large spaces with seven new IDs, equal hashes, empty/small local spaces, the threshold boundary, a genuinely single-element remote space, and CompareDiff head ordering. Replacing only the changed algorithm file with the original main version via a Go overlay makes the new regression tests fail on whole-root element requests.

Validation:

  • go test ./app/ldiff ./commonspace/headsync ./commonspace/spacesyncproto
  • go test -race ./app/ldiff ./commonspace/headsync ./commonspace/spacesyncproto
  • gofmt and git diff --check

Repeated comparisons caused by cached empty object trees in anytype-heart are a separate recovery issue. This PR addresses discovery traffic only and does not change object-tree recovery.

Related object-recovery fix: anyproto/anytype-heart#3322

Treat a hash-only root Count=1 as unknown for large local ranges and retain normal recursive comparison.
Validated package tests, race tests and original-algorithm regression overlay.
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@Kymorphius

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@Kymorphius

Copy link
Copy Markdown
Author

recheck

@cheggaaa

Copy link
Copy Markdown
Member

Thank you so much for this, @Kymorphius! 🙏 Your investigation and the live numbers (71k heads / 9.2 MB down to 468 heads / 70 KB) pinned the problem down precisely, and pointing at tryNodeHeadSync was exactly the right thread to pull.

Digging in, we found the placeholder count wasn't the whole story, so we fixed it at the source in #807 instead of special-casing Count == 1 in ldiff:

  • ldiff counted every head update as a new element, so ranges split and never merged back, and two peers with identical content could end up with different root hashes. Headsync then kept re-comparing spaces that were already in sync, and each of those comparisons hit the full-fetch path you found.
  • A range below a leaf and an empty leaf both hashed to nil, so some differences were missed.
  • TryDiff now starts the walk below the root once DiffTypeCheck has seen different hashes, so the node's cached root answer is never read. Changing the count on the node side turned out to be unsafe for older clients.

On a local network, a 50k-object space with 7 missing ids goes from 50,004 heads / 6.45 MB to 368 heads / 58 KB.

Thanks again — this was a really valuable report.

@cheggaaa cheggaaa closed this Oct 13, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 13, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants