Skip to content

enhance: resolve global cluster topology via DNS SRV records - #3753

Open
huanghaoyuanhhy wants to merge 1 commit into
milvus-io:masterfrom
huanghaoyuanhhy:srv-topology-resolution
Open

enhance: resolve global cluster topology via DNS SRV records#3753
huanghaoyuanhhy wants to merge 1 commit into
milvus-io:masterfrom
huanghaoyuanhhy:srv-topology-resolution

Conversation

@huanghaoyuanhhy

Copy link
Copy Markdown
Contributor

Why

Global cluster connections previously fetched topology by hitting the global endpoint directly. This PR switches the SDK to discover ha-manager seed servers through the _grpc._tcp.<host> SRV record set, probe them concurrently, and use the topology with the highest version — with strict no-rollback semantics for every subsequent refresh path.

What changed

  • SRV discovery: resolve _grpc._tcp.<endpoint hostname> to the ha-manager seed list (typically 1-5 servers, ordered; priority/weight are not differentiated in deployment). Endpoints without SRV records raise instead of falling back to the legacy direct-endpoint path.
  • Seed selection & probing: probe at least two seeds concurrently when more than one is returned — uniformly at random when all weights are zero, weighted otherwise. The first fetch (no cached version) returns the first seed's answer immediately; the remaining seeds keep being polled in the background and trigger the topology replacement flow if a higher version arrives. Refreshes with a cached version poll all seeds and take the highest.
  • Version management (never roll back): memory topology only advances when the fetched version is strictly higher; failures keep the cached topology. Version comparisons now run against the connection strategy's shared topology (single source of truth) instead of a private refresher snapshot, so no path can regress the version.
  • Retry semantics: transient DNS failures and unreachable seeds retry with exponential backoff (3 attempts); deterministic rejections (no SRV records, server-side auth errors) fail fast without retry.
  • The endpoint hostname is passed verbatim via ?endpoint=; the server derives the global cluster id from its first DNS label, so the SDK does no gcid parsing.
  • Adds a dnspython dependency for SRV resolution.

Tests

  • Unit tests covering SRV resolution, candidate selection, concurrent probing (first-answer fast path, background higher-version replacement, wait-all highest-version), retry/backoff, deterministic-failure fast-fail, and refresher version guarding.
  • 193 passed

Deployment note

HA manager SRV records must be in place before this SDK version ships — endpoints without SRV records will raise instead of falling back to the legacy path.

@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: huanghaoyuanhhy
To complete the pull request process, please assign longjiquan after the PR has been reviewed.
You can assign the PR to them by writing /assign @longjiquan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify

mergify Bot commented Aug 10, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.06211% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.10%. Comparing base (08f7e10) to head (cee8d91).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
pymilvus/client/global_topology.py 89.80% 16 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3753      +/-   ##
==========================================
- Coverage   94.12%   94.10%   -0.02%     
==========================================
  Files          76       76              
  Lines       15821    16045     +224     
==========================================
+ Hits        14891    15099     +208     
- Misses        930      946      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Discover ha-manager seed servers through the _grpc._tcp.<host> SRV record
set instead of hitting the global endpoint directly, then concurrently
probe the nearest seeds (spanning priorities) and use the topology with
the highest version.

The topology request now carries the raw endpoint hostname verbatim via
the ?endpoint= query parameter; the server derives the global cluster id
from it (its first DNS label), so the SDK does no gcid parsing. Endpoints
without SRV records raise instead of falling back to the legacy beta path.

Adds a dnspython dependency for SRV resolution.

Signed-off-by: huanghaoyuanhhy <haoyuan.huang@zilliz.com>
@huanghaoyuanhhy
huanghaoyuanhhy force-pushed the srv-topology-resolution branch from bff88a1 to cee8d91 Compare August 10, 2026 08:25
@mergify mergify Bot added the ci-passed label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants