Skip to content

fix(client): enforce sync flush timeout as one end-to-end budget (#3745) - #3751

Open
Anai-Guo wants to merge 1 commit into
milvus-io:masterfrom
Anai-Guo:fix/flush-timeout-budget
Open

fix(client): enforce sync flush timeout as one end-to-end budget (#3745)#3751
Anai-Guo wants to merge 1 commit into
milvus-io:masterfrom
Anai-Guo:fix/flush-timeout-budget

Conversation

@Anai-Guo

@Anai-Guo Anai-Guo commented Aug 7, 2026

Copy link
Copy Markdown

Problem

Fixes #3745.

The synchronous flush APIs don't enforce timeout as a single end-to-end budget. MilvusClient.flush(collection_name, timeout=N) delegates to GrpcHandler.flush:

  • the initial Flush RPC receives the full timeout;
  • _wait_for_flushed then starts a new timer and receives the full timeout again;
  • the multi-collection path restarts that timer for every collection;
  • the poll loop sleeps a fixed 0.5s (5s for flush_all) without capping to the remaining budget.

GrpcHandler.flush_all has the same split-budget behavior between the initial FlushAll RPC and _wait_for_flush_all. A deterministic mocked test with an 8s initial Flush RPC and flush(..., timeout=10) raised its timeout only after ~18.5s of simulated wall-clock.

The async handler is unaffected because its outer retry wrapper bounds the whole coroutine with asyncio.wait_for.

Fix

  • Compute one deadline when the public sync call starts.
  • Pass the remaining budget to the initial RPC and to every GetFlushState / GetFlushAllState poll.
  • Share the same deadline across all collection waits.
  • Cap each poll sleep to what is left of the budget.
  • Apply the same rule to flush_all.

Tests

New tests/unit/grpc_handler/test_flush_timeout.py:

  • test_flush_enforces_end_to_end_timeout_budget — 8s initial RPC + timeout=10; asserts the raise happens within the 10s budget (fails on master, which overshoots to ~18s).
  • test_flush_all_enforces_end_to_end_timeout_budget — same for flush_all.
  • test_flush_without_timeout_still_waits_until_flushedtimeout=None still polls until flushed and does not raise.

All three pass with the fix; the two budget tests fail on master. The existing test_utility.py flush tests continue to pass.

🤖 Generated with Claude Code

…vus-io#3745)

MilvusClient.flush/flush_all delegate to GrpcHandler.flush/flush_all,
which spent the caller's timeout twice: once on the initial Flush/FlushAll
RPC and then again as a fresh timer inside _wait_for_flushed /
_wait_for_flush_all. Each per-collection wait also restarted the timer,
and the poll loop slept a fixed 0.5s (5s for flush_all) without capping
to the remaining budget. A flush(timeout=10) whose initial RPC took 8s
could therefore raise only after ~18s.

Compute a single deadline when the public call starts, pass the remaining
budget to the initial RPC and every GetFlushState/GetFlushAllState poll,
share one deadline across all collection waits, and cap each poll sleep to
what is left. The async handler already bounds the whole coroutine with
asyncio.wait_for and is unaffected.

Signed-off-by: Tai An <antai12232931@anaiguo.com>
@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Anai-Guo
To complete the pull request process, please assign longjiquan after the PR has been reviewed.
You can assign the PR to them by writing /assign @longjiquan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify

mergify Bot commented Aug 7, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Sync flush timeout is not enforced as a single end-to-end budget

2 participants