enhance: use adaptive rate-limited retry strategy for S3 client - #590
enhance: use adaptive rate-limited retry strategy for S3 client#590bigsheeper wants to merge 1 commit into
Conversation
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: bigsheeper The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #590 +/- ##
==========================================
- Coverage 75.48% 75.48% -0.01%
==========================================
Files 164 164
Lines 15818 15813 -5
Branches 2388 2388
==========================================
- Hits 11941 11936 -5
Misses 3877 3877
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks for working on this. I suggest treating retry behavior as an object-storage-wide policy rather than an S3-only implementation detail:
“Consistent” should mean a common behavioral contract, not necessarily identical SDK classes or constants. Where a provider SDK cannot provide equivalent adaptive throttling, we should add a provider adapter or a common limiter above the SDK. Please also add provider-specific fault-injection tests for AWS S3, MinIO, GCS, Aliyun OSS, Tencent COS, Huawei OBS, and Azure to verify the full path from the real SDK error mapping to retry, backoff, and rate limiting. If this is too large for this PR, please clearly scope this PR as S3-compatible-only and track the Azure/native-SDK and credential-path work as explicit follow-ups. |
What & Why
The S3 client's default
ConnectRetryStrategyused a fixed 200ms retry interval (up to ~30 attempts) and had no client-side rate limiting. Under sustained S3 request-rate throttling (503 SlowDown) — e.g. a large bulk import writing many small objects to a cold bucket prefix — this hammers the throttled endpoint at a constant rate and amplifies the overload.This replaces it with a strategy built on the AWS SDK's
AdaptiveRetryStrategy, which adds:The
IsConnectErrorgate is preserved, so both AWS-standard retryable errors and MinIO-specific ones (SlowDown,XMinioServerNotInitialized) remain retriable.Notes
CompleteMultipartUpload, which goes through the SDK's retry machinery) are paced by the limiter.🤖 Generated with Claude Code