Repository navigation
Resilient calls between services: deadlines, budgeted retries, circuit breaker, gateway timeouts - #84
Open
NullButCute wants to merge 3 commits into
Open
NullButCute wants to merge 3 commits into
NullButCute wants to merge 3 commits into
Conversation
The stock check against Inventory (gRPC) had no deadline and no retry, and
the Product client's retries were unlimited by load. A restarting, overloaded
or silently hung Inventory made every order request wait as long as Inventory
did, and a failing dependency received every caller's retries on top of its
normal traffic.
Stock check (gRPC):
- Each attempt has a deadline (3 s). The deadline is sent to Inventory, which
cancels the database query as well. gRPC reports a client-side timeout as
Cancelled; that is translated back so it is recognised as a timeout and
counted by the circuit breaker.
- Connection failures (Unavailable) are retried up to twice with jittered
backoff. A slow or timed-out call is never retried: that only adds load to
a service that is already struggling.
- A circuit breaker opens when half of the recent calls failed (at least five
in 30 s) and lets one trial call through after 15 s.
- Everything that goes wrong surfaces as InventoryUnavailableException, and
the caller's own cancellation stays a cancellation.
- Connections use a 3 s connect timeout and HTTP/2 keep-alive pings so a
vanished peer is noticed in seconds.
Stock cache:
- When a cached stock level expires, concurrent orders for the same product
share one fetch instead of each calling Inventory. The fetch runs with its
own deadline; a caller who gives up only stops waiting.
- When Inventory is unavailable (timeout, open circuit) the stale copy
answers, as before, and only a product with no cached value at all is
rejected.
- The cache depends on a small IStockSource abstraction instead of the
generated gRPC types.
Retry budget (Shared):
- RetryBudget limits the extra load retries may add to a dependency: a share
of the requests seen in a sliding window (default 20 %), with a minimum so
a quiet service can still retry. Used for the stock check and for the
Product client, whose retries are now taken from the budget as well.
- shopeasy_downstream_retries_total{target,outcome} and
shopeasy_downstream_circuit_transitions_total{target,state} show retries
made, retries refused by the budget and circuit state changes.
Settings: Resilience:Inventory (TimeoutSeconds, MaxRetries,
BreakDurationSeconds) and Resilience:RetryBudget (Ratio,
MinRetriesPerWindow, WindowSeconds), validated at start-up, with the defaults
written into appsettings.json.
Tests cover the budget (ratio, minimum, sliding window, concurrency), the
policy against scripted failures (which failures are retried, budget
exhaustion, the circuit opening and closing, a hung call, caller
cancellation), the single-flight fetch, the cache layers including the stale
answer, and the Product client's retries. The gRPC behaviour was also
checked against a real gRPC server: healthy call, hung server, Unavailable
status, refused connection and fast failure while the circuit is open.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
A service that accepts a connection and never answers held the gateway's
connection (and the caller's) open without end, and a request that hit a
connection broken by a restarting service failed although repeating it would
have worked.
- A service that stays silent (no response headers, or no further bytes of
the request or response) for 30 s is cut off: the caller gets 504 Gateway
Timeout and the request to the service is cancelled. The timeout is YARP's
ActivityTimeout, applied to every cluster that does not set its own.
(ASP.NET's request-timeout middleware was tried first; in front of YARP it
answers 400 instead of 504.)
- Connecting is cut off after 5 s, and pooled connections are replaced every
two minutes so the gateway finds new replicas of a scaled service behind
one DNS name instead of keeping its first connections forever.
- GET and HEAD requests without a body are retried (twice, jittered
backoff) after a connection failure: the request never reached the
service, or the connection broke before an answer. POST, PUT, PATCH and
DELETE are never repeated, because the service may have processed them,
and a 5xx answer is passed through unchanged. Retries come from the shared
retry budget, so they cannot multiply the load on a failing service.
shopeasy_gateway_retries_total{outcome} counts them.
- There is deliberately no circuit breaker or passive health check: each
cluster has one destination, so there is nothing to fail over to.
Settings: Resilience:Gateway (ActivityTimeoutSeconds, ConnectTimeoutSeconds,
PooledConnectionLifetimeSeconds, GetRetries) and Resilience:RetryBudget,
validated at start-up, with the defaults in appsettings.json.
The tests put a real YARP gateway in front of fake services that misbehave
(breaking connections, refusing them, restarting, never answering): a GET over a
broken connection succeeds, a POST over the same connection is not repeated,
a 503 is passed through, a refused connection answers 502 quickly, a service
that restarts during the retries is reached again, a silent service ends in
504 and the request to it is cancelled. The retry handler is also tested
against scripted failures, together with the timeout filter and the settings
validation.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
…d the README
The new settings for calls between services are written down where an
operator looks for them, like the other resilience settings:
- docker-compose.yml: Resilience__Inventory__* (Order), Resilience__Gateway__*
(gateway) and Resilience__RetryBudget__* (both) as ${RESILIENCE_...:-default},
the shared budget through a YAML anchor.
- .env.example: the matching commented variables.
- k8s/configmaps/configmap.yaml: the same keys with the same values.
CallResilienceConfigurationTests fails when appsettings.json, Compose,
.env.example or the ConfigMap say something other than the code's defaults, or
when a service gets settings it does not use; the existing drift test counts
the new variables too.
README: a new section on resilient calls between services (deadlines,
retries, circuit breakers and the budget per call, stampede protection, the
deliberate choices and the settings), the interaction table, the pattern
cross-references, the configuration key reference, the test tables and
counts (408 tests, 355 without servers), the roadmap row and the known
limitations (no circuit breaker in the gateway by design, Redis calls
without their own timeouts).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A service that is restarting, overloaded or silently hung is harder on its callers than one that is simply down: every caller waits as long as the callee does, and the slowness spreads upstream. Before this change the Order service's gRPC stock check had no deadline and no retry, the Product client's retries were not limited by load, and the gateway had no timeout for a service that accepts a connection and never answers.
Order service
Unavailable), never after a slow or timed-out call; circuit breaker (half of the recent calls failing, at least 5 in 30 s, open for 15 s). A client-side timeout that gRPC reports asCancelledis translated so it is recognised as a timeout and counted by the breaker. Connect timeout and HTTP/2 keep-alive pings notice a vanished peer in seconds.shopeasy_downstream_retries_total{target,outcome},shopeasy_downstream_circuit_transitions_total{target,state}.Gateway
504 Gateway Timeout, and the request to the service is cancelled (YARPActivityTimeout; ASP.NET's request-timeout middleware answers 400 in front of YARP, so it is not used). Connect timeout 5 s; pooled connections replaced every 2 minutes so new replicas of a scaled service are found.GET/HEADwithout a body are retried (twice) after a connection failure, within the retry budget.POST/PUT/PATCH/DELETEare never repeated; a5xxanswer is passed through unchanged.shopeasy_gateway_retries_total{outcome}.Configuration
Resilience:Inventory,Resilience:GatewayandResilience:RetryBudgetare validated at start-up and written out inappsettings.json,docker-compose.yml(RESILIENCE_*variables),.env.exampleand the Kubernetes ConfigMap.CallResilienceConfigurationTestsfails if any copy differs from the code's defaults.Documentation
New README section on resilient calls between services (per-call table, retry budget, stampede protection, deliberate choices, settings), updated interaction table, configuration reference, test tables and counts, roadmap row and known limitations (including that Redis calls still have no timeouts of their own).
Testing
502in under a second, a silent service ends in504.Unavailablestatus (3 attempts), refused connection, fast failure while the circuit is open.Resilience:Inventory:TimeoutSeconds=0, both start and serve with the defaults; the gateway answers502immediately for an absent service.docker compose config, kustomize + kubeconform (31 resources valid) and the manifest rule check pass.Each of the three commits builds on its own.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
Generated by Claude Code