Repository navigation
Ride out database and network outages; bound database waits (step 9, part 1) - #80
Merged
Merged
Conversation
…e outages
Message consumers and database connections had no defence against a database
that is slow, stuck or briefly gone. Every consumer failure got the same three
quick retries (<15 s in total), and, because EF Core reports a database
failure as an InvalidOperationException ("An exception has been raised that is
likely due to a transient failure") and that type was on the "never retry"
list, a database outage was not retried at all: the message went to the
*_error queue on the first failure. A failure-injection test against the
previous policy reproduces exactly that.
New in ShopEasy.Shared/Resilience (not wired into the services yet):
* TransientFaults.IsTransient(exception) answers "can running this again
later succeed?". It walks the whole exception chain (EF and aggregate
wrappers no longer hide the cause) and says yes for dropped or refused
connections, timeouts, HTTP transport errors, deadlocks and lock timeouts,
and sessions ended by the server (restart, failover, administrator action,
idle_in_transaction_session_timeout 25P03), which Npgsql does not all flag
itself. It says no for what repeats identically: bugs, broken rules, unique
key violations, a missing file, a cancellation (shutdown).
* ConsumerResilience.UseConsumerResilience(options) replaces the duplicated
UseMessageRetry blocks with two stacked policies: an outer patient one for
transient faults only (12 retries, pauses doubling from 1 s to 60 s, about
7 minutes, +/-20 % jitter so replicas do not retry in step) and an inner
quick one for everything else (3 retries, 1-10 s; ArgumentNullException
and InvalidOperationException not retried). Transient faults bypass the
inner policy, so the two never multiply (exactly the configured number of
attempts is made). Settings: Resilience:Consumers:*; nonsensical values are
rejected at start-up.
* PostgresConnection.Harden/FromConfiguration puts bounds on the connection
string: lock_timeout 10 s and idle_in_transaction_session_timeout 60 s
(server-side start-up options), command timeout 30 s, connect/pool-wait
timeout 15 s, a pool of at most 20 connections and TCP keep-alive probes
every 30 s. Anything written in the connection string wins; 0 switches a
limit off. Settings: Resilience:Postgres:*.
Tests (265 in total now, 219 without servers, 46 with PostgreSQL/RabbitMQ):
* TransientFaults, connection-string and retry-schedule unit tests, and the
retry policy on a real MassTransit pipeline (in-memory transport, pauses
shortened to milliseconds): an outage longer than the old four attempts is
ridden out without a Fault, EF-wrapped faults are recognised, the policy
gives up after exactly the configured attempts, a bug gets 3 retries and a
broken rule none.
* Against a real PostgreSQL: a waiter behind a stuck row lock gives up at the
lock timeout (control: without it the waiter just hangs); a session left
idle inside a transaction is ended by the server and its locks are
released; a session terminated by the server is retryable; a runaway query
is cancelled at the command timeout and the connection stays usable; with
the pool used up a caller fails after the connect timeout and the server
never sees more than MaxPoolSize connections.
* Fault injection (a DbCommandInterceptor that makes commands fail like a
lost connection): the real ProcessPaymentConsumer, wired through the
MassTransit outbox, survives six dropped connections in a row and charges
the order exactly once (one payment, one idempotency row, one result event);
an outage longer than the patient retries ends after exactly the configured
attempts with nothing half-written. The real Inventory reserve consumer,
while another session holds the stock row lock, hits the lock timeout
repeatedly and still reserves once the lock is released.
* PaymentPostgresTests now uses the shared policy instead of its own
Immediate(3) copy of production.
ShopEasy.Shared now references MassTransit (was MassTransit.Abstractions) and
Npgsql 8.0.6, and Microsoft.Extensions.Configuration.Binder for the options
binding. Dependabot is told to leave the Npgsql major version to the EF
provider that pins it. CorrelationFilters qualifies Serilog's LogContext,
which became ambiguous with MassTransit.LogContext.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
Wires the helpers added in the previous commit into the services. * Order, Inventory and Payment replace their three identical UseMessageRetry blocks (3 quick retries, InvalidOperationException ignored, which in effect disabled retries for database outages) with cfg.UseConsumerResilience(ConsumerResilienceOptions.From(configuration)). A database or network outage is now ridden out for about 7 minutes with growing pauses before a message is dead-lettered; ordinary failures keep their 3 quick retries. * Identity, Order, Inventory, Payment and Product build their Npgsql connection string with PostgresConnection.FromConfiguration, so every service now has a lock timeout (10 s), an idle-in-transaction timeout (60 s), a command timeout (30 s), a connect/pool-wait timeout (15 s), a pool cap (20) and TCP keep-alive. All of it is configurable under Resilience:Postgres:* and the defaults change nothing for a healthy database. Behaviour to be aware of: a SELECT ... FOR UPDATE that cannot get its row lock within 10 s now fails with SQLSTATE 55P03 (a transient fault, so the consumer retries) instead of queueing without limit; a migration that takes longer than the 30 s command timeout needs Resilience:Postgres: CommandTimeoutSeconds raised. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
* New pattern 24 "Surviving Database and Network Outages": the two kinds of failure and what each gets, why the previous policy never retried a database outage, every Resilience:Postgres:* and Resilience:Consumers:* setting with its default and effect, how to size the connection pool (replicas x MaxPoolSize below max_connections minus a reserve), the PgBouncer caveat for the server-side start-up options, and the deliberate choices (no EF EnableRetryOnFailure, no consumer circuit breaker or kill switch, in-process redelivery instead of a delayed-message plugin). * Configuration key reference, test tables and counts (265: 219 without servers, 46 with PostgreSQL and RabbitMQ) and the roadmap (step 9, part 1 done) updated; two known limitations added (outages longer than the retry window end in the error queues; gRPC calls and gateway routes do not have deadlines and retries yet). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
NullButCute
pushed a commit
that referenced
this pull request
Oct 15, 2026
…e ConfigMap The Resilience:Postgres and Resilience:Consumers settings added in #80 only existed as defaults inside the code, so nobody could see or change them without reading the source. They are now written out where an operator looks: * Every service's appsettings.json has the Resilience:Postgres section (Identity, Product, Order, Inventory, Payment); Order, Inventory and Payment, which consume messages, also have Resilience:Consumers. The values are the code's defaults, so behaviour does not change. * docker-compose.yml passes the same keys as environment variables through two YAML anchors (x-resilience-postgres, x-resilience-consumers) merged into the services that use them, each as "${RESILIENCE_...:-default}". An override goes into .env; .env.example lists every variable, commented out, with its default. * k8s/configmaps/configmap.yaml (shopeasy-config, loaded by every Deployment) has the same keys; changing one needs a rollout restart because environment variables are read at pod start. Normal ASP.NET Core precedence applies: environment variable, then appsettings.json, then the default in the code. Several copies of a default can drift, so ResilienceConfigurationTests (15 tests, no servers needed) fails when any appsettings.json, the Compose file, the ConfigMap or .env.example differs from the code's defaults (checked by changing a value in three of them), when a service with a database or consumers does not get its settings, and it checks with the real Order appsettings.json that the file is read by the services and that an override wins. Verified: docker compose config renders the defaults and an override from .env for the right services (gateway gets none); kustomize + kubeconform -strict and the repository's deployment rules pass; the whole suite is green (280 tests, 234 without servers, 46 with PostgreSQL and RabbitMQ). The README documents where the settings live and how to change them, and its test counts are updated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Part 1 of 5 of the application-level resilience work: failure-injection tests, and the outage handling they showed was missing.
Finding (reproduced with a test against the previous policy)
A database outage was never retried. EF Core reports a database failure as an
InvalidOperationException("An exception has been raised that is likely due to a transient failure"). That type was on the consumers' "never retry" list, so the message went to the*_errorqueue on the first failure. Every other failure got three quick retries (under 15 s in total). Nothing bounded how long a service waited for a lock, a query or a connection.Changes
Consumers (Order, Inventory, Payment): three duplicated
UseMessageRetryblocks are replaced byUseConsumerResilience(...)inShopEasy.Shared/Resilience.InvalidOperationExceptionandArgumentNullExceptionare not retried.TransientFaults) looks through the whole exception chain, so EF wrapping no longer hides the cause.Database connections (all five services):
PostgresConnection.FromConfigurationadds a lock timeout (10 s), an idle-in-transaction timeout (60 s, which releases the locks of a hung pod), a command timeout (30 s), a connect/pool-wait timeout (15 s), a pool cap (20) and TCP keep-alive. Anything written in the connection string wins and0switches a limit off. Settings are underResilience:Postgres:*andResilience:Consumers:*, documented in the README (new pattern 24, including pool sizing).Tests (265 in total: 219 without servers, 46 need PostgreSQL/RabbitMQ)
MaxPoolSizeconnectionsDbCommandInterceptordropping connections):ProcessPaymentConsumerthrough the outbox survives six dropped connections in a row and charges the order exactly once.Also smoke-tested: the Payment service starts, migrates and reports ready against PostgreSQL and RabbitMQ with the hardened connection string.
Behaviour to be aware of
SELECT … FOR UPDATEthat cannot get its lock within 10 s now fails with55P03, which is retried, instead of queueing without limit.Resilience__Postgres__CommandTimeoutSecondsraised.0and configured withALTER ROLE … SET. This is in the README.EnableRetryOnFailure(conflicts with the outbox transaction), a consumer circuit breaker or kill switch, a RabbitMQ delayed-message plugin. Reasons are in the README.ShopEasy.Sharednow referencesMassTransit(was.Abstractions),Npgsql8.0.6 andConfiguration.Binder. Dependabot leaves theNpgsqlmajor to the EF provider.Not in this PR (parts 2–5): idempotent API edges, gRPC deadlines/retries and gateway timeouts, the authorization model, business-level observability.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vuhzoey6h5V9wPPVFuR7ME
Generated by Claude Code