Abort postgres queries whose client died - #474
Merged
Merged
Conversation
Aborts queries whose client died mid-query instead of running to completion for a dead client.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #462
Problem
Postgres backends survive client-container recreation and keep computing for dead clients. Observed 2026-07-25: six
SELECT DISTINCT cardpicker_card.idbackends ran up to 2h31m after their django/worker client containers were recreated by a deploy — a backend mid-query never touches its client socket, so it can't notice the client is gone until results are ready to send.Fix
docker/docker-compose.ymlpostgres service now setscommand: postgres -c client_connection_check_interval=30000(30s) — Postgres 14+ probes the client socket periodically during query execution and aborts the query if the client is gone.docker-compose.prod.ymlextends this service and does not overridecommand, so the flag inherits into prod.Deliberately not
statement_timeout(out of scope per the issue) — BULK-mode eligibility scans legitimately run for minutes-hours with a live client and must not be killed.Verification
docker compose -f docker/docker-compose.yml config(dev file, no sudo) renderscommand: [postgres, -c, client_connection_check_interval=30000]for the postgres service.docker compose -f docker/docker-compose.prod.yml config(with dummy env vars for interpolation, no sudo) renders the samecommandinherited viaextends, confirming prod doesn't override it.docker run --name pg462check ... postgres -c client_connection_check_interval=30000, no connection to the live prod DB or its volume):SHOW client_connection_check_interval;→30s. Container removed after the check.docs_lint.pyclean; pre-commit clean on both changed files.Deploy step (not done in this PR)
Rolling this out requires one deliberate
postgrescontainer recreate (docker compose -f docker-compose.prod.yml up -d postgresor equivalent) — a few seconds of DB downtime. django and worker reconnect on their own; no restart needed for either. Documented indocs/infrastructure.md's deploy section.Checklist
docker/docker-compose.yml: addedcommand: postgres -c client_connection_check_interval=30000to the postgres servicedocker-compose.prod.ymlextends without overridingcommanddocker compose configrenders it (dev + prod)SHOW client_connection_check_interval;with30sdocs/infrastructure.mddeploy section note (rollout requires one deliberate postgres recreate)statement_timeoutuntouched (out of scope)