Skip to content

Abort postgres queries whose client died - #474

Merged
WilfordGrimley merged 1 commit into
masterfrom
pg-client-connection-check
Jul 25, 2026
Merged

Abort postgres queries whose client died#474
WilfordGrimley merged 1 commit into
masterfrom
pg-client-connection-check

Conversation

@WilfordGrimley

Copy link
Copy Markdown

Closes #462

Problem

Postgres backends survive client-container recreation and keep computing for dead clients. Observed 2026-07-25: six SELECT DISTINCT cardpicker_card.id backends ran up to 2h31m after their django/worker client containers were recreated by a deploy — a backend mid-query never touches its client socket, so it can't notice the client is gone until results are ready to send.

Fix

docker/docker-compose.yml postgres service now sets command: postgres -c client_connection_check_interval=30000 (30s) — Postgres 14+ probes the client socket periodically during query execution and aborts the query if the client is gone. docker-compose.prod.yml extends this service and does not override command, so the flag inherits into prod.

Deliberately not statement_timeout (out of scope per the issue) — BULK-mode eligibility scans legitimately run for minutes-hours with a live client and must not be killed.

Verification

  • docker compose -f docker/docker-compose.yml config (dev file, no sudo) renders command: [postgres, -c, client_connection_check_interval=30000] for the postgres service.
  • docker compose -f docker/docker-compose.prod.yml config (with dummy env vars for interpolation, no sudo) renders the same command inherited via extends, confirming prod doesn't override it.
  • Isolated throwaway container (docker run --name pg462check ... postgres -c client_connection_check_interval=30000, no connection to the live prod DB or its volume): SHOW client_connection_check_interval;30s. Container removed after the check.
  • docs_lint.py clean; pre-commit clean on both changed files.

Deploy step (not done in this PR)

Rolling this out requires one deliberate postgres container recreate (docker compose -f docker-compose.prod.yml up -d postgres or equivalent) — a few seconds of DB downtime. django and worker reconnect on their own; no restart needed for either. Documented in docs/infrastructure.md's deploy section.

Checklist

  • docker/docker-compose.yml: added command: postgres -c client_connection_check_interval=30000 to the postgres service
  • Verified docker-compose.prod.yml extends without overriding command
  • Verified docker compose config renders it (dev + prod)
  • Verified a live container answers SHOW client_connection_check_interval; with 30s
  • docs/infrastructure.md deploy section note (rollout requires one deliberate postgres recreate)
  • statement_timeout untouched (out of scope)

Aborts queries whose client died mid-query instead of running to
completion for a dead client.
@WilfordGrimley
WilfordGrimley merged commit 678e7d2 into master Jul 25, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Abort Postgres queries whose client died: client_connection_check_interval

1 participant