fix(stage_e_dispatch): stop the Stage C compute pool from killing the parent's DB connection across fork - #680
Merged
Conversation
… parent's DB connection across fork _run_stage_c deliberately keeps the parent's Django connection open across the ProcessPoolExecutor fork (closing it there previously broke mark_ledger_failed). That means every forked compute worker inherits a LIVE duplicate of the parent's own TCP socket, not an unused handle. _stage_c_compute_worker_init's own docstring wrongly assumed the parent had already closed its connection before forking, so its "close inherited connections" step called the real BaseDatabaseWrapper.close() (psycopg2's PQfinish) on that live duplicate -- sending a wire-level Terminate over the SHARED socket and silently ending the parent's own session too. Postgres logs nothing (a Terminate is a clean disconnect, not a crash), and the parent's very next query then fails with "server closed the connection unexpectedly" -- exactly the stream_full_catalog resume failure this fixes. The fix discards each worker's inherited connection reference directly (`conn.connection = None`) instead of calling close() on it. Django's own connect() resets every other piece of per-connection state unconditionally on next use, so this is sufficient for a genuinely fresh, independent connection on the worker's own first query, without ever touching the shared wire. Added TestStageCComputeWorkerInitRealFork, the first test in this file to exercise a REAL ProcessPoolExecutor fork (every other test replaces it with _SyncStagePoolStub, which never forks and explicitly neutralizes this exact close call) -- confirmed it fails against current master with the identical psycopg2.OperationalError reported in production, and passes with the fix. Updated _SyncStagePoolStub and TestStageCComputeWorkerInit's own fixture to snapshot/restore each connection wrapper's whole __dict__ around the initializer call, since the new discard-not-close behavior mutates wrapper state (autocommit, in_atomic_block) that the old close()-patching neutralization never had to account for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Root-causes and fixes the
stream_full_catalogresume failure reported2026-08-04: every launch since the PR #669/
8b7f02c5deploy died within~15 seconds on the first query inside
dispatch_micro_batch(
psycopg2.OperationalError: server closed the connection unexpectedlyin_partition_by_md5_verdict), with Postgres logging nothing at eitherobserved failure.
Root cause:
_run_stage_cdeliberately keeps the parent's Django DBconnection open across the
ProcessPoolExecutorfork it constructs for theStage C compute pool (closing it there previously broke
mark_ledger_failedon a compute-side crash — see that construction site's own comment). That
means every forked compute worker inherits a live duplicate of the
parent's own TCP socket, not an unused handle.
_stage_c_compute_worker_init'sown docstring wrongly assumed the parent had already closed its connection
before forking, so its "close inherited connections" step called the real
BaseDatabaseWrapper.close()(psycopg2'sPQfinish) on that liveduplicate — sending a wire-level Terminate message over the shared
socket, which silently ends the parent's own session too. A Terminate is a
clean, expected disconnect from Postgres's point of view, not a crash, which
is exactly why nothing was logged server-side. The parent's very next query
(
_partition_by_md5_verdict, immediately after_run_stage_creturns) thenfails with "server closed the connection unexpectedly".
Fix: discard each worker's inherited connection reference directly
(
conn.connection = None) instead of calling.close()on it. Django's ownconnect()unconditionally resets every other piece of per-connection stateon next use, so this is sufficient for a genuinely fresh, independent
connection on the worker's own first query, without ever touching the shared
wire.
Also updates
_SyncStagePoolStubandTestStageCComputeWorkerInit's ownfixture (test-only code) to snapshot/restore each connection wrapper's whole
__dict__around the initializer call, rather than just patching out.close()— the new discard-not-close behavior mutates wrapper state(autocommit,
in_atomic_block) via the lexicon-builder queries that runlater in the same call, which the old close()-patching neutralization never
had to account for.
Checklist
pre-commitand installed the hooks withpre-commit installbefore creating any commits.TestStageCComputeWorkerInitRealFork— the first test intest_stage_e_dispatch.pyto exercise a realProcessPoolExecutorfork (every other test in the file replaces it with
_SyncStagePoolStub, which never forks and explicitly neutralizes thisexact close call). Confirmed it fails against current
masterwith theidentical
psycopg2.OperationalError: server closed the connection unexpectedlyreported in production, and passes with this fix.cardpicker/tests/test_stage_e_dispatch.py: 83 passed.test_stream_full_catalog.py+test_run_pipeline.py: 143 passed.cardpickersuite (host venv,/home/ubuntu/.venvs/mpcautofill-pilot,excludes
test_federation_hash_tool_parity.pywhich needsfederation-hash-tool/, not present outside the container): 3573passed, 8 skipped (named credential/live-fetch skips), 0 failed.