You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Python CI lane 3.11 on run 32481181792 (branch fix/ci-dependency-routing-reliability, commit ac63b7b) wedged at ~66% of the suite and was killed by the 45-minute job ceiling. pytest-timeout fired at 180s and dumped this worker-pool thread:
Timeout (0:03:00)!
Thread 0x00007f65f1bf16c0 (most recent call first):
File ".../src/jobctrl/infrastructure/enrichment/execution_lease.py", line 68 in claim_enrichment_execution_lease_for_run
File ".../src/jobctrl/enrichment/activities.py", line 363 in _claim_activity_enrichment_lease
File ".../src/jobctrl/enrichment/activities.py", line 198 in _run_observed_enrichment
File ".../src/jobctrl/infrastructure/workflow_run_context.py", line 78 in bound
File ".../concurrent/futures/thread.py", line 58 in run
Line 68 is conn.execute("BEGIN IMMEDIATE"). The main thread was alive inside the asyncio event loop (selectors.py select) under a live temporal-test-server-sdk-python-1.26.0; after the dump the process sat silent for 32 more minutes because timeout_method = "signal" raises only in the main thread and cannot unwind a blocked worker-pool thread, and interpreter shutdown then joins the still-blocked executor thread.
Why it is order-dependent and pre-existing
The suite is shuffled by pytest-randomly; all lanes passed on main as recently as 2026-08-15, and sibling lanes (3.12, 3.13) passed on the same commit that hung. The interleaving that leaves a competing SQLite write transaction open while the enrichment activity claims its lease occurs only under some orders.
database.py sets PRAGMA busy_timeout=10000 on its connections, yet the thread blocked far beyond 10s. Either the blocking or the blocked connection did not come from those init paths (default busy_timeout=0 returns SQLITE_BUSY immediately rather than blocking, so a plain lock wait does not explain a 180s+ block), or this is CPython sqlite3 connection-object serialization: two threads sharing one connection object block on the connection's internal mutex with no timeout.
Which test/fixture holds the competing transaction (or shares the connection object) needs the reproducing order; the run's seed is in the job log (Using --randomly-seed=...).
timeout_method = "thread" now dumps every thread and hard-exits at 180s, so this deadlock costs a ~3-minute attributable failure instead of a 45-minute opaque cancellation. That is a mitigation, not the fix: the lease claim path can genuinely deadlock under concurrent test-shaped usage and deserves a bounded claim (e.g., explicit busy_timeout on the owning connection, or a bounded retry with a clear StaleEnrichmentExecutionLease-style error).
Symptom
Python CI lane 3.11 on run 32481181792 (branch
fix/ci-dependency-routing-reliability, commit ac63b7b) wedged at ~66% of the suite and was killed by the 45-minute job ceiling.pytest-timeoutfired at 180s and dumped this worker-pool thread:Line 68 is
conn.execute("BEGIN IMMEDIATE"). The main thread was alive inside the asyncio event loop (selectors.py select) under a livetemporal-test-server-sdk-python-1.26.0; after the dump the process sat silent for 32 more minutes becausetimeout_method = "signal"raises only in the main thread and cannot unwind a blocked worker-pool thread, and interpreter shutdown then joins the still-blocked executor thread.Why it is order-dependent and pre-existing
mainas recently as 2026-08-15, and sibling lanes (3.12, 3.13) passed on the same commit that hung. The interleaving that leaves a competing SQLite write transaction open while the enrichment activity claims its lease occurs only under some orders.execution_lease.pywas last touched in fix(enrichment): reconcile durable workflow ownership #751; nothing in the CI PR modifies it.Open questions for the root cause
database.pysetsPRAGMA busy_timeout=10000on its connections, yet the thread blocked far beyond 10s. Either the blocking or the blocked connection did not come from those init paths (defaultbusy_timeout=0returns SQLITE_BUSY immediately rather than blocking, so a plain lock wait does not explain a 180s+ block), or this is CPythonsqlite3connection-object serialization: two threads sharing one connection object block on the connection's internal mutex with no timeout.Using --randomly-seed=...).Current mitigation (PR #835)
timeout_method = "thread"now dumps every thread and hard-exits at 180s, so this deadlock costs a ~3-minute attributable failure instead of a 45-minute opaque cancellation. That is a mitigation, not the fix: the lease claim path can genuinely deadlock under concurrent test-shaped usage and deserves a bounded claim (e.g., explicit busy_timeout on the owning connection, or a bounded retry with a clear StaleEnrichmentExecutionLease-style error).