Fix Stage E concurrent-dispatch vote-collision IntegrityError - #448
Merged
Conversation
Two concurrent dispatch_micro_batch invocations (django-q2's 8 workers, or the backstop sweep racing an event trigger) could both pass Stage D's per-identity eligibility check before either committed, then race to bulk_create the same (card, anonymous_id) CardPrintingTag - the loser hit IntegrityError and aborted its whole micro-batch (trip envtrip-20260724T214616-be6e5db9, failed run_ids stage-e-stream-20260724T2144*). Adds a pre-write skip-if-exists guard (_split_new_printing_tag_votes, mirroring PR #411's precedent) to run_join_key_calculator and run_fallback_calculator - skip-and-count, not retract-and-recast, since a concurrent race yields the same verdict from the same evidence, not a genuine conclusion change. run_slow_path_calculator needs no equivalent guard (CardScanLog carries no DB uniqueness constraint). Corrects stage-e-operations.md's overstated "eligibility exclude alone is idempotent" claim for the concurrent (not just sequential) case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n gate) Tron gate on PR #448: seven failed run_ids + one winner (not four) = Q_CLUSTER workers=8; separates the vote-collision failure (this guard) from the SEPARATE envtrip-20260724T214616 host-load trip (11.85 vs 7.0, 8 concurrent OCR dispatches on 7 cores, not fixed by this change); qualifies the "same verdict" premise as contingent on unchanged code/ evidence/lexicon, naming reparse_collector_evidence as the remedy otherwise; adds bulk_create(..., ignore_conflicts=True) as the actual crash-proofing against the guard's own residual check-then-insert race window (precedent: local_layout_class_cast.py:300, local_detect_ai_art.py:459, local_identify_printing_tags.py:1246), with a regression test defeating the pre-write check to prove it; adds an ops-doc runbook line against running BULK-mode writes while PASSIVE streaming is enabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
WilfordGrimley
added a commit
that referenced
this pull request
Jul 24, 2026
… body Verifies dispatch_micro_batch itself (post-#448 merge) returns throttled-concurrency-cap and does zero work when every slot is held by a genuinely independent connection, and proceeds normally once released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
WilfordGrimley
added a commit
that referenced
this pull request
Jul 25, 2026
* Add Stage E streaming dispatch concurrency cap Companion to PR #448's vote-collision fix (Tron gate round 1, COMPANION item), kept as a separate PR per that gate's own instruction. The shakedown's first live run had eight concurrent dispatch_micro_batch calls - all running CPU-bound OCR/phash extraction at once - trip the envelope's host-load bar (11.85 vs 7.0 ceiling) on a host with only 7 usable compute cores. The envelope only trips REACTIVELY, after load has already spiked; this cap is PROACTIVE, refusing to even start a dispatch once settings.STAGE_E_MAX_CONCURRENT_DISPATCHES (default 2) concurrent dispatches are already running. Mechanism: Postgres session-scoped advisory locks (cardpicker.stage_e_concurrency), not a cache-based counter (this app's cache is per-process LocMemCache, useless across django-q2's 8 separate worker processes) or a dedicated low-worker queue (disproportionate infra for a conservative cap). Chosen over a DB-row counter specifically for crash safety: a killed process's session-scoped lock auto-releases, no reconciliation code needed - no new migration either. New DispatchOutcome status "throttled-concurrency-cap" (no ledger row, matching the other halted statuses). Tests include genuine cross-session races (a raw psycopg2 connection standing in for a second django-q worker process) and a real-thread contention test with a shared max-concurrently-held counter, not just sequential calls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add integration coverage for the cap against the real merged dispatch body Verifies dispatch_micro_batch itself (post-#448 merge) returns throttled-concurrency-cap and does zero work when every slot is held by a genuinely independent connection, and proceeds normally once released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop backstop sweep on throttled-concurrency-cap; floor slot count at 1 * Fix docs_lint path: MPCAutofill/MPCAutofill/settings.py (doubled dir) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes the vote-collision half of the first shakedown-sweep incident: two
CONCURRENT
dispatch_micro_batchinvocations (django-q2'sQ_CLUSTER["workers"] = 8, or the cron backstop sweep overlapping an eventtrigger) can both pass
run_join_key_calculator's orrun_fallback_calculator's own per-identity eligibility check before eithercommits, then race to
bulk_createthe same(card, anonymous_id)CardPrintingTag- the loser hitIntegrityErroroncardprintingtag_unique_no_match_vote/cardprintingtag_unique_printing_voteand aborted its whole micro-batch. The shakedown run produced exactly
seven failed
PilotRunLedgerrows (run_idsstage-e-stream-20260724T2144*)plus the one dispatch that won the race - seven losers + one winner = eight
total concurrent dispatches, exactly
Q_CLUSTER["workers"] = 8.A second, separate failure the same run hit is explicitly NOT fixed by
this PR:
envtrip-20260724T214616-be6e5db9is abar=host_loadenvelopetrip (observed load
11.85against the7.0ceiling, tripped 0.43s afterthe winning vote landed), caused by those same eight concurrent dispatches
saturating this host's 7 usable cores running OCR/phash extraction at once
(
docs/features/catalog-completion-plan.mdL1794/2248/2366's hardwarecitation and 0.31x concurrency finding) - a resource-contention problem,
not a vote-write correctness one. Fixing the vote collision does nothing to
stop eight dispatches from re-tripping the load bar the moment streaming
resumes; that's the companion PR's job (a
settings.STAGE_E_MAX_CONCURRENT_DISPATCHESconcurrency cap), trackedseparately per Tron's review of this PR.
Two lines of defense against the vote collision, both required:
(
local_calculate_verdicts._split_new_printing_tag_votes, mirroring PRFix local_lands_identify write-path vote collision and dry-run yield display #411's own precedent) - skip-and-count, not retract-and-recast, GIVEN
both racing reads see the same evidence under the same code version and
lexicon (
reparse_collector_evidenceremains the correct remedy if thatassumption doesn't hold, e.g. a mid-race deploy or re-extraction).
bulk_create(..., ignore_conflicts=True)on both guarded call sites -the pre-write check alone is check-then-insert, not atomic, and leaves a
narrow residual query-to-insert race window;
ignore_conflicts=Trueisthe actual crash-proofing, matching this codebase's own established
precedent for the identical shape (
local_layout_class_cast.py:300,local_detect_ai_art.py:459,local_identify_printing_tags.py:1246).run_slow_path_calculatorwas checked for the same hazard and needs noguard - it writes only
CardScanLogrows, which carry no DB uniquenessconstraint at all (append-only by design).
Also corrects
docs/features/stage-e-operations.md's "Resume contract"section (previously overstated the eligibility-exclude alone as sufficient
for idempotence - true only for a sequential re-invocation, not a
concurrent one), and adds a runbook line: do not run a BULK-mode write
command while PASSIVE streaming is enabled, since BULK mode is entirely
outside the envelope's own bars and outside any per-worker concurrency cap.
Checklist
pre-commitand installed the hooks withpre-commit installbefore creating any commits.pytest cardpicker/tests/test_local_calculate_verdicts.py cardpicker/tests/test_stage_e_dispatch.py(137 passed), plus the neighboringtest_local_lands_identify.py/test_reparse_collector_evidence.py/test_command_local_lands_identify.py(89 passed) and a broader sweep of every file importinglocal_calculate_verdicts/stage_e_dispatch(474 passed) - all via the hostmpcautofill-pilotvenv against ephemeral testcontainer Postgres/ES (docs/troubleshooting.md's documented convention), never the live prod stack.black --check,ruff check,isort --check,mypyall clean on every touched file;docs_lint.pyclean.TestSplitNewPrintingTagVotes), an end-to-end "seed the winner's vote, confirm the loser survives it" test in bothTestRunJoinKeyCalculator/TestRunFallbackCalculator, a full-conveyor version intest_stage_e_dispatch.py'sTestConcurrentDispatchVoteCollision, and (added in the Tron-gate correction round) a test that defeats the pre-write check itself to proveignore_conflicts=Truealone survives the residual race window.docs/features/stage-e-operations.md's "Resume contract, extended to a streamed micro-batch" section (idempotence correction, the two-separate-failures distinction, the BULK-mode-vs-PASSIVE-streaming runbook line) and its Observability counter list.Tron gate history
Round 1 (CONFIRMED with required corrections): incident count was wrong
(said four failed rows, actually seven + one winner = eight, matching
Q_CLUSTER["workers"]); the docs conflated the vote-collision failure withthe separate host-load envelope trip; the "never a crash" claim overstated
a check-then-insert guard's actual guarantee; the "necessarily same verdict"
premise needed to be stated as contingent, not absolute. All four addressed
in the second commit on this branch, plus a new regression test proving the
ignore_conflicts=Truebelt-and-suspenders survives even when thepre-write check itself is defeated. The companion concurrency-cap change
(
settings.STAGE_E_MAX_CONCURRENT_DISPATCHES) ships as a separate PR perTron's own instruction to keep it decoupled from this correctness fix.