Make the 7/s Google fetch cap global, not per process

[WilfordGrimley]

Follow-up to #644 (merged). Owner clarification, 2026-07-30:

"to be clear: the 7 fetches per second cap is a global cap, it shouldn't be per process or per core"

The gap

_DestinationLimiter paces with a threading.Lock and a process-local next_allowed. #644's rate_per_sec = 7.0 therefore bought 7/s per process:

fetcher	processes	what the per-process cap actually gave
Pooled runner (run_image_evidence_cohort)	one (internal fetch thread pool)	a genuine 7/s — correct, but by accident
Conveyor (stage_e_dispatch via django-q2)	many (separate OS processes)	N × 7/s. 14/s at the production STAGE_E_MAX_CONCURRENT_DISPATCHES = 2, scaling with the cap

Same per-process trap as threading.Semaphore(max_concurrency) — the one #589 deleted a false ceiling term over. #644 established that the rate limit, not the concurrency limit, is what protects the destination; that is exactly what makes its globality load-bearing rather than a refinement.

Mechanism

One Postgres row per destination (GlobalFetchPace, migration 0102), advanced by a single atomic statement:

UPDATE cardpicker_globalfetchpace
   SET next_allowed_at = GREATEST(clock_timestamp(), next_allowed_at)
                       + (%s * backoff_multiplier) * INTERVAL '1 second'
 WHERE destination = %s
RETURNING ...

That is the same arithmetic acquire() already did in Python, lifted into the one place every process can see. The caller then sleeps to its slot locally, holding no lock and no transaction. Concurrent reservers serialise on Postgres's row lock, and under READ COMMITTED the blocked statement re-evaluates against the committed row — so reservations form one strictly-increasing sequence no matter how many processes compete. Nothing here is advisory or best-effort.

Three properties worth knowing:

• The database's clock (clock_timestamp()), never time.monotonic(), whose epoch is per-process. Persisting one process's monotonic reading for another to read would reintroduce the exact defect while looking coordinated.
• The backoff multiplier and clean streak moved into the same row. Not scope creep — a global rate ceiling requires a global backoff term, or two processes holding different multipliers write conflicting paces into the same row and the effective rate is whatever the least-backed-off process believes. Rate pressure throttles the pass instead of shutting it down; Google ceiling 8.0 -> 7.0 #644's 429/503 semantics and decay schedule are unchanged; only their storage moved. (The streak had to move too: N processes each counting their own would cross the decay threshold N times over and recover N× faster than intended.)
• max_concurrency stays per-process, deliberately — a local resource bound, not the destination-protecting ceiling.

Why a row rather than stage_e_concurrency's advisory locks

Evaluated first, as asked, since that module is this repo's established cross-process primitive. It cannot carry a rate, and both halves of its own reasoning invert here:

1. An advisory lock is a binary held/not-held token. It expresses "how many at once" (concurrency); it cannot express "how many per unit time", because a rate needs a remembered timestamp and a lock stores no value. Faking it — holding each of N locks for exactly 1/R seconds — turns every fetching process into a sleeper occupying a Postgres session, and makes the rate a function of how long a lock is held rather than how often it is acquired.
2. That module rejected a DB row for crash safety: a kill -9'd worker leaves a row-based slot claimed forever. That objection is about a claim. This row holds no claim — only a timestamp. A process killed mid-reservation leaves next_allowed_at at most one interval ahead, self-healing in ~143 ms at 7/s with zero reconciliation code. The very property that made a row unsafe for a slot makes it correct for a pace.

Redis was not an option: docker/docker-compose.prod.yml runs django, worker, nginx, postgres and elasticsearch, and no Redis (Django's cache here is the default per-process LocMemCache). Postgres is the one shared, process-visible state this pipeline guarantees.

Requirement 2 — the pooled runner is not regressed (measured, not assumed)

Reservation cost over 300 steady-state calls against the containerised Postgres the suite uses:

metric	mean	p50	p95	p99
per reservation	1.49 ms	1.30 ms	2.33 ms	8.09 ms

That is 0.78 % of a ~300 ms image fetch at p95, and ~1 % of the 143 ms pacing interval the reservation itself schedules. The pooled runner's fetch threads touch the row only 7 times a second in total, so the row lock runs at roughly a 1 % duty cycle — no contention. No trade needs proposing; the cost is not material.

Connections are a dedicated, thread-cached psycopg2 connection, not django.db.connection. The first reason is a correctness bug, not a preference: a reservation made inside a caller's atomic() block would hold the row lock until that whole transaction committed, serialising every other fetcher in the deployment behind it. autocommit=True releases it the instant the statement returns. The second is cost — one connect per fetch thread for the life of the process, not one per request.

Requirement 4 — degradation

If the row is unreachable, RateBudgetUnavailable is raised, the limiter falls back to its own per-process pacer, and the run continues — logged at ERROR. The ceiling reverts to where this repo already was; it never fails or halts. Deliberately the opposite of stage_e_concurrency's fail-closed, because the downsides differ: an unavailable concurrency cap lets unbounded dispatches pile onto the host, while an unavailable rate budget still leaves every process individually paced at 7/s.

Verification

The key test runs four independent limiters concurrently — a single-instance test cannot detect this defect, which is precisely what let it through. test_four_independent_limiters_share_one_budget drives 4 limiters × 10 requests at a configured 50/s and asserts the aggregate.

Mutation-checked (red → green). Reverting acquire() to per-process pacing (wait_time = self._local_wait()) fails it with the defect reproduced exactly:

AssertionError: 40 requests across 4 independent limiters took 0.181s;
a global 50.0/s cap requires at least 0.780s. A per-process cap would
finish in ~0.180s - that is the defect this asserts against.

0.181s against a required 0.780s — a 4× overshoot at ~221/s against a 50/s cap. Mutation reverted; green.

Also covered: the reservation sequence asserted directly (8 concurrent reservers each get a distinct, correctly-spaced slot); single-process pacing unregressed; degradation still paces and never raises; a truncated table recreates its row rather than wedging; shared backoff visible across reservers, capped, applied to the shared pace; shared streak so decay is not N× faster; decay floored at the configured ceiling.

• Full cardpicker/tests/ suite: 3436 passed, 11 skipped, 0 failed.
• pre-commit run --all-files green. docs_lint.py --strict clean.
• check_migration_leaves.py --base origin/master: one leaf per app (102 migrations). 0102 is a clean single leaf off 0101; no open PR claims a migration.
• The new test file was run 3× consecutively to confirm it is not timing-flaky.

Docs

Per the owner documentation rule, this edits the wiki-facing docs/features/stage-e-operations.md — no dated report. A new subsection under "Rate pressure is throttled, not halted" states the per-process vs global distinction explicitly (with the per-fetcher table above), the mechanism, the three properties, the rejected alternatives, the measured cost, and the degradation behaviour.

Preserved from #644

The throttle-not-halt conversion, the 429/503 classification, the decay schedule and the bar classification are all untouched. This is additive.

🤖 Generated with Claude Code

https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN

[WilfordGrimley]

Closing in favour of #649, which fixes the same defect. Not a quality judgement — I dispatched two agents onto the same task by mistake and both produced sound solutions. #649 wins on three points:

1. The degraded path. This PR fails open: if the coordination store is unreachable it reverts to per-process pacing at the full rate, restoring the exact N x 7/s defect at the moment nobody is watching. Make the 7/s Google fetch cap global, without a migration #649 divides by the dispatch cap + 1, so worst case N x rate/N = rate — the ceiling survives the outage.
2. The proof. Make the 7/s Google fetch cap global, without a migration #649 asserts the aggregate across real forked OS processes, the shape django-q2 actually uses (reverting its fix gave 63.47/s against a 20/s ceiling). This PR proves it across independent limiter instances on threads — a weaker witness for a per-process defect.
3. No migration. Make the 7/s Google fetch cap global, without a migration #649 reuses the existing shared_cache table (0092) under a key prefix Django make_key can never emit. This PR adds 0102 to a graph contended all day.

What this PR had that #649 does not, and which is not being lost: the global backoff multiplier and shared clean-streak. The argument is correct — a shared next_allowed_at advanced by per-process intervals is incoherent, and N processes each counting their own streak recover N times faster than the agreed schedule intends. Filed as #653, citing this PR as the starting point.

Also preserved there: the observation that stage_e_concurrency rejected a DB row for crash safety — a killed worker leaves a claimed slot claimed forever — but that objection is about a claim, and a pace row holds only a timestamp, self-healing within one interval. The property that makes a row unsafe for a slot makes it correct for a pace. That is a genuinely good piece of reasoning and worth keeping findable.

Branch left in place; nothing force-pushed or deleted.