Make the 7/s Google fetch cap global, without a migration

[WilfordGrimley]

Follow-up to #644 (merged). Owner clarification, 2026-07-30:

"to be clear: the 7 fetches per second cap is a global cap, it shouldn't be per process or per core"

Collides with #648, which was opened for the same clarification a few hours earlier and which I only found after building this. Both fix the same defect. The substantive differences are listed at the bottom — pick one, don't merge both.

The defect

_DestinationLimiter keeps _next_allowed, its lock and its semaphore in one Python process's memory.

fetcher	processes	what 7/s actually bought
Pooled runner (run_image_evidence_cohort)	one (internal fetch thread pool)	a genuine 7/s
Conveyor (stage_e_dispatch via django-q2)	separate OS processes	N × 7/s — 14/s at the shipped STAGE_E_MAX_CONCURRENT_DISPATCHES = 2, and rising with that cap

#644 established that the rate limit, not the concurrency limit, is what protects the destination. That is exactly what makes the rate limit's globality load-bearing — and it was missing.

Mechanism — the pacer's own arithmetic, moved somewhere shared

The in-memory pacer already computes next_allowed = max(now, next_allowed) + interval. That is a cursor update. It becomes global by running it in Postgres, in one atomic statement:

INSERT INTO shared_cache (cache_key, value, expires) VALUES (%(key)s, (now + %(gap)s)::text, ...)
ON CONFLICT (cache_key) DO UPDATE
   SET value = (GREATEST(EXCLUDED.value::numeric - %(gap)s, shared_cache.value::numeric) + %(gap)s)::text
RETURNING GREATEST(0, value::numeric - %(gap)s - EXTRACT(EPOCH FROM clock_timestamp()))

ON CONFLICT DO UPDATE takes the row lock for the statement's duration, so the read-modify-write is atomic against every process and thread with no advisory lock, no explicit transaction, no SELECT ... FOR UPDATE pair — one round trip per fetch. The statement returns the caller's own wait in seconds; the caller sleeps locally, holding nothing. Time is Postgres's clock_timestamp() throughout, never a per-process monotonic() epoch.

Why not stage_e_concurrency's advisory locks

Read first, as asked. An advisory lock is a pure mutual-exclusion primitive: no payload, no time dimension. It can say "how many at once"; a rate needs a remembered value plus a clock. The two shapes buildable from locks alone were both rejected, and both rejections are in the module docstring:

• "K slots, each held for K/rate seconds" is a correct rate on paper, but the holder must block for the hold, so throughput becomes a function of the fetch thread count (6 threads × a 1s hold = 6/s under a 7/s ceiling — it under-delivers), and it needs a live Postgres session per in-flight fetch.
• A lock used as a mutex around shared state still needs the shared state — which is the design here, minus the mutex, because ON CONFLICT already provides the exclusion.

No Redis: docker/docker-compose.prod.yml runs django, worker, nginx, postgres, elasticsearch. Checked, as asked.

No new migration

The cursor is one number, and the migration graph is actively contended (#611's guard fails a graph that forks as merged with the base; #648 adds a 0102). It therefore lives in the existing shared_cache table (0092, already created by the migrate that docker/django/entrypoint.sh runs, in prod and CI and every dev database), as a raw numeric string under a key prefix Django's own make_key can never emit (every Django cache key there starts ":<version>:"). The two ways Django could remove the row — cache.clear() and _cull above MAX_ENTRIES = 1000 — are both benign: a missing row is re-inserted at "now", costing at most one immediately-allowed fetch, never a burst.

Requirement 1 — proven with more than one limiter, and with real processes

A single-instance test cannot see this defect, and that is precisely how it shipped. So:

• test_three_processes_share_one_ceiling forks three real OS processes — the shape django-q2 actually uses — each with its own limiter at a 20/s ceiling, and asserts the merged 30-acquisition timeline. Both an upper bound (aggregate ≤ ceiling, plus a sliding-1s-window peak check) and a lower bound on elapsed span, because a per-process pacer fails the second by a factor of the process count.
• test_four_independent_limiters_share_one_ceiling — same discriminator, four independent limiter objects in threads, fast enough for every change.

Non-vacuity, measured: with acquire() reverted to the pre-fix per-process pacer, those two tests report 63.47/s and 110.49/s against their ceilings of 20 and 25. They fail loudly at exactly the ratio the defect predicts.

Requirement 2 — the pooled runner is not regressed (measured)

measurement	result
per reservation, 200 steady-state calls vs container Postgres	1.08–1.36 ms
as a share of the 143 ms interval a 7/s ceiling already imposes	0.76 – 0.95 %
8 acquisitions driven at the real GOOGLE_IMAGE.rate_per_sec = 7.0	1.003 s vs 1.000 s for the ceiling alone — +0.3 %

Both are asserted, not just printed (TestCoordinationCost), so this stays true. No trade needs proposing — the round trip is amortised into a pacing interval that already exists and is two orders of magnitude larger. One dedicated autocommit connection per process (not per thread), so the extra Postgres connection count is 1 per fetching process; the lock serialising threads onto it clears ~925 reservations/s, against a 7/s ceiling.

django.db.connection is deliberately not used: a reservation inside a caller's atomic() block would hold the cursor row lock until that transaction committed, stalling every other fetching process — a global rate ceiling that intermittently becomes a global stop.

Requirement 3 — purely additive

Nothing #644 established is touched: the throttle-not-halt conversion, the 429/503 rate-pressure classification, the backoff decay schedule, and the envelope bar classification (HOST_LOAD / RSS / GOOGLE_LOCKOUT still halt) are all unchanged. Backoff stays per-process and reaches the shared cursor as an already-widened interval, which can only ever slow the aggregate.

Requirement 4 & 6 — degrade: neither open nor closed

If Postgres is unreachable for a reservation, reserve() returns None (never raises) and the caller falls back to its own pacer at rate ÷ (STAGE_E_MAX_CONCURRENT_DISPATCHES + 1) — the conveyor's own cross-process dispatch cap, plus one for a pooled/manual runner that holds no dispatch slot. Worst case, every process fetching: N × rate/N = rate. The ceiling survives the outage.

• Fail open (per-process pacing at the full rate) was rejected: it restores the exact N × 7/s defect at the moment nobody is watching.
• Fail closed (refuse to fetch) was rejected: it hard-stops an unattended 230,753-card pass over a transient blip, the opposite of "throttle, do not shut it down".

A degraded reservation waits. It does not fail, does not halt, does not feed the envelope's fetch-failure window. Failures are latched for 5 s so an outage costs one connection attempt per cooldown rather than one per fetch (asserted).

Requirement 5 — default-on, no flag

The divisor is derived from STAGE_E_MAX_CONCURRENT_DISPATCHES, not a new setting. A second knob for the same fact is just a way for the two to disagree.

Fork safety

os.register_at_fork(after_in_child=...) abandons the inherited connection object rather than closing it — django-q2's Cluster forks its workers, and PQfinish on an inherited handle sends Terminate down a socket the parent is still using. (This is not theoretical: the multi-process test hit exactly that failure via connections.close_all() in the child and killed pytest's own session.)

Docs

docs/features/stage-e-operations.md (wiki Stage-E-Operations), inside #644's own "Rate pressure is throttled, not halted" section. The per-process-versus-global distinction is stated flatly — "not 7/sec for each django-q2 worker, not per dispatch, not per core" — because two readers have now got it wrong, plus what an operator should do instead (add up every fetching process; concurrency knobs now change only how the budget is shared, never how much of it there is) and the degradation behaviour. No dated report.

Differences from #648

	this PR	#648
migration	none — reuses shared_cache (0092)	adds 0102_global_fetch_pace to a contended graph
coordination-store outage	divides the rate across the max process count — ceiling holds	fails open to per-process pacing at the full rate — restores N × 7/s
multi-process proof	forked OS processes + independent instances	independent instances in threads
#644 backoff state	untouched, stays per-process	moved into the shared row (global backoff + shared clean streak)

#648's global-backoff argument is real and this PR does not have it; this PR's no-migration and non-fail-open properties are the trade in the other direction.

🤖 Generated with Claude Code

https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN