Skip to content

perf(stage-c): raise fetch thread count to limiter's own size (issue #566) - #678

Merged
WilfordGrimley merged 1 commit into
masterfrom
perf/stage-c-fetch-threads-6
Aug 4, 2026
Merged

perf(stage-c): raise fetch thread count to limiter's own size (issue #566)#678
WilfordGrimley merged 1 commit into
masterfrom
perf/stage-c-fetch-threads-6

Conversation

@WilfordGrimley

Copy link
Copy Markdown

Summary

Raises _STAGE_C_FETCH_THREADS from 3 to 6 in MPCAutofill/cardpicker/stage_e_dispatch.py. This is a one-constant change plus an extended code comment; no other file is touched.

harvest_fetch_limiter.GOOGLE_IMAGE is already configured rate_per_sec=7.0, max_concurrency=6 — the limiter was sized for six concurrent fetches, and the fetch pool was running at half that. Fetch threads are I/O-blocked on a socket and cost approximately no CPU, so this does not move host load the way a compute worker would. The rate ceiling stays global regardless of thread count: harvest_rate_coordinator.reserve() paces every fetch through a single cursor row in the shared_cache Postgres table (INSERT ... ON CONFLICT DO UPDATE + clock_timestamp()), so six threads in one process still cannot exceed 7/s — they queue on the cursor, not the socket. The only route past the ceiling is two processes fetching at once, which is a sequencing question, not a thread-count one.

At the prior 3 fetch / 3 compute split, fetch was measured as the bottleneck (~3.3/s vs compute's ~4.6/s), so compute workers idled waiting for images. This takes fetch off the critical path and leaves compute as the sole constraint.

Unchanged, deliberately (hard cap, tied to real host-load envelope trips):

  • _STAGE_C_COMPUTE_WORKERS = 3docs/reports/2026-07-20-pipeline-compute-profile.md measured 6-way concurrency on this CPU-bound OCR stage running 3.25x SLOWER per card while burning 27.7x more CPU-seconds.
  • _STAGE_C_POOL_QUEUE_DEPTH = 6 — the backpressure bound that keeps RSS flat (2 × compute workers, follows that constant).
  • harvest_fetch_limiter.py — not touched at all.

Honesty note: the throughput claim above is arithmetic derived from the measured per-card fetch/compute split, not a fresh benchmark of this specific change.

Deploy note: this change takes effect only on a rebuild. A catalog pass is currently stopped mid-run at a known cursor awaiting resume — deploying mid-pass would kill that pass. Land this PR; deploy after the pass drains.

Test plan

  • Full backend suite via host venv (/home/ubuntu/.venvs/mpcautofill-pilot/bin/python -m pytest . -q from MPCAutofill/): 3576 passed, 8 skipped, 0 failed in 482.92s (baseline at 8b7f02c was 3571 passed/8 skipped/0 failed; the small passed-count delta is network-dependent live-fetch tests, not a regression — 0 failures either way).
  • _STAGE_C_FETCH_THREADS/_STAGE_C_COMPUTE_WORKERS/_STAGE_C_POOL_QUEUE_DEPTH confirmed via grep: only the fetch-threads value changed (3 → 6); compute workers and queue depth untouched.
  • git diff --stat confirms exactly one file changed: MPCAutofill/cardpicker/stage_e_dispatch.py.
  • python manage.py makemigrations --check --dry-run --skip-checks — "No changes detected".
  • pre-commit hooks (black, isort, ruff, mypy, prettier) — all passed on commit.

CLAUDE.md task-end checks

  • Wiki maintenance: no user-visible or admin-visible behavior changed (backend perf constant only) — no wiki update needed.
  • Extractable-primitives ledger: this is not an extraction and has no dependency changes — no ledger row needed.
  • Constant-rename equivalence tool: not applicable — this changes a constant's value, not its name/location.
  • Docs convention: no doc file needed editing for this change beyond what's referenced above (the compute-profile report already exists and is cited, not modified).

…566)

_STAGE_C_FETCH_THREADS 3 -> 6, matching harvest_fetch_limiter.GOOGLE_IMAGE's
own max_concurrency=6. The rate ceiling is enforced globally through
harvest_rate_coordinator's Postgres cursor, not per-thread, so this cannot
exceed the existing 7/s cap; it only lets fetch use the concurrency the
limiter was already sized for instead of running at less than half of it.
Compute workers and the pool queue depth are untouched - they're a hard cap
tied to a measured 3.25x slowdown at 6-way OCR concurrency
(docs/reports/2026-07-20-pipeline-compute-profile.md).

Not deployed by this change alone - takes effect on next rebuild.
@WilfordGrimley
WilfordGrimley merged commit 2e15718 into master Aug 4, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant