perf(stage-c): raise fetch thread count to limiter's own size (issue #566) - #678
Merged
Merged
Conversation
…566) _STAGE_C_FETCH_THREADS 3 -> 6, matching harvest_fetch_limiter.GOOGLE_IMAGE's own max_concurrency=6. The rate ceiling is enforced globally through harvest_rate_coordinator's Postgres cursor, not per-thread, so this cannot exceed the existing 7/s cap; it only lets fetch use the concurrency the limiter was already sized for instead of running at less than half of it. Compute workers and the pool queue depth are untouched - they're a hard cap tied to a measured 3.25x slowdown at 6-way OCR concurrency (docs/reports/2026-07-20-pipeline-compute-profile.md). Not deployed by this change alone - takes effect on next rebuild.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Raises
_STAGE_C_FETCH_THREADSfrom 3 to 6 inMPCAutofill/cardpicker/stage_e_dispatch.py. This is a one-constant change plus an extended code comment; no other file is touched.harvest_fetch_limiter.GOOGLE_IMAGEis already configuredrate_per_sec=7.0, max_concurrency=6— the limiter was sized for six concurrent fetches, and the fetch pool was running at half that. Fetch threads are I/O-blocked on a socket and cost approximately no CPU, so this does not move host load the way a compute worker would. The rate ceiling stays global regardless of thread count:harvest_rate_coordinator.reserve()paces every fetch through a single cursor row in the shared_cache Postgres table (INSERT ... ON CONFLICT DO UPDATE+clock_timestamp()), so six threads in one process still cannot exceed 7/s — they queue on the cursor, not the socket. The only route past the ceiling is two processes fetching at once, which is a sequencing question, not a thread-count one.At the prior 3 fetch / 3 compute split, fetch was measured as the bottleneck (~3.3/s vs compute's ~4.6/s), so compute workers idled waiting for images. This takes fetch off the critical path and leaves compute as the sole constraint.
Unchanged, deliberately (hard cap, tied to real host-load envelope trips):
_STAGE_C_COMPUTE_WORKERS = 3—docs/reports/2026-07-20-pipeline-compute-profile.mdmeasured 6-way concurrency on this CPU-bound OCR stage running 3.25x SLOWER per card while burning 27.7x more CPU-seconds._STAGE_C_POOL_QUEUE_DEPTH = 6— the backpressure bound that keeps RSS flat (2 × compute workers, follows that constant).harvest_fetch_limiter.py— not touched at all.Honesty note: the throughput claim above is arithmetic derived from the measured per-card fetch/compute split, not a fresh benchmark of this specific change.
Deploy note: this change takes effect only on a rebuild. A catalog pass is currently stopped mid-run at a known cursor awaiting resume — deploying mid-pass would kill that pass. Land this PR; deploy after the pass drains.
Test plan
/home/ubuntu/.venvs/mpcautofill-pilot/bin/python -m pytest . -qfromMPCAutofill/): 3576 passed, 8 skipped, 0 failed in 482.92s (baseline at 8b7f02c was 3571 passed/8 skipped/0 failed; the small passed-count delta is network-dependent live-fetch tests, not a regression — 0 failures either way)._STAGE_C_FETCH_THREADS/_STAGE_C_COMPUTE_WORKERS/_STAGE_C_POOL_QUEUE_DEPTHconfirmed via grep: only the fetch-threads value changed (3 → 6); compute workers and queue depth untouched.git diff --statconfirms exactly one file changed:MPCAutofill/cardpicker/stage_e_dispatch.py.python manage.py makemigrations --check --dry-run --skip-checks— "No changes detected".CLAUDE.md task-end checks