Skip to content

Retrofit fetch-thread-pool + compute-process-pool into _run_stage_c (issue #566) #669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WilfordGrimley merged 3 commits into from
Aug 3, 2026
Merged

Retrofit fetch-thread-pool + compute-process-pool into _run_stage_c (issue #566) #669

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dfd2e61
Upstream drift log: weekly refresh
github-actions[bot] Aug 3, 2026
d45011d
fix(stage-c): drain in-flight compute on lockout, fix pool DB connect…
WilfordGrimley Aug 3, 2026
bf4bf90
fix(stage-c): delete inline pooled-compute test twin, run real convey…
WilfordGrimley Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
597 changes: 323 additions & 274 deletions MPCAutofill/cardpicker/stage_e_dispatch.py
View file
Open in desktop

Large diffs are not rendered by default.

497 changes: 396 additions & 101 deletions MPCAutofill/cardpicker/tests/test_stage_e_dispatch.py
View file
Open in desktop

Large diffs are not rendered by default.

12 changes: 12 additions & 0 deletions MPCAutofill/cardpicker/tests/test_stage_e_shakedown.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,22 @@
from cardpicker.models import EnvelopeTrip, ImageEvidence, PilotRunLedger
from cardpicker.stage_e_concurrency import _LOCK_NAMESPACE
from cardpicker.tests.factories import CardFactory, ImageEvidenceFactory, SourceFactory
from cardpicker.tests.test_stage_e_dispatch import _SyncStagePoolStub

STREAMING_ON = override_settings(STAGE_E_STREAMING_ENABLED=True)


@pytest.fixture(autouse=True)
def _sync_stage_c_pools(monkeypatch: pytest.MonkeyPatch) -> None:
"""This driver dispatches through the real `dispatch_micro_batch` -> `_run_stage_c` for every
test in this module that isn't stubbed at the `dispatch_micro_batch` boundary itself - see
`_SyncStagePoolStub`'s own docstring in `test_stage_e_dispatch.py` for why the real
`ThreadPoolExecutor`/`ProcessPoolExecutor` module names must be replaced rather than left real
in a test process."""
monkeypatch.setattr(stage_e_dispatch, "ThreadPoolExecutor", _SyncStagePoolStub)
monkeypatch.setattr(stage_e_dispatch, "ProcessPoolExecutor", _SyncStagePoolStub)


def _blank_tail_evidence(card: Any, **overrides: Any) -> ImageEvidence:
"""A CURRENT, full-manifest ImageEvidence row carrying the Bug-A tail's own signature - the
same "every field blank" shape `test_stage_e_dispatch.py::_full_evidence` builds, kept as its
Expand Down
12 changes: 12 additions & 0 deletions MPCAutofill/cardpicker/tests/test_stream_full_catalog.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,22 @@
ImageEvidenceFactory,
SourceFactory,
)
from cardpicker.tests.test_stage_e_dispatch import _SyncStagePoolStub

STREAMING_ON = override_settings(STAGE_E_STREAMING_ENABLED=True)


@pytest.fixture(autouse=True)
def _sync_stage_c_pools(monkeypatch: pytest.MonkeyPatch) -> None:
"""This driver dispatches through the real `dispatch_micro_batch` -> `_run_stage_c` for every
test in this module that isn't stubbed at the `dispatch_micro_batch` boundary itself (via
`_install_recording_dispatch`) - see `_SyncStagePoolStub`'s own docstring in
`test_stage_e_dispatch.py` for why the real `ThreadPoolExecutor`/`ProcessPoolExecutor` module
names must be replaced rather than left real in a test process."""
monkeypatch.setattr(stage_e_dispatch, "ThreadPoolExecutor", _SyncStagePoolStub)
monkeypatch.setattr(stage_e_dispatch, "ProcessPoolExecutor", _SyncStagePoolStub)


def _exit_code(*argv: Any, **kwargs: Any) -> int:
"""Run the command and return THE EXIT CODE a supervisor would actually observe.

Expand Down
14 changes: 7 additions & 7 deletions docs/upstreaming/drift-log.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ rebase a branch onto the new upstream tip (see
`conventions.md`'s automation-composition note — this workflow
never rebases anything itself).

Last run: 2026-07-27 11:25 UTC, upstream/master @ c3d10253e581a1e4fc52b7c4efd9d150b00de026
Last run: 2026-08-03 11:26 UTC, upstream/master @ c5220cf0840e9c6d8f7da30e22877a7c1d5432ea

| Branch | Applies clean onto upstream/master? | Upstream commits since fork point | Files upstream touched that this branch also touches | Last checked |
| --- | --- | --- | --- | --- |
| `upstream-feat-local-file-source` | yes | 0 | 0 file(s) | 2026-07-27 |
| `upstream-fix-frontend-searchable-the` | yes | 1 | 0 file(s) | 2026-07-27 |
| `upstream-fix-image-cdn-cors` | yes | 2 | 0 file(s) | 2026-07-27 |
| `upstream-fix-pdf-canvas-preview` | yes | 2 | 0 file(s) | 2026-07-27 |
| `upstream-fix-pdf-eager-wasm-load` | yes | 2 | 0 file(s) | 2026-07-27 |
| `upstream-fix-pdf-thumbnail-worker-route` | yes | 2 | 0 file(s) | 2026-07-27 |
| `upstream-feat-local-file-source` | yes | 14 | 0 file(s) | 2026-08-03 |
| `upstream-fix-frontend-searchable-the` | yes | 15 | 0 file(s) | 2026-08-03 |
| `upstream-fix-image-cdn-cors` | yes | 16 | 0 file(s) | 2026-08-03 |
| `upstream-fix-pdf-canvas-preview` | yes | 16 | 1 file(s) | 2026-08-03 |
| `upstream-fix-pdf-eager-wasm-load` | yes | 16 | 0 file(s) | 2026-08-03 |
| `upstream-fix-pdf-thumbnail-worker-route` | yes | 16 | 0 file(s) | 2026-08-03 |
2 changes: 1 addition & 1 deletion image-cdn/wrangler.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,4 @@ simple = { limit = 12000, period = 10 }
[[ratelimits]]
name = "IMAGE_FULL_TIER_RATE_LIMITER"
namespace_id = "1002"
simple = { limit = 30, period = 10 }
simple = { limit = 60, period = 10 }
Loading