feat(warm_catalog_stats): retire sweep gate as default, compute hourly regardless - #670
Merged
Merged
Conversation
…y regardless
Retires the 2026-07-29 sweep gate as the default behaviour per the
owner's 2026-08-03 ruling. The gate was freezing the stats page
indefinitely under the streaming micro-batch sweep design, where a
RUNNING PilotRunLedger row is perpetually present.
What changed:
- WARM_CATALOG_STATS_SWEEP_GATE_ENABLED default: True → False
- Command now computes all five panels every hourly run by default,
printing "sweep gate: disabled (default) — computing all five
panels" so operators can see which mode ran from the log.
- The gate is preserved as an opt-in emergency lever:
set WARM_CATALOG_STATS_SWEEP_GATE_ENABLED=true in the env to
restore the 2026-07-29 skip behaviour.
- _find_blocking_sweep helper and staleness bound unchanged.
Rationale (three-part, verified):
(a) MVCC safety — plain SELECTs never block on streaming
micro-batch INSERTs.
(b) Existing sweep-artifact filtering — runHistory filters
slow-path rows (catalog_stats.py lines ~391–396); vote panels
count only human sources.
(c) ~9s measured full compute, trivial once an hour.
Tests: 5 new (test_warm_catalog_stats.py) + 1 updated existing test.
70 passed, 0 failed (2 pre-existing staticfiles-manifest failures
excluded from scope). Pre-commit (black/isort/ruff/mypy/prettier)
clean on all touched files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retires the 2026-07-29 sweep gate as the default behaviour per the owner's 2026-08-03 ruling (reverses the 2026-07-29 gate ruling). The gate was freezing the stats page indefinitely under the streaming micro-batch sweep design, where a RUNNING PilotRunLedger row is perpetually present.
What changed:
WARM_CATALOG_STATS_SWEEP_GATE_ENABLEDdefault:True→False"sweep gate: disabled (default) — computing all five panels"so operators can see which mode ran from the log.WARM_CATALOG_STATS_SWEEP_GATE_ENABLED=truein the env to restore the 2026-07-29 skip behaviour._find_blocking_sweephelper and staleness bound unchanged.Three-part rationale (verified):
SELECTqueries — they never block on or are blocked by the streaming sweep's tiny micro-batchINSERTs.runHistoryfilters slow-path rows (catalog_stats.pylines ~391–396), and the vote panels count only human sources (HUMAN_SOURCES = USER/ADMIN/FEDERATED), so mid-sweep numbers are stable and correct.Test plan
python3 -m pytest MPCAutofill/cardpicker/tests/test_warm_catalog_stats.py -x -q— 5 passed, 0 failedpython3 -m pytest MPCAutofill/cardpicker/tests/test_catalog_stats.py -x -q -k "not (TestSharedCacheNotConfigured or TestGetCatalogStatsView)"— 65 passed, 0 failed (2 pre-existing staticfiles-manifest failures excluded from scope)pre-commit run --fileson all touched files — black, isort, ruff, mypy, prettier: all PassedTasks-end checks (catalog repo CLAUDE.md):
docs/features/catalog-stats.md— the Sweep-gate section rewritten in place to describe the gate as OFF by default with opt-in re-enablement path.docs/features/catalog-stats.md— updated in-place (this change). No other doc changes needed.