Skip to content

feat(warm_catalog_stats): retire sweep gate as default, compute hourly regardless - #670

Merged
WilfordGrimley merged 1 commit into
masterfrom
feat/warm-catalog-stats-default-compute
Aug 3, 2026
Merged

feat(warm_catalog_stats): retire sweep gate as default, compute hourly regardless#670
WilfordGrimley merged 1 commit into
masterfrom
feat/warm-catalog-stats-default-compute

Conversation

@WilfordGrimley

Copy link
Copy Markdown

Summary

Retires the 2026-07-29 sweep gate as the default behaviour per the owner's 2026-08-03 ruling (reverses the 2026-07-29 gate ruling). The gate was freezing the stats page indefinitely under the streaming micro-batch sweep design, where a RUNNING PilotRunLedger row is perpetually present.

What changed:

  • WARM_CATALOG_STATS_SWEEP_GATE_ENABLED default: TrueFalse
  • Command now computes all five panels every hourly run by default, printing "sweep gate: disabled (default) — computing all five panels" so operators can see which mode ran from the log.
  • The gate is preserved as an opt-in emergency lever: set WARM_CATALOG_STATS_SWEEP_GATE_ENABLED=true in the env to restore the 2026-07-29 skip behaviour.
  • _find_blocking_sweep helper and staleness bound unchanged.

Three-part rationale (verified):

  1. MVCC safety. The five aggregations are plain SELECT queries — they never block on or are blocked by the streaming sweep's tiny micro-batch INSERTs.
  2. Sweep artifacts already filtered. runHistory filters slow-path rows (catalog_stats.py lines ~391–396), and the vote panels count only human sources (HUMAN_SOURCES = USER/ADMIN/FEDERATED), so mid-sweep numbers are stable and correct.
  3. ~9s measured compute. Full compute of all five panels measures ~9s (last ungated run 2026-08-02T16:00:22Z → 16:00:31Z), trivial load once an hour.

Test plan

  • python3 -m pytest MPCAutofill/cardpicker/tests/test_warm_catalog_stats.py -x -q — 5 passed, 0 failed
  • python3 -m pytest MPCAutofill/cardpicker/tests/test_catalog_stats.py -x -q -k "not (TestSharedCacheNotConfigured or TestGetCatalogStatsView)" — 65 passed, 0 failed (2 pre-existing staticfiles-manifest failures excluded from scope)
  • pre-commit run --files on all touched files — black, isort, ruff, mypy, prettier: all Passed
  • CI — not watched by this session per dispatch convention

Tasks-end checks (catalog repo CLAUDE.md):

  • wiki: docs/features/catalog-stats.md — the Sweep-gate section rewritten in place to describe the gate as OFF by default with opt-in re-enablement path.
  • docs: docs/features/catalog-stats.md — updated in-place (this change). No other doc changes needed.
  • extractable-primitives: none — this is a config default flip + command docstring update; snip no new primitive extracted or destroyed, no dependency on vote/consensus/auth.

…y regardless

Retires the 2026-07-29 sweep gate as the default behaviour per the
owner's 2026-08-03 ruling. The gate was freezing the stats page
indefinitely under the streaming micro-batch sweep design, where a
RUNNING PilotRunLedger row is perpetually present.

What changed:
- WARM_CATALOG_STATS_SWEEP_GATE_ENABLED default: True → False
- Command now computes all five panels every hourly run by default,
  printing "sweep gate: disabled (default) — computing all five
  panels" so operators can see which mode ran from the log.
- The gate is preserved as an opt-in emergency lever:
  set WARM_CATALOG_STATS_SWEEP_GATE_ENABLED=true in the env to
  restore the 2026-07-29 skip behaviour.
- _find_blocking_sweep helper and staleness bound unchanged.

Rationale (three-part, verified):
(a) MVCC safety — plain SELECTs never block on streaming
    micro-batch INSERTs.
(b) Existing sweep-artifact filtering — runHistory filters
    slow-path rows (catalog_stats.py lines ~391–396); vote panels
    count only human sources.
(c) ~9s measured full compute, trivial once an hour.

Tests: 5 new (test_warm_catalog_stats.py) + 1 updated existing test.
70 passed, 0 failed (2 pre-existing staticfiles-manifest failures
excluded from scope). Pre-commit (black/isort/ruff/mypy/prettier)
clean on all touched files.
@WilfordGrimley
WilfordGrimley merged commit 9f08b56 into master Aug 3, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant