Skip to content

feat(run_pipeline): add --force-reextract option for bulk re-extraction runs - #667

Merged
WilfordGrimley merged 3 commits into
masterfrom
feat/force-reextract-feature
Jul 31, 2026
Merged

feat(run_pipeline): add --force-reextract option for bulk re-extraction runs#667
WilfordGrimley merged 3 commits into
masterfrom
feat/force-reextract-feature

Conversation

@WilfordGrimley

@WilfordGrimley WilfordGrimley commented Jul 31, 2026

Copy link
Copy Markdown

Summary

  • Adds --force-reextract to manage.py run_pipeline (run_pipeline.py): Stage C's already_done_ids manifest filter is the one skip that is NOT run-scoped (it is global by design — evidence already extracted with the current extractor versions needs no redo on any run). The flag clears that filter for the pass, so every eligible card is extracted fresh and its ImageEvidence row is overwritten in place (keyed on (card_id, content_hash) via get_or_create). This enables a full bulk re-extraction pass over the whole catalogue.
  • Fixes a second, pre-existing bug the full-bulk pass would have hit immediately: dispatch_micro_batch's new optional ledger_run_id parameter decouples a micro-batch's PilotRunLedger row identity from the run_id stamped on its data rows. PilotRunLedger.run_id is a UNIQUE constraint, and run_pipeline was passing the same bare operator run_id to every micro-batch — so batch 0 wrote its ledger row and batch 1 (and every later batch) died with an IntegrityError: duplicate key value violates unique constraint "cardpicker_pilotrunledger_run_id_key". Any multi-batch pass under one --run-id could never finish a whole catalogue. The pipeline now passes ledger_run_id=<run_id>-<attempt timestamp>-b<batch num>, unique per attempt and per batch, while data rows keep the operator's clean run_id (channel_report scopes by the run_id on the rows). When ledger_run_id is None (every other caller: the event system, stream_full_catalog, stage_e_shakedown, stream_backstop_sweep), behavior is byte-identical.
  • Restores --skip-stage-c: the --force-reextract block in the original feature commit accidentally REPLACED the --skip-stage-c parser argument instead of adding alongside it (run_pipeline.py:441 still reads it and the bare-invocation end-to-end test exercises it). This is what broke the Backend-tests check on the original head.
  • Docs updated in place (features/stage-e-operations.md): the monolith section documents --force-reextract; the ledger-convention section documents the per-micro-batch suffixed ledger rows.

Test plan

  • New regression test test_a_multi_batch_run_gives_every_micro_batch_a_unique_ledger_row (test_run_pipeline.py): --batch-size 1 over the fixture cohort spans multiple micro-batches; asserts the summary ledger row is COMPLETED, every dispatch ledger row is unique and COMPLETED, no dispatch row carries the bare run_id, and ImageEvidence still lands under the clean run_id. This test FAILS on the pre-fix code (IntegrityError on batch 1) and passes after.
  • python3 -m pytest cardpicker/tests/test_run_pipeline.py cardpicker/tests/test_stage_e_dispatch.py -p no:cacheprovider -q — full backend test suite for both touched areas: 106 passed, 230 warnings, ~205s, exit 0.
  • Focused re-run after the black reformat: test_a_multi_batch_run_gives_every_micro_batch_a_unique_ledger_row + test_a_bare_invocation_runs_every_stage_and_produces_rows — 2 passed in ~39s.
  • pre-commit run (ruff, isort, black, mypy, prettier) green on the touched Python files.
  • Docs lint not yet run on this head (docs_lint.py runs in CI); the edit was made in place in the existing reference file, no new doc files added.
  • CI — not watched by this session per dispatch convention.

Task-end checks

  • Wiki: no user-facing page changes — this is operator tooling; docs/features/stage-e-operations.md (part of the published docs surface) is updated in this PR. No wiki page name change required.
  • Extractable-primitives ledger: no primitive created or destroyed — --force-reextract is a flag on an existing command and ledger_run_id extends an existing entangled pipeline module. No docs/upstreaming/extractable-primitives.md row change.
  • Docs updated where appropriate (see Summary); the operational reference was edited in place per the docs convention.

… unique ledger row

- Re-add --skip-stage-c parser argument, accidentally replaced by the
  --force-reextract block in e505c56 (run_pipeline.py:441 still reads it,
  the bare-invocation end-to-end test still exercises it).
- dispatch_micro_batch: new optional ledger_run_id param decouples the
  PilotRunLedger row's identity from the run_id stamped on data rows.
  A multi-batch pass under one --run-id used to hand every dispatch the
  same id, colliding on the run_id UNIQUE constraint at batch 1 and
  guaranteeing any whole-catalogue pass died with an IntegrityError.
- run_pipeline passes ledger_run_id='<run_id>-<attempt ts>-b<batch>',
  unique per attempt and per batch, while data rows keep the operator's
  clean run_id (channel_report scopes by the run_id on the rows).
- Regression test: --batch-size 1 over the fixture cohort spans multiple
  micro-batches; asserts every dispatch ledger row is unique and COMPLETED
  and ImageEvidence still lands under the clean run_id.
…o-batch ledger row

Task-end check from the catalog repo's CLAUDE.md: this PR changes what an
ADMIN does (a new run_pipeline flag, and the streaming ledger now carries
per-micro-batch suffixed run_ids), so the operational reference is updated
in place rather than appended.

- The monolith section: --force-reextract overrides the one not-run-scoped
  Stage C skip (the extractor-manifest already_done_ids filter), forcing a
  full re-extraction that overwrites evidence in place.
- The ledger-convention section: dispatch_micro_batch's ledger_run_id
  param gives every micro-batch a unique suffixed ledger row while data
  keeps the operator's clean run_id.
@WilfordGrimley
WilfordGrimley merged commit 06d936b into master Jul 31, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant