feat(run_pipeline): add --force-reextract option for bulk re-extraction runs - #667
Merged
Conversation
… unique ledger row - Re-add --skip-stage-c parser argument, accidentally replaced by the --force-reextract block in e505c56 (run_pipeline.py:441 still reads it, the bare-invocation end-to-end test still exercises it). - dispatch_micro_batch: new optional ledger_run_id param decouples the PilotRunLedger row's identity from the run_id stamped on data rows. A multi-batch pass under one --run-id used to hand every dispatch the same id, colliding on the run_id UNIQUE constraint at batch 1 and guaranteeing any whole-catalogue pass died with an IntegrityError. - run_pipeline passes ledger_run_id='<run_id>-<attempt ts>-b<batch>', unique per attempt and per batch, while data rows keep the operator's clean run_id (channel_report scopes by the run_id on the rows). - Regression test: --batch-size 1 over the fixture cohort spans multiple micro-batches; asserts every dispatch ledger row is unique and COMPLETED and ImageEvidence still lands under the clean run_id.
…o-batch ledger row Task-end check from the catalog repo's CLAUDE.md: this PR changes what an ADMIN does (a new run_pipeline flag, and the streaming ledger now carries per-micro-batch suffixed run_ids), so the operational reference is updated in place rather than appended. - The monolith section: --force-reextract overrides the one not-run-scoped Stage C skip (the extractor-manifest already_done_ids filter), forcing a full re-extraction that overwrites evidence in place. - The ledger-convention section: dispatch_micro_batch's ledger_run_id param gives every micro-batch a unique suffixed ledger row while data keeps the operator's clean run_id.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--force-reextracttomanage.py run_pipeline(run_pipeline.py): Stage C'salready_done_idsmanifest filter is the one skip that is NOT run-scoped (it is global by design — evidence already extracted with the current extractor versions needs no redo on any run). The flag clears that filter for the pass, so every eligible card is extracted fresh and itsImageEvidencerow is overwritten in place (keyed on(card_id, content_hash)viaget_or_create). This enables a full bulk re-extraction pass over the whole catalogue.dispatch_micro_batch's new optionalledger_run_idparameter decouples a micro-batch'sPilotRunLedgerrow identity from therun_idstamped on its data rows.PilotRunLedger.run_idis a UNIQUE constraint, andrun_pipelinewas passing the same bare operatorrun_idto every micro-batch — so batch 0 wrote its ledger row and batch 1 (and every later batch) died with anIntegrityError: duplicate key value violates unique constraint "cardpicker_pilotrunledger_run_id_key". Any multi-batch pass under one--run-idcould never finish a whole catalogue. The pipeline now passesledger_run_id=<run_id>-<attempt timestamp>-b<batch num>, unique per attempt and per batch, while data rows keep the operator's cleanrun_id(channel_report scopes by the run_id on the rows). Whenledger_run_idis None (every other caller: the event system,stream_full_catalog,stage_e_shakedown,stream_backstop_sweep), behavior is byte-identical.--skip-stage-c: the--force-reextractblock in the original feature commit accidentally REPLACED the--skip-stage-cparser argument instead of adding alongside it (run_pipeline.py:441 still reads it and the bare-invocation end-to-end test exercises it). This is what broke the Backend-tests check on the original head.--force-reextract; the ledger-convention section documents the per-micro-batch suffixed ledger rows.Test plan
test_a_multi_batch_run_gives_every_micro_batch_a_unique_ledger_row(test_run_pipeline.py):--batch-size 1over the fixture cohort spans multiple micro-batches; asserts the summary ledger row is COMPLETED, every dispatch ledger row is unique and COMPLETED, no dispatch row carries the barerun_id, and ImageEvidence still lands under the clean run_id. This test FAILS on the pre-fix code (IntegrityError on batch 1) and passes after.python3 -m pytest cardpicker/tests/test_run_pipeline.py cardpicker/tests/test_stage_e_dispatch.py -p no:cacheprovider -q— full backend test suite for both touched areas: 106 passed, 230 warnings, ~205s, exit 0.test_a_multi_batch_run_gives_every_micro_batch_a_unique_ledger_row+test_a_bare_invocation_runs_every_stage_and_produces_rows— 2 passed in ~39s.pre-commit run(ruff, isort, black, mypy, prettier) green on the touched Python files.Task-end checks
docs/features/stage-e-operations.md(part of the published docs surface) is updated in this PR. No wiki page name change required.--force-reextractis a flag on an existing command andledger_run_idextends an existing entangled pipeline module. Nodocs/upstreaming/extractable-primitives.mdrow change.