THE MONOLITH: one command that runs the whole pipeline end to end (#618)

[WilfordGrimley]

What this is

One command that runs the whole identification pipeline end to end. Owner brief: "1 click". Every stage already existed and every stage was already run separately; what did not exist was anything that ran them together, in order, under one run_id.

manage.py run_pipeline

is a complete, working, from-scratch, whole-catalogue run that writes. No flag is required.

Stage 0   Scryfall reference refresh, once, at the front
Stage E   operating-envelope preflight
Stage C   evidence extraction (the pooled engine)
Stage D   join-key → fallback → illustration → slow-path, then the three attribute chips
Stage C+  distance-0 cluster vote propagation
Stage E   fidelity gate — machine-only resolutions must be zero
end       channel_report

This is assembly, not invention. run_pipeline.py contains no pipeline logic: it is imports, sequencing, run_id threading and error handling. Every stage is reached by calling the module that already owned it.

Wiring, per stage

stage	what is called	notes
0	stream_full_catalog.run_stage_zero_freshness → printing_metadata_import	full-set value-diffing upsert, so re-running IS the backfill
E	operating_envelope.current_trip / check_envelope, stage_e_dispatch._sample_envelope_signals	current_trip first, no self-resume
C	run_image_evidence_cohort via call_command	pooled engine, run-scoped resume (#645), md5 evidence transfer, RSS guard — none of it re-derived
D	stage_e_dispatch._run_stage_d, batch_ids=None	the one place the dependency order lives; carries the three chip casters from #654
C+	local_clustering.compute_two_threshold_clusters + local_identify_printing_tags.build_propagated_cluster_votes	the pilot capability no engine had
E	local_identify_printing_tags.verify_zero_resolutions	the same gate local_calculate_verdicts runs between calculators
end	channel_report via call_command	reported, never folded into the exit status

Two thin extractions, no copies

Both lift an existing body to a module-level callable and have the original call it. Behaviour is unchanged.

1. stream_full_catalog.Command._run_stage_zero_freshness → module-level run_stage_zero_freshness(require_fresh, is_resume, write, warn). It now also returns the bulk-file vintage (remote updated_at, cache path, cache mtime age, whether it refreshed, import stats), which the monolith records on its ledger row so a run's conclusions can be dated. stream_full_catalog's method is a wrapper that discards it and prints exactly what it always printed.
2. run_pilot's propagate_cluster_vote closure → module-level build_propagated_cluster_votes(...), returning rows instead of writing them. It closed over six pieces of run_pilot-local state and was reachable from nothing; the closure now calls it and keeps its own batching and written-id ledgers.

Three parameters added to existing functions

stage_e_dispatch._run_stage_d / _run_attribute_chip_casters / _run_illustration_calculator:

• batch_ids: Optional[list[int]] — None is bulk mode, which every calculator underneath already supported. One Stage D now serves both engines.
• dry_run: bool = False — threaded through, was hard-coded False.

Both defaults preserve existing behaviour byte-for-byte for the conveyor.

Write polarity — the organ table

Owner ruling: "the eventual intention for the monolith is that default is to write and flags are what prevents it. (opposite)". This is the inverse of every organ called. Inheriting their defaults is the dangerous failure: a pass that computes everything, logs every card, reports success and persists nothing is indistinguishable from a working run except by row counts.

organ	its own gate	what run_pipeline passes
import_scryfall_printing_metadata	none — always writes	called; skipped under --dry-run (no preview mode)
run_image_evidence_cohort	--dry-run (already write-first)	forwards --dry-run; nothing overridden
run_join_key_calculator	dry_run=True default	dry_run=False
run_fallback_calculator	dry_run=True default	dry_run=False
run_illustration_calculator	dry_run=True default	dry_run=False
run_slow_path_calculator	dry_run=True default	dry_run=False
run_layout_class_cast (border)	dry_run=True default	dry_run=False
run_attribute_chip_cast (frame + bleed)	dry_run=True default	dry_run=False
cluster vote propagation	none (this PR's own write)	gated on not dry_run

enforce_dry_run_precondition (the forced dry-run, #362) is deliberately not added: a write-by-default command has no --write to gate, and requiring a prior dry-run would put a flag back in front of the working run. Verified it only arms inside Stage C's --card-ids-file path — write_mode=(not dry_run) and bool(card_ids_file_for_scope) — so a bare run never trips it. --skip-dryrun-check is forwarded for the file path. Pinned by test_a_bare_run_never_trips_the_forced_dry_run_precondition.

--dry-run is a real pass that withholds the write, not a plan: every stage executes, every calculator reports what it would cast, and channel_report still runs. Exits 0.

Organs not wired (not in the brief's sequence): local_lands_identify, local_detect_ai_art, local_residual_classify, local_name_frequency_elimination (see finding 1), run_pilot's OCR/phash voting (standing owner deferral).

Verification

End-to-end test: cardpicker/tests/test_run_pipeline.py, 22 tests, drives the whole command over a 3-card fixture cohort. Only the network is stubbed (_fetch_one_card, _compute_one_card, and Stage 0's real download). Every Stage D calculator, all three chip casters, clustering, propagation, the gate and channel_report run for real against the test DB.

Mutation table — every stage unwired at the source, in turn:

#	mutation	result
M1	Stage 0 not called	🔴 2 failed
M2	envelope preflight not called	🔴 1 failed
M3	Stage C not called	🔴 3 failed
M4	Stage D not called	🔴 2 failed
M5	attribute chips not called	🔴 2 failed
M6	cluster propagation write removed	🔴 2 failed
M7	fidelity gate not called	🔴 1 failed
M8	channel_report not called	🔴 1 failed
M9	Stage D reverts to its organs' dry_run=True default	🔴 2 failed
M10	--dry-run leaks a write through Stage C	🔴 1 failed
—	baseline, unmutated	🟢 22 passed

M7 and M8 initially survived and the assertions were strengthened: violations == 0 is what you get whether the gate found nothing or was never called, and the CHANNEL REPORT banner is printed by run_pipeline itself either way. Both now assert a string only the stage can emit.

docs_lint.py --strict: clean. Full cardpicker/tests/ suite: 3541 passed, 8 skipped. CI: 13/13 green.

One cross-test defect found and fixed on the way: test_run_pipeline.py passed alone and halted every test at the envelope preflight inside the full suite. stage_e_dispatch._window is a process-global rolling fetch-outcome window that _sample_envelope_signals reads, and four fetch failures recorded by test_stage_e_dispatch.py earlier in the same pytest process tripped fetch_failure_rate at 4/4. Same autouse reset fixture that file already carries.

Findings — recorded, not fixed

Per the brief: this command's first run is how we discover what is broken, and changing what it measures while building it means the first reading cannot be trusted.

1. run_name_frequency_elimination is a census predicate that leaks across runs. Its "exactly one unresolved eligible card for this name" test counts over _eligible_base_queryset called with no run_id (local_identify_printing_tags.py:2013), so the pool is depleted by its own prior votes. A name that gains a second card after run 1 can have that card voted on in run 2 where an empty-vote catalogue would abstain — a fresh wrong positive assertion, not a carried-forward one. It has never run in production, so the fix window is open. Not wired into the monolith.
2. Only CardPrintingTag is archived on supersession. models.vote_archive_model maps that one model; CardTagVote (all three chip families) and CardIllustrationVote are deleted outright. So the printing grain stays diffable across generations via --generation-diff; the tag and illustration grains do not. CardIllustrationVote is additionally the only production caller of purge_stale_machine_votes that bypasses vote_write.purge_and_write_votes, so it would never record superseded_by_run_id even if registered.
3. No retention janitor exists (Retention janitor: sweep the oldest runs of CardScanLog abstention history #575 is referenced in four places as forward-looking prose only). ArchivedCardPrintingTag has no deletion path at all except FK cascade. Nothing can sweep this run's rows.
4. The attribute-chip and layout casters have no run_id scoping and no value-comparison split, so a card whose evidence is later re-extracted can never have its chip re-read under a new run. Monotonic suppression, not a false fire. Fresh surface — both were only wired into an engine in Wire the attribute-chip casters into an engine; strip the vote from the uncalled wrapper #654.
5. _split_new_printing_tag_votes compares VALUES, so an identical recomputed verdict is skipped and the original row survives with its original run_id. Correct and desirable, but it means filter(run_id=R) returns what R changed, not what R concluded.
6. PilotRunLedger.run_id is UNIQUE, so the monolith and its delegated Stage C cannot share a row. The data keeps the clean run_id and this command's summary row takes a -pipeline suffix — because channel_report scopes a channel's counts by the run_id on the rows, and a Stage C whose evidence carried a different run_id from the votes would read as a silent channel.
7. Three write-gate conventions across ten commands (--write opt-in, --dry-run opt-out, and both plus enforce_dry_run_precondition). There is no single convention to inherit, which is what makes this surface error-prone even after this PR pins it. Noted, not fixed.
8. bleed_diff_mm needs no backfill. image_evidence.compute_card_evidence already calls compute_bleed_diff_mm unconditionally (image_evidence.py:976). It was 97.9% NULL only because the field was added without an extractor version bump, so no existing row was re-extracted. A from-scratch run fills it.
9. Stage C's writes still fire the evidence-change post_save echo. suppress_evidence_change_echo is a contextvars flag set by the dispatch loop; bulk-mode writes are deliberately unflagged, and the pooled engine's writes happen in worker processes/threads anyway. Since Stage D is invoked explicitly, the echo is redundant work rather than a correctness problem, and it is inert unless a django-q cluster is running with STAGE_E_STREAMING_ENABLED.
10. docs/features/stage-e-operations.md is still not in .github/wiki-publish-map.json (issue 11 docs/features pages are absent from the wiki publish map, including stage-e-operations.md #657). Not fixed here, per the brief. Nothing enforces the absence — check_wiki_publish_map only validates listed entries.

What is thin

• Stage E is a preflight, not a running envelope. The trip check happens once, before Stage C. Rate pressure and the global 7/s ceiling apply per request underneath Stage C (harvest_fetch_limiter / harvest_rate_coordinator) and are unchanged, but the monolith does not re-sample host load mid-pass the way the conveyor does per micro-batch.
• Cluster propagation carries Stage D's verdicts, not the pilot's OCR/phash votes — see DEVIATIONS below.
• Stage D runs in one bulk pass, so a kill loses that pass's uncommitted work. Stage C's within-run resume is unaffected.

Deviations

1. Cluster propagation propagates Stage D's printing verdicts, not the pilot's OCR/phash votes. The brief says carry the clustering in and explicitly says not to carry pilot OCR/phash voting. Those two together leave no pilot vote to propagate, so the monolith propagates the votes the run actually cast, under the same identity, with the same already-voted guard. Same rule, different upstream.
2. The monolith clusters over Card by stored content_phash (the whole catalogue), not over a selection pool. run_pilot clusters over its eligibility-narrowed pool, which makes membership a function of what earlier runs voted on — a genuine 3-card cluster can present as a 2-card one, and dropping the lowest-pk member changes which card becomes representative. Clustering over the catalogue is what makes the answer independent of run history.
3. stage_e_dispatch._run_stage_d gained two parameters rather than the monolith re-listing its six calls. Re-listing would have been a second copy of the load-bearing dependency order.

Docs

docs/identification-pipeline.md and docs/features/stage-e-operations.md — the living pages, edited in place. No dated report.

Live state

Nothing deployed, nothing run against production. Branch feat/monolith-one-command; no migrations (checked origin/master's migration tree immediately before committing — none needed).

🤖 Generated with Claude Code

https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN