Skip to content

Audit: does the pipeline still catch what it aims to? The composition check - #616

Merged
WilfordGrimley merged 1 commit into
masterfrom
docs/pipeline-coverage-audit
Jul 30, 2026
Merged

Audit: does the pipeline still catch what it aims to? The composition check#616
WilfordGrimley merged 1 commit into
masterfrom
docs/pipeline-coverage-audit

Conversation

@WilfordGrimley

Copy link
Copy Markdown

Read-only audit. No production write, no management command, no migration, no deploy. One new document under docs/reports/, indexed, docs_lint.py --strict clean. No pipeline code changed.

The owner's question: "have we verified that with all of our deletions that the pipeline actually still catches all that it aims to? do we need to do any more reordering or culling? readd anything we dropped?"

Every change of the last three days was verified in isolation. Nothing had checked them in composition. This does: every vote channel, calculator identity, attribute-chip channel and Stage C extractor enumerated from code, measured against production, and traced to the engine that can reach it.

The answers

Q1 — does it still catch everything? No. Three channels are down.

  • Frame-style chips (Old Border/Modern Border) and bleed-edge chips (appropriate-bleed) are at zero machine rows, have no substitute channel, and are unreachable from both the pooled runner and the conveyor. Three independent failures stacked on one channel. 142,633 + 2,786 votes are re-derivable from stored ImageEvidence with no image fetch at all.
  • Border chips survived the same purge only because they were being computed twicelocal-fallback-v1 (now 0) and layout-class-cast-v1 (216,476). Had the duplication been culled first, border colour would be zero too.
  • The known hole is therefore smaller than 53,966 and differently shaped: the border share is covered, the frame and bleed shares are not covered by anything.

Q2 — more reordering or culling? Yes to both.

  • _slow_path_eligible_cards_queryset has no exclusion for the illustration calculator's identity. The code comment admits the wiring "would need" it. A card illustration resolves is still routed to a human reviewer in the same run. PR Run-scoped eligibility, the value-comparing split, and an archive for superseded votes #604 does not close this; it fires on the first stage-d-illustration-v2 run.
  • Four Stage D readers gate on one extractor_versions key and then read six extractors' fields ungated. Two degrade in the strict direction — a missing artist_ocr makes illus_anchor_fired read False, classifies every frame modern, and can veto a genuine old-frame card as frame-mismatch, which is deliberately not rescannable. Permanent wrong skip. The correct pattern (REQUIRED_EXTRACTOR_KEYS) already exists three times in the same codebase.
  • image_evidence.extract_card_evidence has zero production callers — only tests. Both engines call compute_card_evidence directly. It is the only non-pilot caller of cast_border_attribute_vote. Exactly the defect shape PR Wire collector-line artist recovery into the pooled Stage C cohort runner #581 fixed for artist recovery.

Q3 — what to re-add? Frame-style chips (142,633), bleed-edge chips (2,786), border-chip arrears (~2,648). Nothing else — no other purge removed something nothing re-derives.

Also established

  • local-name-frequency-v1 and scryfall-tagger-v1 have zero PilotRunLedger rows, ever. Never run, not merely dormant — that closes the open retire-or-keep diagnosis with a fact.
  • The pooled runner writes no votes at all; every Stage D identity is conveyor-only. Only 1 of 21 vote channels is reachable from it, and it writes no votes.
  • docs/reference/skip-reasons.md was checked value by value against production. It holds unchanged — the one roster in this audit that survived contact with production.
  • 10,127 Card rows have no ImageEvidence row at all.

Every zero is classified into one of the four states the brief asked to be kept apart (purged-with-no-substitute / purged-but-superseded / wired-but-never-run / deliberately-retired-and-replaced), and every finding is marked LIVE or LATENT against the deployed build.

🤖 Generated with Claude Code

https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN

… production

Every change of the last three days was verified in isolation; none was
verified in composition. This enumerates every vote channel, calculator
identity, attribute-chip channel and Stage C extractor from code, measures
each against production, and traces each to the engine that can reach it.

Four findings, ranked:

1. Frame-style (Old/Modern Border) and bleed-edge (appropriate-bleed) chips
   are at ZERO machine rows with no substitute channel AND no wiring into
   either engine. 142,633 + 2,786 votes are re-derivable from stored
   ImageEvidence with no image fetch. Border chips survived the same purge
   only because they were being computed twice.

2. image_evidence.extract_card_evidence has zero production callers - only
   tests. Both engines call compute_card_evidence directly. It is the only
   non-pilot caller of cast_border_attribute_vote. Same defect shape as the
   artist recovery PR #581 fixed.

3. _slow_path_eligible_cards_queryset has no exclusion for the illustration
   calculator's identity; the code comment says the wiring "would need" it.
   A card illustration resolves is still routed to a human. PR #604 does not
   close this.

4. Four Stage D readers gate on one extractor key and then read six
   extractors' fields ungated. Two degrade in the strict direction, one into
   a permanently non-rescannable frame-mismatch skip.

Also: local-name-frequency-v1 and scryfall-tagger-v1 have zero PilotRunLedger
rows ever - never run, not merely dormant. docs/reference/skip-reasons.md was
checked value by value against production and holds unchanged.

READ-ONLY: no write, no management command, no migration, no deploy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant