From 8cef20db6556fe9963c4f0212278da143ecafa47 Mon Sep 17 00:00:00 2001 From: wilfordgrimley <2397930+WilfordGrimley@users.noreply.github.com> Date: Wed, 22 Jul 2026 20:02:40 +0000 Subject: [PATCH 1/2] Add Stage D fallback channel calculator + deductive-backfill exclusion guard Pre-fire prep (owner-bundled ahead of the full-catalog Stage D fire, code only): ports local_fallback.py's border/artist/symbol evidence-combination model to Stage D (calculate_fallback_verdict/run_fallback_calculator, own anonymous_id) for cards the join-key calculator found no confident hit for, and adds the missing deductive-backfill exclusion (constant #3 from the pipeline-fidelity gate) to the shared eligible-cards queryset so a repeated Stage D fire never re-votes a card the backfill already covered. Co-Authored-By: Claude Fable 5 --- .../cardpicker/local_calculate_verdicts.py | 436 +++++++++++++++++- .../commands/local_calculate_verdicts.py | 70 ++- .../tests/test_local_calculate_verdicts.py | 410 +++++++++++++++- docs/features/catalog-completion-plan.md | 50 ++ docs/pipeline-fidelity-gate.md | 10 +- .../reports/2026-07-22-knowledge-inventory.md | 48 +- 6 files changed, 982 insertions(+), 42 deletions(-) diff --git a/MPCAutofill/cardpicker/local_calculate_verdicts.py b/MPCAutofill/cardpicker/local_calculate_verdicts.py index 9eac307ae..d62063c2b 100644 --- a/MPCAutofill/cardpicker/local_calculate_verdicts.py +++ b/MPCAutofill/cardpicker/local_calculate_verdicts.py @@ -197,6 +197,98 @@ `CanonicalCard`/`CanonicalPrintingMetadata`/`DFCPair` DB fixtures, not a live fetch - Stage D consumes stored evidence + Scryfall-backed models, it never touches a live image, so Stage C's golden-set convention of a real network fetch over 30 pinned cards doesn't apply here). + +PRE-FIRE PREP (this PR, owner-bundled ahead of the full-catalog Stage D fire - two pieces, both +code-only, neither runs any prod extraction/write): + +PIECE 1 - THE FALLBACK CHANNEL CALCULATOR (`calculate_fallback_verdict`/`run_fallback_calculator`, +own `anonymous_id="stage-d-fallback-v1"`): Stage D's own port of `local_fallback.py`'s pilot +"Pass 2" evidence-combination model (that module's own docstring: "fires only when pass 1 +(OCR/phash) yields no accepted vote for a card") - here, "pass 1" is the join-key calculator +above, and this calculator is Stage D's own pass 2, run over exactly the cards the join-key +calculator already concluded have no confident hit (the SAME population +`run_slow_path_calculator` already routes to human review - `_fallback_eligible_cards_queryset` +reuses that exact eligibility shape). Unlike the pilot's own `run_fallback_for_card` (which crops +and phash-scans a LIVE `PIL.Image`), this calculator operates ENTIRELY off already-persisted +`ImageEvidence` fields - it never re-fetches an image (this file never has - "we index, we do not +store images", CLAUDE.md's Governing premise): + + - border sub-check: `evidence.layout_class` (Stage C's own `classify_border_color` output, + PROTECTED CORE, already computed) fed straight into `local_fallback.filter_by_border_color` + (PROTECTED CORE, called not modified) - identical to the pilot's own border filter, just fed a + pre-computed reading instead of a fresh pixel sample. + - artist sub-check: `evidence.artist_ocr_name` (Stage C's own `artist_ocr` extractor, which + itself calls `local_fallback.extract_artist_name` - the SAME "Illus. " extraction the + pilot's own `detect_illus_anchor` performs, confirmed by reading `image_evidence.py`'s own + `artist_ocr` section) fed straight into `local_fallback.match_artist` (PROTECTED CORE, called + not modified). + - symbol sub-check (`_filter_by_symbol_phash`): the pilot's own `find_symbol_matches` scans a + live crop against a rendered keyrune glyph via phash; here, `evidence.symbol_phash` (Stage C's + own precomputed region hash) is compared to each candidate's DISTINCT expansion's rendered + glyph (`local_fallback.render_set_symbol`, PROTECTED CORE, called not modified) via the SAME + pure-Hamming-distance-arithmetic reimplementation `_symbol_phash_tiebreak` above already + established for the join-key calculator's own symbol tie-break (`SYMBOL_DISTANCE_THRESHOLD`/ + `SYMBOL_MARGIN`, PROTECTED CORE constants, reused verbatim) - duplicated rather than shared + with `_symbol_phash_tiebreak` (this module's own "duplicate the arithmetic, reimplement + nothing decision-shaped" convention, same reasoning `JOIN_KEY_CONFIDENCE_BOTH`'s own comment + already gives) because the two return different shapes: `_symbol_phash_tiebreak` returns one + winning `CandidatePrinting` for its own tie-break call site, `_filter_by_symbol_phash` returns + the FULL SET of surviving candidate pks, mirroring `find_symbol_matches`'s own return + convention and this calculator's own filter-intersection model. + + A vote is cast ONLY when the intersection across every sub-check that DID produce a reading + narrows to EXACTLY ONE candidate - `local_fallback.py`'s own documented rule, reproduced exactly, + never loosened. No agreement/corroboration layer (frame/copyright-year/truncated-image) is + applied here, unlike the join-key calculator above - the task scope is a FAITHFUL port of + `local_fallback.py`'s own decision model, not an augmented one; `local_fallback.py` itself never + performed those checks, so adding them here would not be "reproducing local-fallback-v1's + decision" as scoped. `FALLBACK_CONFIDENCE_MULTI_EVIDENCE`/`FALLBACK_CONFIDENCE_SINGLE_EVIDENCE` + (imported from `local_fallback`, not duplicated - these ARE the pilot's own exact values, not a + new Stage-D-specific tier) are used verbatim, unlike the join-key calculator's own brand-new + confidence constants. + + `source=VoteSource.OCR` (not `VoteSource.DEDUCTION`): `VoteSource`'s own docstring in + `models.py` explicitly names "the border/artist/symbol evidence-combination fallback" as part of + OCR's own umbrella definition ("everything in `cardpicker.local_identify_printing_tags`/ + `local_fallback` that actually looks at the card image") - this calculator inspects image-derived + evidence (border/artist/symbol readings), it does not perform `deductive_backfill.py`'s own + "pure logical inference from already-trusted structured data, zero image inspection." Machine + weight (`PRINTING_TAG_MACHINE_WEIGHT`) either way - the human-backed consensus gate in + `vote_consensus.resolve_weighted_consensus` (PROTECTED CORE, unmodified) applies identically + regardless of which `VoteSource` value is used, so this is a naming-precision choice, not a + soundness one. + + Wired into `run_slow_path_calculator`'s own eligibility query + (`_slow_path_eligible_cards_queryset`): a card this calculator successfully votes on is excluded + from slow-path routing (an additional exclusion alongside its own pre-existing ones) - otherwise, + within the SAME invocation, slow-path would route a card to human review that this calculator + resolves moments later, since the management command runs join-key -> fallback -> slow-path in + that order. A card this calculator merely SCANNED but abstained on (no-evidence/eliminated/ + ambiguous) is NOT excluded - it still has no confident automated hit and belongs in the review + queue exactly as before. + +PIECE 2 - CONSTANT #3 (`docs/pipeline-fidelity-gate.md` SS3 item 3, +`docs/reports/2026-07-22-knowledge-inventory.md`'s MISSING item 3): `_eligible_cards_queryset` now +also excludes any card already carrying a `VoteSource.DEDUCTION` printing vote - the pilot never +re-voted a card `deductive_backfill.py`'s own `DEDUCTIVE_BACKFILL_ANONYMOUS_ID="deductive-backfill-v1"` +pass had already voted for (28,112 live production votes, `run_id=None`, per the gate page's SS6). +Filtered by `source=VoteSource.DEDUCTION` rather than the literal `anonymous_id` value - +deliberately generalized (this PR's own task brief) so ANY prior deduction-class vote (a future +second deduction engine, not just this one literal identity) is excluded the same way, without a +hard import-time dependency on `deductive_backfill.py`'s own constant (matching this module's own +`JOIN_KEY_CONFIDENCE_BOTH` comment's "avoid a hard cross-module dependency over one constant" +precedent) - `deductive_backfill.py`'s own module docstring confirms it is the sole producer of +`VoteSource.DEDUCTION` votes today, so this changes nothing operationally now and only +future-proofs. Applied inside the ONE shared `_eligible_cards_queryset` helper both the join-key +and fallback calculators call - the operational guard that makes a repeated multi-pass Stage D +fire idempotent against the backfill population, independent of which calculator runs. Explicitly +NOT bundling constants #1 (`RESOLUTION_FLOOR_DPI`) or #2 (`EXCLUDED_RESOLVED_TAGS`) here - their +forward-impact sizing is still open per the gate page, and the owner did not include them in this +PRE-FIRE PREP. + +This PR is CODE ONLY: it does not run the full-catalog Stage D fire, the targeted re-extraction of +issue #340's 373-card cohort, or any other prod extraction/write - both remain separate, +owner-gated prod steps. """ import logging @@ -209,9 +301,12 @@ from django.db.models import Q, QuerySet from cardpicker.local_fallback import ( + FALLBACK_CONFIDENCE_MULTI_EVIDENCE, + FALLBACK_CONFIDENCE_SINGLE_EVIDENCE, SYMBOL_DISTANCE_THRESHOLD, SYMBOL_MARGIN, classify_frame_style, + filter_by_border_color, frame_style_is_consistent, match_artist, render_set_symbol, @@ -626,18 +721,31 @@ class JoinKeyCalculatorResult: audit: list[dict[str, object]] = field(default_factory=list) -def _eligible_cards_queryset(anonymous_id: str) -> "QuerySet[Card]": +def _eligible_cards_queryset( + anonymous_id: str, rescannable_skip_reasons: frozenset[str] = JOIN_KEY_RESCANNABLE_SKIP_REASONS +) -> "QuerySet[Card]": """ Mirrors `local_identify_printing_tags._eligible_base_queryset`'s shape (unresolved, no confirmed indexing match, card_type=CARD only, no existing vote from this calculator's own anonymous_id, no non-rescannable scan-log row for it) - a fresh, independent eligibility query rather than a call into that function directly, since this calculator's resume/skip population (cards with a CURRENT `ImageEvidence` row) is a genuinely different concept from - the live pilot's own per-run candidate selection, not a variant of it. + the live pilot's own per-run candidate selection, not a variant of it. `rescannable_skip_reasons` + defaults to `JOIN_KEY_RESCANNABLE_SKIP_REASONS` (this function's original, only caller for a + long time); `run_fallback_calculator` passes its own `FALLBACK_RESCANNABLE_SKIP_REASONS` + instead, since the two calculators' own skip vocabularies mean different things by the same + "transient, re-selectable" concept. + + CONSTANT #3 (module docstring's PIECE 2, `docs/pipeline-fidelity-gate.md` SS3 item 3): + also excludes any card already carrying a `VoteSource.DEDUCTION` printing vote - see the + module docstring's PIECE 2 section for the full reasoning (why `source=DEDUCTION` rather than + the literal `deductive_backfill.DEDUCTIVE_BACKFILL_ANONYMOUS_ID`). Shared by BOTH calculators + that call this helper (join-key and fallback) - the guard that makes a repeated multi-pass + Stage D fire idempotent against the backfill population, independent of which calculator runs. """ non_rescannable_scanned_card_ids = ( CardScanLog.objects.filter(anonymous_id=anonymous_id) - .exclude(skip_reason__in=JOIN_KEY_RESCANNABLE_SKIP_REASONS) + .exclude(skip_reason__in=rescannable_skip_reasons) .values_list("card_id", flat=True) ) return ( @@ -647,6 +755,7 @@ def _eligible_cards_queryset(anonymous_id: str) -> "QuerySet[Card]": card_type=CardTypes.CARD, ) .exclude(printing_tags__anonymous_id=anonymous_id) + .exclude(printing_tags__source=VoteSource.DEDUCTION) .exclude(pk__in=non_rescannable_scanned_card_ids) .distinct() .select_related("source") @@ -749,6 +858,305 @@ def run_join_key_calculator( return result +# --------------------------------------------------------------------------------------------- +# PIECE 1: the fallback channel calculator (module docstring) - Stage D's own port of +# local_fallback.py's pilot "Pass 2" evidence-combination model. Own anonymous_id, same rationale +# as JOIN_KEY_ANONYMOUS_ID's own comment - a distinct, independently purgeable/re-runnable +# population, kept separate from the pilot's own "local-fallback-v1" identity (that identity +# belongs to the live legacy engine, which continues to run against a fresh per-invocation fetch; +# this calculator consumes stored ImageEvidence instead, a genuinely different population). +# --------------------------------------------------------------------------------------------- + +STAGE_D_FALLBACK_ANONYMOUS_ID = "stage-d-fallback-v1" + +# This calculator's own skip vocabulary. Deliberately NOT "no-evidence" for the +# no-sub-check-produced-a-reading case (unlike local_fallback.FallbackOutcome's own literal +# "no-evidence" naming for the identical concept) - "no-evidence" is already Stage D's own +# established name (see JOIN_KEY_RESCANNABLE_SKIP_REASONS above) for a DIFFERENT concept ("this +# card's ImageEvidence row itself doesn't exist yet") - reusing it here for a different meaning, +# even scoped to a different anonymous_id, would be a needless collision in a reader's head for +# no benefit. "eliminated"/"ambiguous" ARE kept verbatim from the pilot's own vocabulary - those +# two carry the same meaning here as there, no rename needed. +FALLBACK_NO_EVIDENCE_SKIP_REASON = "no-evidence" # this calculator's own ImageEvidence-row-missing case, same meaning as JOIN_KEY's own identical string, different anonymous_id scope +FALLBACK_NO_SUB_CHECK_EVIDENCE_SKIP_REASON = "no-sub-check-evidence" # local_fallback.FallbackOutcome's own "no-evidence" concept, renamed to avoid colliding with the line above +FALLBACK_RESCANNABLE_SKIP_REASONS = frozenset({FALLBACK_NO_EVIDENCE_SKIP_REASON}) + + +@dataclass(frozen=True) +class FallbackVerdict: + """ + Pure result of one card's fallback-calculator run (module docstring's PIECE 1) - no DB write + has happened yet, mirrors `JoinKeyVerdict`'s own compute/persist split. Exactly one of two + shapes: a positive match (`printing_pk` set, `is_no_match` always False - this calculator, like + the pilot's own fallback pass, never casts a genuine no-match vote, only a match or an + abstention) or a named skip (`skip_reason` set, `printing_pk` is None). + """ + + card_id: int + printing_pk: Optional[int] = None + confidence: Optional[float] = None + detail: str = "" + skip_reason: str = "" + evidence_types_used: tuple[str, ...] = () + + +def _filter_by_symbol_phash(symbol_phash: Optional[int], candidates: list[CandidatePrinting]) -> Optional[set[int]]: + """ + The fallback calculator's own symbol sub-check (module docstring's PIECE 1) - the SAME + pure-Hamming-distance-arithmetic reimplementation `_symbol_phash_tiebreak` above already + established (`render_set_symbol`, PROTECTED CORE, called not modified; `SYMBOL_DISTANCE_THRESHOLD`/ + `SYMBOL_MARGIN`, PROTECTED CORE constants, reused verbatim), duplicated rather than shared with + `_symbol_phash_tiebreak` (this module's own "duplicate the arithmetic, reimplement nothing + decision-shaped" convention - see `JOIN_KEY_CONFIDENCE_BOTH`'s own comment for the same + reasoning applied to a constant rather than a function) because the two return different + shapes: `_symbol_phash_tiebreak` returns one winning `CandidatePrinting` for the join-key + calculator's own tie-break call site (only ever called against an already-narrowed "ambiguous" + subset), while this returns the FULL SET of surviving candidate pks across the WHOLE candidate + list passed in - mirroring `local_fallback.find_symbol_matches`'s own return convention exactly, + which is what this calculator's border/artist/symbol INTERSECTION model needs to compose with + `filter_by_border_color`/`match_artist`'s own `Optional[set[int]]` shape. + + Returns `None` (no reading - filters nothing) if `symbol_phash` is `None`, no candidate's + expansion glyph could be rendered, the best distance exceeds `SYMBOL_DISTANCE_THRESHOLD`, or a + runner-up sits within `SYMBOL_MARGIN` of the best (an unresolved tie) - the same four + abstention cases `find_symbol_matches`'s own docstring documents for its live-image-scan + version. + """ + if symbol_phash is None: + return None + + distances: list[tuple[str, int]] = [] + seen_expansions: set[str] = set() + for candidate in candidates: + if candidate.expansion_code in seen_expansions: + continue + seen_expansions.add(candidate.expansion_code) + reference = render_set_symbol(candidate.expansion_code) + if reference is None: + continue + reference_hash_int = twos_complement(str(imagehash.phash(reference)), _SYMBOL_HASH_BITS) + distances.append((candidate.expansion_code, _hamming_distance(symbol_phash, reference_hash_int))) + + if not distances: + return None + + distances.sort(key=lambda pair: pair[1]) + best_expansion, best_distance = distances[0] + if best_distance > SYMBOL_DISTANCE_THRESHOLD: + return None + if len(distances) > 1 and (distances[1][1] - best_distance) <= SYMBOL_MARGIN: + return None + + return {c.pk for c in candidates if c.expansion_code == best_expansion} + + +def calculate_fallback_verdict( + card_id: int, evidence: ImageEvidence, candidates: list[CandidatePrinting] +) -> FallbackVerdict: + """ + The fallback channel calculator (module docstring's PIECE 1) - Stage D's own port of + `local_fallback.run_fallback_for_card`'s evidence-combination model, operating ENTIRELY off + already-persisted `ImageEvidence` fields (never a live image/re-OCR - this file never + re-fetches). Each sub-check is either a DIRECT CALL into `local_fallback`'s own pure decision + function (`filter_by_border_color`, `match_artist` - both PROTECTED CORE, neither touches a raw + image, both accept already-extracted strings - called, never reimplemented, the same + "import helpers, call don't reimplement" pattern `_apply_agreement_checks` above already + established) or, for the symbol sub-check, `_filter_by_symbol_phash`'s own reimplemented + arithmetic (see that function's own docstring). + + Conservative by design, reproducing `local_fallback.py`'s own documented rule exactly (that + module's own docstring: "A vote is written only when the intersection across every sub-check + that DID produce a reading narrows to EXACTLY ONE candidate") - never loosened, never extended + with the join-key calculator's own agreement/corroboration layer (frame/copyright-year/ + truncated-image checks do not exist in `local_fallback.py` and are deliberately NOT added + here - this is a faithful port of that module's own decision model, not an augmented one). + + Pure function, no DB write (aside from the one read-only `CanonicalCard` query below, mirroring + `_apply_agreement_checks`'s own single-query pattern) - callers persist via + `CardPrintingTag`/`CardScanLog` exactly like the join-key calculator's own + `run_join_key_calculator`. Same name-scoping caller contract as `calculate_join_key_verdict`: + `candidates` MUST already be narrowed to this card's own name (`_resolve_candidates_for_card`). + """ + candidate_pks = {c.pk for c in candidates} + canonicals = { + c.pk: c + for c in CanonicalCard.objects.select_related("artist", "printing_metadata").filter(pk__in=candidate_pks) + } + artist_by_pk = {pk: c.artist.name for pk, c in canonicals.items()} + border_color_by_pk = { + pk: c.printing_metadata.border_color + for pk, c in canonicals.items() + if getattr(c, "printing_metadata", None) is not None and c.printing_metadata.border_color + } + + border_filtered = filter_by_border_color(evidence.layout_class or None, candidates, border_color_by_pk) + artist_filtered = ( + match_artist(evidence.artist_ocr_name, candidates, artist_by_pk) if evidence.artist_ocr_name else None + ) + symbol_filtered = _filter_by_symbol_phash(evidence.symbol_phash, candidates) + + survivors = set(candidate_pks) + evidence_types_used: list[str] = [] + for name, filtered in (("border", border_filtered), ("artist", artist_filtered), ("symbol", symbol_filtered)): + if filtered is not None: + survivors &= filtered + evidence_types_used.append(name) + + if not evidence_types_used: + return FallbackVerdict(card_id=card_id, skip_reason=FALLBACK_NO_SUB_CHECK_EVIDENCE_SKIP_REASON) + if len(survivors) == 0: + return FallbackVerdict( + card_id=card_id, skip_reason="eliminated", evidence_types_used=tuple(evidence_types_used) + ) + if len(survivors) > 1: + return FallbackVerdict(card_id=card_id, skip_reason="ambiguous", evidence_types_used=tuple(evidence_types_used)) + + confidence = ( + FALLBACK_CONFIDENCE_MULTI_EVIDENCE if len(evidence_types_used) > 1 else FALLBACK_CONFIDENCE_SINGLE_EVIDENCE + ) + return FallbackVerdict( + card_id=card_id, + printing_pk=next(iter(survivors)), + confidence=confidence, + evidence_types_used=tuple(evidence_types_used), + ) + + +@dataclass +class FallbackCalculatorResult: + dry_run: bool = False + run_id: str = "" + cards_considered: int = 0 + votes_would_cast: int = 0 + votes_written: int = 0 + skip_counts: dict[str, int] = field(default_factory=dict) + # capped audit sample, mirroring JoinKeyCalculatorResult.audit's own convention. + audit: list[dict[str, object]] = field(default_factory=list) + + +def _fallback_eligible_cards_queryset() -> "QuerySet[Card]": + """ + Cards the join-key calculator already concluded have no confident hit - the SAME population + `_slow_path_eligible_cards_queryset` below selects from (a real `is_no_match` vote, or a + non-rescannable skip in `JOIN_KEY_NO_HIT_SKIP_REASONS`) - that this calculator's own + `STAGE_D_FALLBACK_ANONYMOUS_ID` hasn't already processed (scanned OR voted), via the shared + `_eligible_cards_queryset` helper (which also carries PIECE 2's own deduction-vote exclusion, + applied identically to both calculators). + """ + join_key_no_match_card_ids = CardPrintingTag.objects.filter( + anonymous_id=JOIN_KEY_ANONYMOUS_ID, is_no_match=True + ).values_list("card_id", flat=True) + join_key_no_hit_scanned_card_ids = CardScanLog.objects.filter( + anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason__in=JOIN_KEY_NO_HIT_SKIP_REASONS + ).values_list("card_id", flat=True) + return _eligible_cards_queryset( + STAGE_D_FALLBACK_ANONYMOUS_ID, rescannable_skip_reasons=FALLBACK_RESCANNABLE_SKIP_REASONS + ).filter(Q(pk__in=join_key_no_match_card_ids) | Q(pk__in=join_key_no_hit_scanned_card_ids)) + + +def run_fallback_calculator( + run_id: Optional[str] = None, + dry_run: bool = True, + chunk_size: int = 500, + audit_sample_size: int = 20, + default_cards_path: Optional[Path] = None, +) -> FallbackCalculatorResult: + """ + Batch runner for PIECE 1 (module docstring) - mirrors `run_join_key_calculator`'s own shape + (dry-run default, CardScanLog/CardPrintingTag batching, `resolve_and_persist_printing` called + per touched card, `PilotRunLedger`/gate-check wiring living in the management command exactly + like the join-key calculator's own). Only ever considers cards the join-key calculator ALREADY + concluded have no confident hit (`_fallback_eligible_cards_queryset`) - this calculator is + Stage D's own "pass 2", the same relationship `local_fallback.py`'s own module docstring + documents between the pilot's pass 1 (OCR/phash) and pass 2 (fallback). `default_cards_path` is + threaded through to `_resolve_candidates_for_card` exactly as `run_join_key_calculator`'s own + parameter is. + """ + run_id = run_id or generate_run_id() + index = CandidateNameIndex() + result = FallbackCalculatorResult(dry_run=dry_run, run_id=run_id) + + votes_batch: list[CardPrintingTag] = [] + scan_log_batch: list[CardScanLog] = [] + touched_card_ids: list[int] = [] + + for card in _fallback_eligible_cards_queryset().iterator(chunk_size=chunk_size): + if card.content_phash is None: + continue # no stable hash yet to key a CURRENT ImageEvidence lookup against + + evidence = ( + ImageEvidence.objects.filter(card_id=card.pk, content_hash=card.content_phash) + .filter(extractor_versions__has_key="collector_line_ocr") + .order_by("-updated_at") + .first() + ) + if evidence is None: + result.skip_counts[FALLBACK_NO_EVIDENCE_SKIP_REASON] = ( + result.skip_counts.get(FALLBACK_NO_EVIDENCE_SKIP_REASON, 0) + 1 + ) + if not dry_run: + scan_log_batch.append( + CardScanLog( + card_id=card.pk, + anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID, + run_id=run_id, + skip_reason=FALLBACK_NO_EVIDENCE_SKIP_REASON, + ) + ) + continue + + result.cards_considered += 1 + candidates = _resolve_candidates_for_card(card.name, index, default_cards_path=default_cards_path) + verdict = calculate_fallback_verdict(card.pk, evidence, candidates) + + if verdict.skip_reason: + result.skip_counts[verdict.skip_reason] = result.skip_counts.get(verdict.skip_reason, 0) + 1 + if not dry_run: + scan_log_batch.append( + CardScanLog( + card_id=card.pk, + anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID, + run_id=run_id, + skip_reason=verdict.skip_reason, + ) + ) + continue + + result.votes_would_cast += 1 + if len(result.audit) < audit_sample_size: + result.audit.append( + { + "card_id": card.pk, + "detail": verdict.detail, + "evidence_types_used": list(verdict.evidence_types_used), + } + ) + + if not dry_run: + votes_batch.append( + CardPrintingTag( + card_id=card.pk, + printing_id=verdict.printing_pk, + is_no_match=False, + anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID, + source=VoteSource.OCR, + confidence=verdict.confidence, + run_id=run_id, + ) + ) + touched_card_ids.append(card.pk) + + if not dry_run: + CardPrintingTag.objects.bulk_create(votes_batch) + CardScanLog.objects.bulk_create(scan_log_batch) + for touched_card in Card.objects.filter(pk__in=touched_card_ids): + resolve_and_persist_printing(touched_card) + + result.votes_written = len(votes_batch) + + return result + + # Slow-path routing (module docstring's "two further cheap additions", item 1; owner decision, # public issue #220): own anonymous_id, same rationale as JOIN_KEY_ANONYMOUS_ID's own comment - a # distinct, independently purgeable/re-runnable population from every other engine's. This @@ -870,6 +1278,16 @@ def _slow_path_eligible_cards_queryset() -> "QuerySet[Card]": A card the join-key calculator hasn't looked at yet at all (no vote, no scan-log row) is simply not yet in scope - this calculator only ever consumes the join-key calculator's own output, it never runs independently of it. + + ALSO excludes any card the fallback calculator (`STAGE_D_FALLBACK_ANONYMOUS_ID`, module + docstring's PIECE 1) already successfully voted on - a real printing match, not merely a scan + it abstained on. This is the wiring that makes PIECE 1 actually take effect: the management + command runs join-key -> fallback -> slow-path in that order, and without this exclusion + slow-path would route a card to human review that the fallback calculator resolves moments + earlier in the SAME invocation. A card the fallback calculator merely SCANNED but abstained on + (`no-evidence-types-used`/`eliminated`/`ambiguous`) is deliberately NOT excluded here - it + still has no confident automated hit from either calculator and belongs in the review queue + exactly as before this PR. """ join_key_no_match_card_ids = CardPrintingTag.objects.filter( anonymous_id=JOIN_KEY_ANONYMOUS_ID, is_no_match=True @@ -880,6 +1298,9 @@ def _slow_path_eligible_cards_queryset() -> "QuerySet[Card]": already_routed_card_ids = CardScanLog.objects.filter(anonymous_id=SLOW_PATH_ANONYMOUS_ID).values_list( "card_id", flat=True ) + fallback_voted_card_ids = CardPrintingTag.objects.filter( + anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID, is_no_match=False + ).values_list("card_id", flat=True) return ( Card.objects.filter( printing_tag_status=PrintingTagStatus.UNRESOLVED, @@ -888,6 +1309,7 @@ def _slow_path_eligible_cards_queryset() -> "QuerySet[Card]": ) .filter(Q(pk__in=join_key_no_match_card_ids) | Q(pk__in=join_key_no_hit_scanned_card_ids)) .exclude(pk__in=already_routed_card_ids) + .exclude(pk__in=fallback_voted_card_ids) .distinct() .select_related("source") ) @@ -975,6 +1397,14 @@ def run_slow_path_calculator( "JOIN_KEY_RESCANNABLE_SKIP_REASONS", "JOIN_KEY_NO_HIT_SKIP_REASONS", "COPYRIGHT_YEAR_MISMATCH_THRESHOLD_YEARS", + "STAGE_D_FALLBACK_ANONYMOUS_ID", + "FALLBACK_NO_EVIDENCE_SKIP_REASON", + "FALLBACK_NO_SUB_CHECK_EVIDENCE_SKIP_REASON", + "FALLBACK_RESCANNABLE_SKIP_REASONS", + "FallbackVerdict", + "calculate_fallback_verdict", + "FallbackCalculatorResult", + "run_fallback_calculator", "SLOW_PATH_ANONYMOUS_ID", "SLOW_PATH_TO_REVIEW_REASON", "SLOW_PATH_RAW_SIGNAL_FIELDS", diff --git a/MPCAutofill/cardpicker/management/commands/local_calculate_verdicts.py b/MPCAutofill/cardpicker/management/commands/local_calculate_verdicts.py index 5f60ff96c..efe15461b 100644 --- a/MPCAutofill/cardpicker/management/commands/local_calculate_verdicts.py +++ b/MPCAutofill/cardpicker/management/commands/local_calculate_verdicts.py @@ -5,6 +5,8 @@ from cardpicker.local_calculate_verdicts import ( JOIN_KEY_ANONYMOUS_ID, + STAGE_D_FALLBACK_ANONYMOUS_ID, + run_fallback_calculator, run_join_key_calculator, run_slow_path_calculator, ) @@ -20,13 +22,16 @@ class Command(BaseCommand): help = ( "Stage D (docs/features/catalog-completion-plan.md, public issue #152): the join-key " "calculator - the fast-path deduction step over Stage C's ImageEvidence rows (collector-" - "line OCR + set-symbol phash tie-break, plus a copyright-year era cross-check) - plus the " - "slow-path routing calculator (owner decision, issue #220) that sends every card the " - "join-key calculator couldn't confidently resolve to the human review queue, carrying its " - "raw extracted signals. Casts CardPrintingTag votes via the existing, unmodified " - "vote-consensus machinery; never resolves a card by itself - the slow-path half casts no " - "votes at all. Defaults to dry-run and requires an explicit --write to actually write, " - "matching local_residual_classify's own convention." + "line OCR + set-symbol phash tie-break, plus a copyright-year era cross-check) - then the " + "fallback channel calculator (Stage D's own port of local_fallback.py's pilot 'Pass 2' " + "border/artist/symbol evidence-combination model, run only over cards the join-key " + "calculator found no confident hit for) - then the slow-path routing calculator (owner " + "decision, issue #220) that sends every card NEITHER of the two calculators above could " + "confidently resolve to the human review queue, carrying its raw extracted signals. Casts " + "CardPrintingTag votes via the existing, unmodified vote-consensus machinery; never " + "resolves a card by itself - the slow-path half casts no votes at all. Defaults to dry-run " + "and requires an explicit --write to actually write, matching local_residual_classify's " + "own convention." ) def add_arguments(self, parser: Any) -> None: @@ -98,11 +103,52 @@ def handle(self, *args: Any, **kwargs: Any) -> None: ) print(f"Gate check passed: 0/{len(touched_card_ids)} touched cards resolved machine-only.") - # Slow-path routing (owner decision, issue #220): runs AFTER the join-key pass above - # in the SAME invocation/run_id - it only ever consumes that pass's own no-hit output - # (see run_slow_path_calculator's own docstring), so sequencing here matters even - # though both ship in this one command/PR. Casts no CardPrintingTag at all (it has no - # printing to vote for), so there is no analogous gate check to run for it. + # Fallback channel calculator (PIECE 1 of this PR's pre-fire prep bundle): runs AFTER + # the join-key pass above in the SAME invocation/run_id - it only ever consumes cards + # the join-key calculator found no confident hit for (see + # _fallback_eligible_cards_queryset's own docstring), so sequencing here matters. + # Ordered BEFORE slow-path routing below deliberately: a card this calculator resolves + # must not also get routed to human review in the same invocation (see + # _slow_path_eligible_cards_queryset's own new exclusion for the wiring this depends on). + fallback_result = run_fallback_calculator(run_id=run_id, dry_run=dry_run, chunk_size=kwargs["chunk_size"]) + votes_written += fallback_result.votes_written + would_cast += fallback_result.votes_would_cast + print( + f"[fallback] considered={fallback_result.cards_considered} " + f"votes={'written=' + str(fallback_result.votes_written) if not dry_run else 'would_cast=' + str(fallback_result.votes_would_cast)} " + f"skip_counts={dict(fallback_result.skip_counts)}" + ) + for entry in fallback_result.audit[:10]: + print(f" sample: {entry}") + + if not dry_run: + # same rationale as the join-key gate check above - re-derived from this run's own + # freshly-written votes (scoped by run_id + anonymous_id) rather than the capped + # audit sample. + fallback_touched_card_ids = list( + CardPrintingTag.objects.filter( + run_id=run_id, anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID + ).values_list("card_id", flat=True) + ) + fallback_violations = verify_zero_resolutions(fallback_touched_card_ids) + if fallback_violations: + raise CommandError( + f"GATE VIOLATION: {len(fallback_violations)} card(s) resolved to a printing " + f"from this single-anonymous_id machine pass alone, which should be " + f"structurally impossible per resolve_weighted_consensus's own human-" + f"backed gate - STOP and investigate. Affected card pks: " + f"{fallback_violations[:50]}" + (" (truncated)" if len(fallback_violations) > 50 else "") + ) + print( + f"Gate check passed: 0/{len(fallback_touched_card_ids)} fallback-touched cards " + "resolved machine-only." + ) + + # Slow-path routing (owner decision, issue #220): runs AFTER both calculators above in + # the SAME invocation/run_id - it only ever consumes their own no-hit output (see + # run_slow_path_calculator's own docstring), so sequencing here matters even though all + # three ship in this one command. Casts no CardPrintingTag at all (it has no printing + # to vote for), so there is no analogous gate check to run for it. slow_path_result = run_slow_path_calculator(run_id=run_id, dry_run=dry_run, chunk_size=kwargs["chunk_size"]) print( f"[slow-path] considered={slow_path_result.cards_considered} " diff --git a/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py b/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py index bea8c7390..1e5c73dbb 100644 --- a/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py +++ b/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py @@ -26,6 +26,8 @@ from cardpicker.local_calculate_verdicts import ( COPYRIGHT_YEAR_MISMATCH_THRESHOLD_YEARS, + FALLBACK_NO_EVIDENCE_SKIP_REASON, + FALLBACK_NO_SUB_CHECK_EVIDENCE_SKIP_REASON, JOIN_KEY_ANONYMOUS_ID, JOIN_KEY_CONFIDENCE_ARTIST_DISAGREEMENT, JOIN_KEY_CONFIDENCE_BOTH, @@ -34,14 +36,22 @@ JOIN_KEY_NO_MATCH_CONFIDENCE, SLOW_PATH_ANONYMOUS_ID, SLOW_PATH_TO_REVIEW_REASON, + STAGE_D_FALLBACK_ANONYMOUS_ID, + _filter_by_symbol_phash, _resolve_candidates_for_card, _symbol_phash_tiebreak, + calculate_fallback_verdict, calculate_join_key_verdict, calculate_slow_path_verdict, + run_fallback_calculator, run_join_key_calculator, run_slow_path_calculator, ) -from cardpicker.local_fallback import render_set_symbol +from cardpicker.local_fallback import ( + FALLBACK_CONFIDENCE_MULTI_EVIDENCE, + FALLBACK_CONFIDENCE_SINGLE_EVIDENCE, + render_set_symbol, +) from cardpicker.local_identify_printing_tags import ( CandidateNameIndex, CandidatePrinting, @@ -1057,3 +1067,401 @@ def test_stale_evidence_since_the_join_key_pass_is_not_routed(self, db): assert result.cards_considered == 0 assert CardScanLog.objects.filter(anonymous_id=SLOW_PATH_ANONYMOUS_ID).count() == 0 + + +class TestFilterBySymbolPhash: + """Mirrors TestSymbolPhashTiebreak's own cases - same underlying arithmetic, different return + shape (a full set of surviving pks vs. one winning CandidatePrinting), see + _filter_by_symbol_phash's own docstring for why the two are duplicated rather than shared.""" + + def test_returns_none_without_a_symbol_hash(self): + candidates = [CandidatePrinting(pk=1, expansion_code="mom", collector_number="158")] + assert _filter_by_symbol_phash(None, candidates) is None + + def test_returns_none_with_no_candidates(self): + assert _filter_by_symbol_phash(_hash_of("mom"), []) is None + + def test_returns_none_for_an_unrenderable_expansion_code(self): + candidates = [CandidatePrinting(pk=1, expansion_code="zzznotarealcode", collector_number="1")] + assert _filter_by_symbol_phash(_hash_of("mom"), candidates) is None + + def test_returns_every_pk_sharing_the_winning_expansion(self): + candidates = [ + CandidatePrinting(pk=1, expansion_code="mir", collector_number="1"), + CandidatePrinting(pk=2, expansion_code="mir", collector_number="2"), + CandidatePrinting(pk=3, expansion_code="som", collector_number="1"), + ] + assert _filter_by_symbol_phash(_hash_of("mir"), candidates) == {1, 2} + + +class TestCalculateFallbackVerdict: + """PIECE 1 (module docstring) - the border/artist/symbol intersection model, ported off + already-persisted ImageEvidence fields rather than a live image. See local_fallback.py's own + module docstring for the evidence-combination model this reproduces exactly.""" + + def test_border_alone_narrows_to_one_and_casts_a_vote(self, db): + printing_black = CanonicalCardFactory(name="Test Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing_black, border_color="black") + printing_white = CanonicalCardFactory(name="Test Card", expansion__code="vow", collector_number="200") + CanonicalPrintingMetadataFactory(canonical_card=printing_white, border_color="white") + card = CardFactory(name="Test Card") + candidates = [ + CandidatePrinting(pk=printing_black.pk, expansion_code="mom", collector_number="158"), + CandidatePrinting(pk=printing_white.pk, expansion_code="vow", collector_number="200"), + ] + evidence = _evidence(card, layout_class="black") + + verdict = calculate_fallback_verdict(card.pk, evidence, candidates) + + assert verdict.printing_pk == printing_black.pk + assert verdict.evidence_types_used == ("border",) + assert verdict.confidence == FALLBACK_CONFIDENCE_SINGLE_EVIDENCE + assert verdict.skip_reason == "" + + def test_symbol_alone_narrows_to_one_and_casts_a_vote(self, db): + printing_a = CanonicalCardFactory(name="Test Card", expansion__code="mir", collector_number="1") + printing_b = CanonicalCardFactory(name="Test Card", expansion__code="som", collector_number="1") + card = CardFactory(name="Test Card") + candidates = [ + CandidatePrinting(pk=printing_a.pk, expansion_code="mir", collector_number="1"), + CandidatePrinting(pk=printing_b.pk, expansion_code="som", collector_number="1"), + ] + evidence = _evidence(card, symbol_phash=_hash_of("mir")) + + verdict = calculate_fallback_verdict(card.pk, evidence, candidates) + + assert verdict.printing_pk == printing_a.pk + assert verdict.evidence_types_used == ("symbol",) + assert verdict.confidence == FALLBACK_CONFIDENCE_SINGLE_EVIDENCE + + def test_border_and_artist_agreement_gives_multi_evidence_confidence(self, db): + printing_a = CanonicalCardFactory( + name="Test Card", expansion__code="mom", collector_number="158", artist__name="Rebecca Guay" + ) + CanonicalPrintingMetadataFactory(canonical_card=printing_a, border_color="black") + printing_b = CanonicalCardFactory( + name="Test Card", expansion__code="vow", collector_number="200", artist__name="Someone Else" + ) + CanonicalPrintingMetadataFactory(canonical_card=printing_b, border_color="white") + card = CardFactory(name="Test Card") + candidates = [ + CandidatePrinting(pk=printing_a.pk, expansion_code="mom", collector_number="158"), + CandidatePrinting(pk=printing_b.pk, expansion_code="vow", collector_number="200"), + ] + evidence = _evidence(card, layout_class="black", artist_ocr_name="Rebecca Guay") + + verdict = calculate_fallback_verdict(card.pk, evidence, candidates) + + assert verdict.printing_pk == printing_a.pk + assert set(verdict.evidence_types_used) == {"border", "artist"} + assert verdict.confidence == FALLBACK_CONFIDENCE_MULTI_EVIDENCE + + def test_border_and_artist_disagreement_abstains_never_a_false_accept(self, db): + """The no-false-accept property (module docstring): border evidence alone points at + printing_a, artist evidence alone points at printing_b - their intersection is empty, so + this MUST abstain ('eliminated'), never pick either candidate.""" + printing_a = CanonicalCardFactory( + name="Test Card", expansion__code="mom", collector_number="158", artist__name="Rebecca Guay" + ) + CanonicalPrintingMetadataFactory(canonical_card=printing_a, border_color="black") + printing_b = CanonicalCardFactory( + name="Test Card", expansion__code="vow", collector_number="200", artist__name="Someone Else" + ) + CanonicalPrintingMetadataFactory(canonical_card=printing_b, border_color="white") + card = CardFactory(name="Test Card") + candidates = [ + CandidatePrinting(pk=printing_a.pk, expansion_code="mom", collector_number="158"), + CandidatePrinting(pk=printing_b.pk, expansion_code="vow", collector_number="200"), + ] + # border evidence -> printing_a ("black"); artist evidence -> printing_b ("Someone Else") + evidence = _evidence(card, layout_class="black", artist_ocr_name="Someone Else") + + verdict = calculate_fallback_verdict(card.pk, evidence, candidates) + + assert verdict.printing_pk is None + assert verdict.skip_reason == "eliminated" + + def test_ambiguous_when_the_only_reading_matches_more_than_one_candidate(self, db): + printing_a = CanonicalCardFactory(name="Test Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing_a, border_color="black") + printing_b = CanonicalCardFactory(name="Test Card", expansion__code="vow", collector_number="200") + CanonicalPrintingMetadataFactory(canonical_card=printing_b, border_color="black") + card = CardFactory(name="Test Card") + candidates = [ + CandidatePrinting(pk=printing_a.pk, expansion_code="mom", collector_number="158"), + CandidatePrinting(pk=printing_b.pk, expansion_code="vow", collector_number="200"), + ] + evidence = _evidence(card, layout_class="black") + + verdict = calculate_fallback_verdict(card.pk, evidence, candidates) + + assert verdict.printing_pk is None + assert verdict.skip_reason == "ambiguous" + + def test_no_sub_check_produced_a_reading_abstains_even_with_a_single_candidate(self, db): + """A single remaining candidate is NOT itself evidence - local_fallback.py's own rule + (module docstring) checks "did any sub-check produce a reading at all" BEFORE ever looking + at how many candidates survive, so a lone candidate with zero corroborating evidence must + still abstain, not be nodded through by default.""" + printing = CanonicalCardFactory(name="Test Card", expansion__code="mom", collector_number="158") + card = CardFactory(name="Test Card") + candidates = [CandidatePrinting(pk=printing.pk, expansion_code="mom", collector_number="158")] + evidence = _evidence(card) # no layout_class, no artist_ocr_name, no symbol_phash + + verdict = calculate_fallback_verdict(card.pk, evidence, candidates) + + assert verdict.printing_pk is None + assert verdict.skip_reason == FALLBACK_NO_SUB_CHECK_EVIDENCE_SKIP_REASON + + +class TestRunFallbackCalculator: + def _no_hit_card(self, *, skip_reason="no-text", is_no_match=False, **evidence_overrides): + card = CardFactory(name="Some Card", content_phash=42) + evidence = _evidence(card, **evidence_overrides) + if is_no_match: + CardPrintingTag.objects.create( + card=card, printing=None, is_no_match=True, anonymous_id=JOIN_KEY_ANONYMOUS_ID, source=VoteSource.OCR + ) + else: + CardScanLog.objects.create(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason=skip_reason) + return card, evidence + + def test_dry_run_counts_without_writing(self, db): + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") + self._no_hit_card(layout_class="black") + + result = run_fallback_calculator(dry_run=True) + + assert result.cards_considered == 1 + assert result.votes_would_cast == 1 + assert CardPrintingTag.objects.filter(anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID).count() == 0 + assert CardScanLog.objects.filter(anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID).count() == 0 + + def test_write_casts_a_vote_and_never_resolves_alone(self, db): + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") + card, _ = self._no_hit_card(layout_class="black") + + result = run_fallback_calculator(dry_run=False) + + assert result.votes_written == 1 + vote = CardPrintingTag.objects.get(card=card, anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID) + assert vote.printing_id == printing.pk + assert vote.source == VoteSource.OCR + assert vote.run_id == result.run_id + + card.refresh_from_db() + # a single VoteSource.OCR vote (weight 0.5) can never clear the human-backed gate alone. + assert card.printing_tag_status == PrintingTagStatus.UNRESOLVED + + def test_a_card_the_join_key_calculator_already_resolved_is_not_eligible(self, db): + card = CardFactory(name="Some Card", content_phash=42) + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") + _evidence(card, layout_class="black") + CardPrintingTag.objects.create( + card=card, printing=printing, is_no_match=False, anonymous_id=JOIN_KEY_ANONYMOUS_ID, source=VoteSource.OCR + ) + + result = run_fallback_calculator(dry_run=False) + + assert result.cards_considered == 0 + assert CardPrintingTag.objects.filter(anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID).count() == 0 + + def test_skip_writes_a_scan_log_row(self, db): + card, _ = self._no_hit_card() # no layout_class/artist_ocr_name/symbol_phash at all + + result = run_fallback_calculator(dry_run=False) + + assert result.votes_written == 0 + assert CardPrintingTag.objects.filter(anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID).count() == 0 + log = CardScanLog.objects.get(card=card, anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID) + assert log.skip_reason == FALLBACK_NO_SUB_CHECK_EVIDENCE_SKIP_REASON + + def test_idempotent_against_its_own_anonymous_id(self, db): + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") + card, _ = self._no_hit_card(layout_class="black") + + first = run_fallback_calculator(dry_run=False) + assert first.votes_written == 1 + + second = run_fallback_calculator(dry_run=False) + assert second.cards_considered == 0 + assert CardPrintingTag.objects.filter(card=card, anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID).count() == 1 + + def test_card_without_evidence_is_a_rescannable_no_evidence_skip(self, db): + card = CardFactory(name="Some Card", content_phash=42) + CardScanLog.objects.create(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason="no-text") + + result = run_fallback_calculator(dry_run=False) + + assert result.cards_considered == 0 + assert result.skip_counts.get(FALLBACK_NO_EVIDENCE_SKIP_REASON) == 1 + log = CardScanLog.objects.get(anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID) + assert log.skip_reason == FALLBACK_NO_EVIDENCE_SKIP_REASON + + # rescannable: adding evidence and re-running picks the card back up. + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") + _evidence(card, layout_class="black") + + second = run_fallback_calculator(dry_run=False) + assert second.cards_considered == 1 + assert second.votes_written == 1 + + def test_evidence_from_a_stale_content_hash_is_not_used(self, db): + card = CardFactory(name="Some Card", content_phash=99) + CardScanLog.objects.create(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason="no-text") + _evidence(card, content_hash=42, layout_class="black") # stale - card.content_phash is 99 + + result = run_fallback_calculator(dry_run=False) + + assert result.cards_considered == 0 + assert result.skip_counts.get(FALLBACK_NO_EVIDENCE_SKIP_REASON) == 1 + + +class TestFallbackSlowPathInteraction: + def test_a_card_the_fallback_calculator_resolved_is_not_routed_to_slow_path(self, db): + """Wiring necessity (module docstring's PIECE 1 section): without this exclusion, + slow-path would route a card to human review that the fallback calculator resolves + moments earlier in the SAME invocation - the management command runs join-key -> fallback + -> slow-path in that order.""" + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") + card = CardFactory(name="Some Card", content_phash=42) + _evidence(card, layout_class="black") + CardScanLog.objects.create(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason="no-text") + CardPrintingTag.objects.create( + card=card, + printing=printing, + is_no_match=False, + anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID, + source=VoteSource.OCR, + ) + + result = run_slow_path_calculator(dry_run=False) + + assert result.cards_considered == 0 + assert CardScanLog.objects.filter(anonymous_id=SLOW_PATH_ANONYMOUS_ID).count() == 0 + + def test_a_card_the_fallback_calculator_only_scanned_is_still_routed(self, db): + """The exclusion is scoped to a real fallback VOTE only - a card the fallback calculator + scanned but abstained on (no confident hit from either calculator) still has nothing + automated resolving it, and belongs in the review queue exactly as before this PR.""" + card = CardFactory(name="Some Card", content_phash=42) + _evidence(card) + CardScanLog.objects.create(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason="no-text") + CardScanLog.objects.create( + card=card, + anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID, + skip_reason=FALLBACK_NO_SUB_CHECK_EVIDENCE_SKIP_REASON, + ) + + result = run_slow_path_calculator(dry_run=False) + + assert result.cards_considered == 1 + assert CardScanLog.objects.filter(anonymous_id=SLOW_PATH_ANONYMOUS_ID, card=card).count() == 1 + + +class TestDeductionVoteExclusion: + """PIECE 2 (module docstring) - constant #3, docs/pipeline-fidelity-gate.md SS3 item 3 / + docs/reports/2026-07-22-knowledge-inventory.md's MISSING item 3. Exercised against BOTH + calculators, since `_eligible_cards_queryset` is shared by both.""" + + def test_join_key_calculator_excludes_a_card_with_a_prior_deduction_vote(self, db): + card = CardFactory(name="Some Card", content_phash=42) + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + _evidence(card, collector_line_set_code="mom", collector_line_collector_number="158") + CardPrintingTag.objects.create( + card=card, + printing=printing, + is_no_match=False, + anonymous_id="deductive-backfill-v1", + source=VoteSource.DEDUCTION, + ) + + result = run_join_key_calculator(dry_run=False) + + assert result.cards_considered == 0 + assert CardPrintingTag.objects.filter(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 0 + + def test_join_key_calculator_still_processes_a_card_without_a_deduction_vote(self, db): + card = CardFactory(name="Some Card", content_phash=42) + CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + _evidence(card, collector_line_set_code="mom", collector_line_collector_number="158") + + result = run_join_key_calculator(dry_run=False) + + assert result.votes_written == 1 + + def test_fallback_calculator_excludes_a_card_with_a_prior_deduction_vote(self, db): + card = CardFactory(name="Some Card", content_phash=42) + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") + _evidence(card, layout_class="black") + CardScanLog.objects.create(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason="no-text") + CardPrintingTag.objects.create( + card=card, + printing=printing, + is_no_match=False, + anonymous_id="deductive-backfill-v1", + source=VoteSource.DEDUCTION, + ) + + result = run_fallback_calculator(dry_run=False) + + assert result.cards_considered == 0 + assert CardPrintingTag.objects.filter(anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID).count() == 0 + + def test_exclusion_generalizes_to_any_deduction_source_vote_not_just_the_literal_identity(self, db): + """Deliberate generalization (module docstring's PIECE 2): filtered by + source=VoteSource.DEDUCTION, not the literal 'deductive-backfill-v1' anonymous_id - a + differently-named future deduction engine is excluded the exact same way.""" + card = CardFactory(name="Some Card", content_phash=42) + printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") + _evidence(card, collector_line_set_code="mom", collector_line_collector_number="158") + CardPrintingTag.objects.create( + card=card, + printing=printing, + is_no_match=False, + anonymous_id="some-future-deduction-engine-v1", + source=VoteSource.DEDUCTION, + ) + + result = run_join_key_calculator(dry_run=False) + + assert result.cards_considered == 0 + + def test_running_the_fire_twice_does_not_double_vote_via_either_mechanism(self, db): + """Exercises both idempotence mechanisms a repeated multi-pass Stage D fire relies on, in + one test: the calculator's own pre-existing anonymous_id self-exclusion (card_a, voted by + THIS calculator's own first pass) and PIECE 2's new deduction-source exclusion (card_b, + already voted by a prior deductive-backfill pass before this calculator ever saw it).""" + card_a = CardFactory(name="Card A", content_phash=1) + CanonicalCardFactory(name="Card A", expansion__code="mom", collector_number="158") + _evidence(card_a, collector_line_set_code="mom", collector_line_collector_number="158") + + card_b = CardFactory(name="Card B", content_phash=2) + printing_b = CanonicalCardFactory(name="Card B", expansion__code="vow", collector_number="200") + _evidence(card_b, collector_line_set_code="vow", collector_line_collector_number="200") + CardPrintingTag.objects.create( + card=card_b, + printing=printing_b, + is_no_match=False, + anonymous_id="deductive-backfill-v1", + source=VoteSource.DEDUCTION, + ) + + first = run_join_key_calculator(dry_run=False) + assert first.cards_considered == 1 # only card_a - card_b already excluded via PIECE 2 + assert CardPrintingTag.objects.filter(card=card_a, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 1 + assert CardPrintingTag.objects.filter(card=card_b, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 0 + + second = run_join_key_calculator(dry_run=False) + # card_a: excluded by its own anonymous_id now carrying a vote; card_b: still + # deduction-excluded, exactly as on the first pass. + assert second.cards_considered == 0 + assert CardPrintingTag.objects.filter(card=card_a, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 1 + assert CardPrintingTag.objects.filter(card=card_b, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 0 diff --git a/docs/features/catalog-completion-plan.md b/docs/features/catalog-completion-plan.md index 4775ba3a6..cca00a573 100644 --- a/docs/features/catalog-completion-plan.md +++ b/docs/features/catalog-completion-plan.md @@ -2624,6 +2624,56 @@ clean (no model change — every field these checks read already existed on votes, no backup needed, since nothing is persisted. Production cutover is the only phase that uses `--write --run-id `. +**Pre-fire prep — the fallback channel calculator + constant #3 (built +2026-07-22, bundled ahead of the full-catalog fire per owner ruling; see +[[../pipeline-fidelity-gate.md]] §3 for the gate context)**: two pieces, +both code-only — neither runs the full-catalog fire, the targeted +re-extraction of issue #340's 373-card cohort, or any other prod +extraction/write, both of which remain separate, owner-gated prod steps. + +1. **`calculate_fallback_verdict`/`run_fallback_calculator`** (own + `anonymous_id="stage-d-fallback-v1"`) — Stage D's own port of + `local_fallback.py`'s pilot "Pass 2" evidence-combination model (that + module's own docstring: fires only when pass 1 yields no accepted vote + for a card), run over exactly the cards `run_join_key_calculator` + already concluded have no confident hit — the SAME population + `run_slow_path_calculator` routes to human review. Unlike the pilot's + own `run_fallback_for_card` (which crops/scans a LIVE image), this + calculator operates entirely off already-persisted `ImageEvidence` + fields: `layout_class`/`artist_ocr_name` feed straight into + `local_fallback.filter_by_border_color`/`match_artist` (PROTECTED CORE, + called not modified), and a new `_filter_by_symbol_phash` compares + `symbol_phash` against each candidate's rendered keyrune glyph + (`local_fallback.render_set_symbol`, PROTECTED CORE) via the same + pure-Hamming-distance-arithmetic reimplementation + `_symbol_phash_tiebreak` already established for the join-key + calculator's own symbol tie-break. A vote is cast ONLY when the + intersection across every sub-check that produced a reading narrows to + EXACTLY ONE candidate — `local_fallback.py`'s own documented rule, + reproduced exactly, with no added agreement/corroboration layer (that + layer doesn't exist in `local_fallback.py`, so this is a faithful port, + not an augmented one). `source=VoteSource.OCR` (not `DEDUCTION`) — the + enum's own docstring in `models.py` explicitly names "the + border/artist/symbol evidence-combination fallback" as part of OCR's + own umbrella definition. Wired into the management command between the + join-key and slow-path calculators (its own gate check via + `verify_zero_resolutions`, same as the join-key calculator's own), and + `_slow_path_eligible_cards_queryset` now also excludes any card this + calculator successfully voted on, so a card it resolves is not also + routed to human review in the same invocation. +2. **Constant #3** (`docs/pipeline-fidelity-gate.md` §3 item 3 / + `docs/reports/2026-07-22-knowledge-inventory.md`'s MISSING item 3): the + shared `_eligible_cards_queryset` helper (both calculators call it) now + also excludes any card already carrying a `VoteSource.DEDUCTION` + printing vote — the pilot never re-voted a card + `deductive_backfill.py`'s own `deductive-backfill-v1` pass had already + voted for (28,112 live production votes). Filtered by + `source=VoteSource.DEDUCTION` rather than the literal `anonymous_id` + value — deliberately generalized so any future deduction-class engine + is excluded the same way. Constants #1 (`RESOLUTION_FLOOR_DPI`) and #2 + (`EXCLUDED_RESOLVED_TAGS`) are explicitly NOT part of this bundle — + their forward-impact sizing is still open per the gate page. + **Stage E resume contract (owner directive, 2026-07-19 — full spec on task #147, acceptance test folded into task #156's soak gate)**: resumability is a TESTED requirement, not an assumed property — this diff --git a/docs/pipeline-fidelity-gate.md b/docs/pipeline-fidelity-gate.md index c23adb932..d343edc8e 100644 --- a/docs/pipeline-fidelity-gate.md +++ b/docs/pipeline-fidelity-gate.md @@ -69,11 +69,17 @@ The knowledge-inventory sweep confirmed three pilot-era constants have live: all 28,112 carry `run_id=None`, `anonymous_id="deductive-backfill-v1"`, `created_at` between 2026-07-14T18:21:49Z and 2026-07-14T18:22:05Z. See `journal/2026-07-14-deductive-printing-tag-backfill.md` (gitignored, - machine-local) for that run's own narrative. + machine-local) for that run's own narrative. **ADDRESSED IN CODE + (2026-07-22, pre-fire prep PR)**: `local_calculate_verdicts._eligible_cards_queryset` + now excludes `source=VoteSource.DEDUCTION` (generalized past the + literal `anonymous_id`, so a future second deduction engine is covered + too), applied to both the join-key AND new fallback calculators — code + only, not yet fired against production. None of these three are soundness violations — the human-backed consensus gate still applies to every vote Stage D casts regardless. -**Owner ruling needed**: are these three must-fix-before-fire, or an +**Owner ruling still needed on #1/#2** (item #3 above is now addressed in +code, pending the fire): are the remaining two must-fix-before-fire, or an accepted gap the gate can clear without them? Full detail, plus 3 lower grade "open items" that are separate from these 3 MISSING findings: [`reports/2026-07-22-knowledge-inventory.md`](reports/2026-07-22-knowledge-inventory.md). diff --git a/docs/reports/2026-07-22-knowledge-inventory.md b/docs/reports/2026-07-22-knowledge-inventory.md index 26a965d08..7989c5fb8 100644 --- a/docs/reports/2026-07-22-knowledge-inventory.md +++ b/docs/reports/2026-07-22-knowledge-inventory.md @@ -96,7 +96,7 @@ list is authoritative and was used, not the informal comment. | constant | value | current home | status | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | -| `FALLBACK_CONFIDENCE_MULTI_EVIDENCE=0.8`, `FALLBACK_CONFIDENCE_SINGLE_EVIDENCE=0.7` | unchanged | still exported; the fallback engine itself (border+artist+symbol evidence-combination) has no Stage D calculator yet — explicitly deferred, not silently dropped (see below) | SAME (engine not yet ported to Stage D, openly deferred) | +| `FALLBACK_CONFIDENCE_MULTI_EVIDENCE=0.8`, `FALLBACK_CONFIDENCE_SINGLE_EVIDENCE=0.7` | unchanged | imported verbatim (not duplicated) by Stage D's own `calculate_fallback_verdict` (2026-07-22 pre-fire prep PR — see below, no longer deferred) | SAME, actively reused | | `BORDER_ATTRIBUTE_VOTE_CONFIDENCE=0.75`, `GROUND_TRUTH_ATTRIBUTE_VOTE_CONFIDENCE=0.95` | unchanged | same module | SAME | | `ARTIST_CROP_BOX=(0.0,0.82,1.0,1.0)`, `ARTIST_FUZZY_MATCH_THRESHOLD=0.8` | unchanged | `ARTIST_CROP_BOX` consumed by `image_evidence.py`'s `crop_coordinates`; `match_artist` (which uses the 0.8 ratio) called directly by Stage D's artist-OCR corroboration check | SAME, actively reused | | `SYMBOL_STRIP_BOX=(0.78,0.55,1.0,0.80)`, `SYMBOL_DISTANCE_THRESHOLD=6`, `SYMBOL_MARGIN=6`, `SYMBOL_RENDER_SIZE=64` | unchanged | `SYMBOL_STRIP_BOX` consumed by `image_evidence.py`'s `symbol_region` extractor; `SYMBOL_DISTANCE_THRESHOLD`/`SYMBOL_MARGIN`/`render_set_symbol` reused directly by Stage D's `_symbol_phash_tiebreak` | SAME, actively reused | @@ -107,28 +107,28 @@ list is authoritative and was used, not the informal comment. ### `local_identify_printing_tags.py` orchestration constants (legacy engine, not protected core) -| constant | pilot value | current home | status | evidence | -| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `OCR_ANONYMOUS_ID`, `PHASH_ANONYMOUS_ID` | as stated | unchanged in legacy engine | SAME | diff | -| `OCR_CONFIDENCE_BOTH=0.85`, `OCR_CONFIDENCE_COLLECTOR_ONLY=0.75` | as stated | unchanged in legacy; duplicated (not imported, deliberately — avoids a hard cross-module import-time dependency) into Stage D as `JOIN_KEY_CONFIDENCE_BOTH=0.85`/`JOIN_KEY_CONFIDENCE_COLLECTOR_ONLY=0.75` | SAME VALUE, moved/duplicated into Stage D | `local_calculate_verdicts.py` comment: "Same two-tier split... duplicated as literals rather than imported" | -| — (no pilot analogue) | n/a | `JOIN_KEY_CONFIDENCE_SYMBOL_TIEBREAK=0.75` | NEW in Stage D (the symbol-phash tie-break didn't exist as a join-key concept in the pilot's 3-channel model) | `local_calculate_verdicts.py` §7 framing | -| — (no pilot analogue) | n/a | `JOIN_KEY_CONFIDENCE_ARTIST_DISAGREEMENT=0.65` | NEW in Stage D, precedent from issue #207's `OCR_NO_MATCH_CONFIDENCE`/`FALLBACK_NO_MATCH_CONFIDENCE=0.6` pattern | `local_calculate_verdicts.py` comment | -| `PHASH_MAX_CANDIDATES=12` | see phash section above | | | | -| `DEFAULT_BATCH_SIZE=25` (per-chunk flush + gate-check granularity — "a kill loses at most one batch") | as stated | Stage D's `run_join_key_calculator` accumulates ALL votes/scan-log rows across the ENTIRE eligible-cohort loop and does one `bulk_create` + one gate check only at the very end of the whole run (no periodic flush observed anywhere in `local_calculate_verdicts.py`) | **CHANGED, unexplained — open item, see below** | read `run_join_key_calculator`'s full body: `votes_batch`/`scan_log_batch` only flushed after the `for card in ...` loop exits | -| `EXCLUDED_RESOLVED_TAGS = ["custom-art", "non-english"]` | excludes cards already carrying a resolved custom-art/non-english tag from selection | **absent from Stage D's `_eligible_cards_queryset`** | **MISSING — see MISSING section** | `grep` for `custom-art`/`non-english`/`EXCLUDED_RESOLVED_TAGS`/`tags__contains` across `local_calculate_verdicts.py`, `image_evidence.py`, `run_image_evidence_cohort.py`: zero hits | -| `.exclude(printing_tags__anonymous_id=DEDUCTIVE_BACKFILL_ANONYMOUS_ID)` (don't pile a weaker vote onto a card the exact-by-construction deductive backfill already covered) | as stated | absent from Stage D's `_eligible_cards_queryset` | MISSING (lower severity — see MISSING section) | same `_eligible_cards_queryset` read | -| `RESOLUTION_FLOOR_DPI = 200` (empirically-validated: dpi≤150 degrades OCR yield, dpi≥200 matches/exceeds native — cards below this floor are never even fetched) | as stated | legacy engine retains it; **no analogue anywhere in `run_image_evidence_cohort.py`/`image_evidence.py`/`local_calculate_verdicts.py`** — Stage C's cohort selection filters only on `content_phash__isnull=False`, no `dpi` condition | **MISSING — see MISSING section** | `grep -rn "RESOLUTION_FLOOR\|dpi__lt\|floor"` across the three Stage C/D files: zero hits; cohort selection query read directly (`run_image_evidence_cohort.py` lines ~618–624) | -| `DEFAULT_FETCH_DPI = 250` (via `image_cdn_fetch.py`, imported not duplicated) | as stated | unchanged; same module, same value, used by both the legacy engine and Stage C | SAME | `image_cdn_fetch.py` at both commits: identical `DEFAULT_FETCH_DPI: Optional[int] = 250` | -| `DEFAULT_WORKERS = 2` (measured live against a 2-CPU-core box, "matches the core count exactly") | as stated | legacy engine retains `2`; Stage C's `run_image_evidence_cohort.py` sets `DEFAULT_WORKERS = 7` (env `STAGE_C_WORKERS`, default `"7"`) and `DEFAULT_FETCH_THREADS = 8` | **DELIBERATELY CHANGED** — re-measured after the box was resized to 7 usable cores (Stage C canary work, PR #224 + the 2026-07-20 63.1%-efficiency reprofile) | `run_image_evidence_cohort.py` line 176/180; `docs/features/catalog-completion-plan.md`'s Stage C section | -| Fetch-budget mechanism (`--fetch-budget`, counts real CDN requests, protects the Worker's shared rate limit) | as stated | legacy retains `--fetch-budget`; Stage C's real protection is `harvest_fetch_limiter.py`'s `GOOGLE_IMAGE` token-bucket rate limiter (8.0/s ceiling, task #165's concurrency-raise probe, composes with `local_phash.py`'s own `_RateLimiter`) | **DELIBERATELY CHANGED** — moved from a count-based budget to a real-time rate limiter, a strictly more precise mechanism for the same protective purpose | `local_phash.py`'s `DEFAULT_BACKFILL_RATE_LIMIT_PER_SEC` comment; `harvest_fetch_limiter.py` module | -| `verify_zero_resolutions` (the human-backed consensus gate check, re-derived post-write and must find 0 violations) | as stated, checked after every batch flush | **reused directly** (imported, not reimplemented) by Stage D's management command, checked once at end of run; measured 0/8,925 on the 2026-07-21 `staged-write-20260721T0434Z` run | SAME mechanism, actively reused, and independently re-verified | `management/commands/local_calculate_verdicts.py` imports `verify_zero_resolutions` from `local_identify_printing_tags`; `theory.md` §7b | -| `generate_run_id`, `CandidateNameIndex`, `CandidatePrinting` | as stated | reused directly (imported) by Stage D | SAME, actively reused | `local_calculate_verdicts.py` imports | -| Coverage-gap + abstention-aware selection ordering (`_coverage_priority_key`, `HARD_NAME_MIN_ATTEMPTS=5`) | pilot-era queue-ordering heuristic for a per-invocation selection loop | legacy-only concept; Stage C's own cohort-selection in `run_image_evidence_cohort.py` uses a _different_, independently-implemented edhrec-rank priority ordering (`name_rank`/`priority_key`, cold-tail-last) | analogous re-implementation, not shared code — not itself concerning, but two independent orderings exist for two different populations | `run_image_evidence_cohort.py` "Step 1"/"priority_key" | -| `NAME_FREQUENCY_ANONYMOUS_ID`, `NAME_FREQUENCY_CONFIDENCE=0.6` (structural deduction-by-elimination, no image evidence at all) | as stated | legacy-only; no Stage D analogue found | **OPEN ITEM, not classified as hard MISSING** — this is a structural (non-visual) deduction outside Stage D's own stated join-key scope, and may simply not have been in scope for porting yet | `grep` across Stage D files found no analogue; not addressed one way or the other in `catalog-completion-plan.md`'s Stage D section | -| `cast_border_attribute_vote`/`cast_frame_style_vote` "ground-truth-preferred" override (once a printing is confirmed this run, prefer Scryfall's own border/frame value over the pixel/OCR estimate for the ATTRIBUTE-chip vote) | as stated | legacy-only; not observed in Stage D (Stage D casts no attribute-chip votes at all — only the printing `CardPrintingTag`) | superseded-by-architecture (attribute-chip voting stayed with the legacy engine; Stage D is printing-identification only) | `local_calculate_verdicts.py` casts no `CardTagVote` | -| Two-engine "disagreement" handling (`disagreement-with-other-engine` skip, only relevant when OCR and phash are separate parallel channels that can disagree) | as stated | no Stage D analogue — structurally doesn't apply, since collector-line-OCR + set-symbol-phash are ONE join key, not two channels that can disagree | superseded-by-architecture (explicitly, by design) | `local_calculate_verdicts.py` module docstring: "not three parallel decoders... ONE near-unique join key" | -| `local_clustering.compute_two_threshold_clusters` (`NEAR_DUPLICATE_MAX_DISTANCE=2`, d=0 exact / d≤2 near-duplicate two-tier printing-vote propagation) | as stated | `local_clustering.py` is byte-for-byte unchanged since the pilot commit (`diff` empty); still called by the legacy engine's `run_pilot`. **No equivalent PRINTING-vote propagation call site found in Stage D** — the only current d=0-propagation code is `local_residual_classify.run_d0_sibling_artist_propagation`, which is ARTIST-vote-scoped, not printing-vote-scoped | partially superseded / **open item, not confirmed absent by exhaustive search** — flagged honestly rather than either asserted MISSING or waved through | `local_clustering.py` diff empty; `grep` for calls to `compute_two_threshold_clusters` outside `local_identify_printing_tags.py` found none in Stage D files | -| Fallback engine's own candidate-narrowing evidence-combination model (border+artist+symbol intersection, `run_fallback_for_card`) | pilot's "Pass 2" | not yet ported to Stage D — explicitly named in `local_calculate_verdicts.py`'s own "STILL DEFERRED" list as a distinct future item, alongside slow-path visual matching | **openly deferred in-code, not silently dropped** | `local_calculate_verdicts.py` module docstring's "STILL DEFERRED" section | +| constant | pilot value | current home | status | evidence | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `OCR_ANONYMOUS_ID`, `PHASH_ANONYMOUS_ID` | as stated | unchanged in legacy engine | SAME | diff | +| `OCR_CONFIDENCE_BOTH=0.85`, `OCR_CONFIDENCE_COLLECTOR_ONLY=0.75` | as stated | unchanged in legacy; duplicated (not imported, deliberately — avoids a hard cross-module import-time dependency) into Stage D as `JOIN_KEY_CONFIDENCE_BOTH=0.85`/`JOIN_KEY_CONFIDENCE_COLLECTOR_ONLY=0.75` | SAME VALUE, moved/duplicated into Stage D | `local_calculate_verdicts.py` comment: "Same two-tier split... duplicated as literals rather than imported" | +| — (no pilot analogue) | n/a | `JOIN_KEY_CONFIDENCE_SYMBOL_TIEBREAK=0.75` | NEW in Stage D (the symbol-phash tie-break didn't exist as a join-key concept in the pilot's 3-channel model) | `local_calculate_verdicts.py` §7 framing | +| — (no pilot analogue) | n/a | `JOIN_KEY_CONFIDENCE_ARTIST_DISAGREEMENT=0.65` | NEW in Stage D, precedent from issue #207's `OCR_NO_MATCH_CONFIDENCE`/`FALLBACK_NO_MATCH_CONFIDENCE=0.6` pattern | `local_calculate_verdicts.py` comment | +| `PHASH_MAX_CANDIDATES=12` | see phash section above | | | | +| `DEFAULT_BATCH_SIZE=25` (per-chunk flush + gate-check granularity — "a kill loses at most one batch") | as stated | Stage D's `run_join_key_calculator` accumulates ALL votes/scan-log rows across the ENTIRE eligible-cohort loop and does one `bulk_create` + one gate check only at the very end of the whole run (no periodic flush observed anywhere in `local_calculate_verdicts.py`) | **CHANGED, unexplained — open item, see below** | read `run_join_key_calculator`'s full body: `votes_batch`/`scan_log_batch` only flushed after the `for card in ...` loop exits | +| `EXCLUDED_RESOLVED_TAGS = ["custom-art", "non-english"]` | excludes cards already carrying a resolved custom-art/non-english tag from selection | **absent from Stage D's `_eligible_cards_queryset`** | **MISSING — see MISSING section** | `grep` for `custom-art`/`non-english`/`EXCLUDED_RESOLVED_TAGS`/`tags__contains` across `local_calculate_verdicts.py`, `image_evidence.py`, `run_image_evidence_cohort.py`: zero hits | +| `.exclude(printing_tags__anonymous_id=DEDUCTIVE_BACKFILL_ANONYMOUS_ID)` (don't pile a weaker vote onto a card the exact-by-construction deductive backfill already covered) | as stated | absent from Stage D's `_eligible_cards_queryset` | MISSING (lower severity — see MISSING section) | same `_eligible_cards_queryset` read | +| `RESOLUTION_FLOOR_DPI = 200` (empirically-validated: dpi≤150 degrades OCR yield, dpi≥200 matches/exceeds native — cards below this floor are never even fetched) | as stated | legacy engine retains it; **no analogue anywhere in `run_image_evidence_cohort.py`/`image_evidence.py`/`local_calculate_verdicts.py`** — Stage C's cohort selection filters only on `content_phash__isnull=False`, no `dpi` condition | **MISSING — see MISSING section** | `grep -rn "RESOLUTION_FLOOR\|dpi__lt\|floor"` across the three Stage C/D files: zero hits; cohort selection query read directly (`run_image_evidence_cohort.py` lines ~618–624) | +| `DEFAULT_FETCH_DPI = 250` (via `image_cdn_fetch.py`, imported not duplicated) | as stated | unchanged; same module, same value, used by both the legacy engine and Stage C | SAME | `image_cdn_fetch.py` at both commits: identical `DEFAULT_FETCH_DPI: Optional[int] = 250` | +| `DEFAULT_WORKERS = 2` (measured live against a 2-CPU-core box, "matches the core count exactly") | as stated | legacy engine retains `2`; Stage C's `run_image_evidence_cohort.py` sets `DEFAULT_WORKERS = 7` (env `STAGE_C_WORKERS`, default `"7"`) and `DEFAULT_FETCH_THREADS = 8` | **DELIBERATELY CHANGED** — re-measured after the box was resized to 7 usable cores (Stage C canary work, PR #224 + the 2026-07-20 63.1%-efficiency reprofile) | `run_image_evidence_cohort.py` line 176/180; `docs/features/catalog-completion-plan.md`'s Stage C section | +| Fetch-budget mechanism (`--fetch-budget`, counts real CDN requests, protects the Worker's shared rate limit) | as stated | legacy retains `--fetch-budget`; Stage C's real protection is `harvest_fetch_limiter.py`'s `GOOGLE_IMAGE` token-bucket rate limiter (8.0/s ceiling, task #165's concurrency-raise probe, composes with `local_phash.py`'s own `_RateLimiter`) | **DELIBERATELY CHANGED** — moved from a count-based budget to a real-time rate limiter, a strictly more precise mechanism for the same protective purpose | `local_phash.py`'s `DEFAULT_BACKFILL_RATE_LIMIT_PER_SEC` comment; `harvest_fetch_limiter.py` module | +| `verify_zero_resolutions` (the human-backed consensus gate check, re-derived post-write and must find 0 violations) | as stated, checked after every batch flush | **reused directly** (imported, not reimplemented) by Stage D's management command, checked once at end of run; measured 0/8,925 on the 2026-07-21 `staged-write-20260721T0434Z` run | SAME mechanism, actively reused, and independently re-verified | `management/commands/local_calculate_verdicts.py` imports `verify_zero_resolutions` from `local_identify_printing_tags`; `theory.md` §7b | +| `generate_run_id`, `CandidateNameIndex`, `CandidatePrinting` | as stated | reused directly (imported) by Stage D | SAME, actively reused | `local_calculate_verdicts.py` imports | +| Coverage-gap + abstention-aware selection ordering (`_coverage_priority_key`, `HARD_NAME_MIN_ATTEMPTS=5`) | pilot-era queue-ordering heuristic for a per-invocation selection loop | legacy-only concept; Stage C's own cohort-selection in `run_image_evidence_cohort.py` uses a _different_, independently-implemented edhrec-rank priority ordering (`name_rank`/`priority_key`, cold-tail-last) | analogous re-implementation, not shared code — not itself concerning, but two independent orderings exist for two different populations | `run_image_evidence_cohort.py` "Step 1"/"priority_key" | +| `NAME_FREQUENCY_ANONYMOUS_ID`, `NAME_FREQUENCY_CONFIDENCE=0.6` (structural deduction-by-elimination, no image evidence at all) | as stated | legacy-only; no Stage D analogue found | **OPEN ITEM, not classified as hard MISSING** — this is a structural (non-visual) deduction outside Stage D's own stated join-key scope, and may simply not have been in scope for porting yet | `grep` across Stage D files found no analogue; not addressed one way or the other in `catalog-completion-plan.md`'s Stage D section | +| `cast_border_attribute_vote`/`cast_frame_style_vote` "ground-truth-preferred" override (once a printing is confirmed this run, prefer Scryfall's own border/frame value over the pixel/OCR estimate for the ATTRIBUTE-chip vote) | as stated | legacy-only; not observed in Stage D (Stage D casts no attribute-chip votes at all — only the printing `CardPrintingTag`) | superseded-by-architecture (attribute-chip voting stayed with the legacy engine; Stage D is printing-identification only) | `local_calculate_verdicts.py` casts no `CardTagVote` | +| Two-engine "disagreement" handling (`disagreement-with-other-engine` skip, only relevant when OCR and phash are separate parallel channels that can disagree) | as stated | no Stage D analogue — structurally doesn't apply, since collector-line-OCR + set-symbol-phash are ONE join key, not two channels that can disagree | superseded-by-architecture (explicitly, by design) | `local_calculate_verdicts.py` module docstring: "not three parallel decoders... ONE near-unique join key" | +| `local_clustering.compute_two_threshold_clusters` (`NEAR_DUPLICATE_MAX_DISTANCE=2`, d=0 exact / d≤2 near-duplicate two-tier printing-vote propagation) | as stated | `local_clustering.py` is byte-for-byte unchanged since the pilot commit (`diff` empty); still called by the legacy engine's `run_pilot`. **No equivalent PRINTING-vote propagation call site found in Stage D** — the only current d=0-propagation code is `local_residual_classify.run_d0_sibling_artist_propagation`, which is ARTIST-vote-scoped, not printing-vote-scoped | partially superseded / **open item, not confirmed absent by exhaustive search** — flagged honestly rather than either asserted MISSING or waved through | `local_clustering.py` diff empty; `grep` for calls to `compute_two_threshold_clusters` outside `local_identify_printing_tags.py` found none in Stage D files | +| Fallback engine's own candidate-narrowing evidence-combination model (border+artist+symbol intersection, `run_fallback_for_card`) | pilot's "Pass 2" | **PORTED 2026-07-22** (pre-fire prep PR): `calculate_fallback_verdict`/`run_fallback_calculator` (own `anonymous_id="stage-d-fallback-v1"`) reproduce the same border/artist/symbol intersection off already-persisted `ImageEvidence` fields, calling `local_fallback.filter_by_border_color`/`match_artist`/`render_set_symbol` (PROTECTED CORE, called not modified) — no live-image re-scan, unlike the pilot's own live-crop version | SAME decision model, ported (was: openly deferred in-code) | `local_calculate_verdicts.py` module docstring's PIECE 1 section; `docs/features/catalog-completion-plan.md`'s "Pre-fire prep" entry | ### `local_calculate_verdicts.py` (Stage D) — new constants with no pilot precedent @@ -167,7 +167,7 @@ Three items, confirmed absent by direct `grep`/read of `local_calculate_verdicts 2. **`EXCLUDED_RESOLVED_TAGS = ["custom-art", "non-english"]`** (pilot). A card already carrying a resolved custom-art or non-english tag has its printing-identification precondition (an authentic depiction of a real printing) already falsified — the pilot excluded these from selection entirely. Stage D's `_eligible_cards_queryset` in `local_calculate_verdicts.py` has no equivalent `.exclude(tags__contains=...)` clause. Confirmed by `grep` for `custom-art`/`non-english`/`EXCLUDED_RESOLVED_TAGS`/`tags__contains` across all three Stage C/D files: zero hits. Checked for a same-effect-different-mechanism explanation before calling this MISSING (not just absence-of-the-literal-clause): `tag_consensus.py`'s `resolve_and_persist_tag_votes` only ever writes `card.tags`/`card.tag_vote_statuses` — it never touches `printing_tag_status` or `canonical_card`, so a resolved custom-art/non-english tag does NOT get a card excluded through Stage D's `printing_tag_status=UNRESOLVED`/`canonical_card__isnull=True` filters either; there is no other path that produces the pilot's exclusion effect. This does not create a soundness violation (the human-backed consensus gate still applies to any vote cast), but it does mean Stage D would spend real OCR/phash effort — and potentially cast a real `is_no_match` or match vote — on a card whose own tags already say the exercise is moot, which the pilot's own design explicitly avoided. Not previously tracked anywhere found in `docs/features/catalog-completion-plan.md`/`docs/theory.md` — appears to be a genuinely new finding from this sweep, not a known, already-accepted gap. -3. **`.exclude(printing_tags__anonymous_id=DEDUCTIVE_BACKFILL_ANONYMOUS_ID)`** (pilot). Lower severity than the two above, and narrower than it first looks: verified against `printing_consensus.py` directly — a card deductive backfill has actually _resolved_ (`resolve_and_persist_printing` sets `inferred_canonical_card` + `printing_tag_status=RESOLVED`) is already excluded from Stage D's query by its own pre-existing `printing_tag_status=UNRESOLVED` filter, no separate exclusion needed. This gap therefore only bites the narrower "deductive backfill already cast a vote for this card, but it's still sitting UNRESOLVED (pending human confirmation, or the vote alone wasn't sufficient)" subset — the pilot deliberately didn't let its own weaker engines pile a redundant vote onto exactly that subset. Since Stage D's own vote is still gated by the same human-backed consensus rule, this is a redundancy/efficiency gap on that narrower subset, not a soundness one — included for completeness, not alarm. +3. **`.exclude(printing_tags__anonymous_id=DEDUCTIVE_BACKFILL_ANONYMOUS_ID)`** (pilot). Lower severity than the two above, and narrower than it first looks: verified against `printing_consensus.py` directly — a card deductive backfill has actually _resolved_ (`resolve_and_persist_printing` sets `inferred_canonical_card` + `printing_tag_status=RESOLVED`) is already excluded from Stage D's query by its own pre-existing `printing_tag_status=UNRESOLVED` filter, no separate exclusion needed. This gap therefore only bites the narrower "deductive backfill already cast a vote for this card, but it's still sitting UNRESOLVED (pending human confirmation, or the vote alone wasn't sufficient)" subset — the pilot deliberately didn't let its own weaker engines pile a redundant vote onto exactly that subset. Since Stage D's own vote is still gated by the same human-backed consensus rule, this is a redundancy/efficiency gap on that narrower subset, not a soundness one — included for completeness, not alarm. **ADDRESSED IN CODE (2026-07-22, pre-fire prep PR)**: `_eligible_cards_queryset` now excludes `source=VoteSource.DEDUCTION` (deliberately generalized past the literal `anonymous_id`), shared by both the join-key and new fallback calculators — code only, not yet fired against production. No other pilot-era constant, threshold, crop box, regex, or skip-reason was found with genuinely no current home; every other divergence found a place in one of the non-MISSING buckets above, including several where the current home is the SAME code, literally imported and called (not reimplemented) by the new pipeline. From 55397ed5f131abaa0669c8c1cb4efb3ee770905b Mon Sep 17 00:00:00 2001 From: wilfordgrimley <2397930+WilfordGrimley@users.noreply.github.com> Date: Wed, 22 Jul 2026 21:58:51 +0000 Subject: [PATCH 2/2] Drop constant #3 (deductive-backfill exclusion) per owner ruling Owner ruled: the 2026-07-14 backfill is pure name/metadata deduction with sound votes (15-card sample all correct); excluding those cards would strand ~27,819 sound-but-UNRESOLVED cards for no protective benefit, since the human-backed consensus gate already makes re-evaluation safe. Removes the .exclude(printing_tags__source=VoteSource.DEDUCTION) clause and its tests; the pre-existing stable-anonymous_id idempotence exclusion is untouched. Fallback channel calculator unchanged. Docs reframed to record this as an intentional non-restoration, not a gap. Co-Authored-By: Claude Fable 5 --- .../cardpicker/local_calculate_verdicts.py | 63 ++++++----- .../tests/test_local_calculate_verdicts.py | 102 ----------------- docs/features/catalog-completion-plan.md | 106 ++++++++++-------- docs/pipeline-fidelity-gate.md | 30 +++-- .../reports/2026-07-22-knowledge-inventory.md | 2 +- 5 files changed, 114 insertions(+), 189 deletions(-) diff --git a/MPCAutofill/cardpicker/local_calculate_verdicts.py b/MPCAutofill/cardpicker/local_calculate_verdicts.py index d62063c2b..fc38ae66e 100644 --- a/MPCAutofill/cardpicker/local_calculate_verdicts.py +++ b/MPCAutofill/cardpicker/local_calculate_verdicts.py @@ -267,24 +267,31 @@ ambiguous) is NOT excluded - it still has no confident automated hit and belongs in the review queue exactly as before. -PIECE 2 - CONSTANT #3 (`docs/pipeline-fidelity-gate.md` SS3 item 3, -`docs/reports/2026-07-22-knowledge-inventory.md`'s MISSING item 3): `_eligible_cards_queryset` now -also excludes any card already carrying a `VoteSource.DEDUCTION` printing vote - the pilot never -re-voted a card `deductive_backfill.py`'s own `DEDUCTIVE_BACKFILL_ANONYMOUS_ID="deductive-backfill-v1"` -pass had already voted for (28,112 live production votes, `run_id=None`, per the gate page's SS6). -Filtered by `source=VoteSource.DEDUCTION` rather than the literal `anonymous_id` value - -deliberately generalized (this PR's own task brief) so ANY prior deduction-class vote (a future -second deduction engine, not just this one literal identity) is excluded the same way, without a -hard import-time dependency on `deductive_backfill.py`'s own constant (matching this module's own -`JOIN_KEY_CONFIDENCE_BOTH` comment's "avoid a hard cross-module dependency over one constant" -precedent) - `deductive_backfill.py`'s own module docstring confirms it is the sole producer of -`VoteSource.DEDUCTION` votes today, so this changes nothing operationally now and only -future-proofs. Applied inside the ONE shared `_eligible_cards_queryset` helper both the join-key -and fallback calculators call - the operational guard that makes a repeated multi-pass Stage D -fire idempotent against the backfill population, independent of which calculator runs. Explicitly -NOT bundling constants #1 (`RESOLUTION_FLOOR_DPI`) or #2 (`EXCLUDED_RESOLVED_TAGS`) here - their -forward-impact sizing is still open per the gate page, and the owner did not include them in this -PRE-FIRE PREP. +CONSTANT #3 - INTENTIONALLY NOT RESTORED (owner ruling, 2026-07-22, superseding an earlier draft +of this PR that DID add a `.exclude(printing_tags__source=VoteSource.DEDUCTION)` clause here): +`docs/pipeline-fidelity-gate.md` SS3 item 3 / `docs/reports/2026-07-22-knowledge-inventory.md`'s +former MISSING item 3 flagged that the pilot never re-voted a card +`deductive_backfill.py`'s own `DEDUCTIVE_BACKFILL_ANONYMOUS_ID="deductive-backfill-v1"` pass had +already voted for (28,112 live production votes, `run_id=None`, per the gate page's SS6). A +read-only backfill investigation found this exclusion would be a net-negative single-cohort +carve-out, not a restoration worth making: that 2026-07-14 backfill is pure name/metadata +deduction (never phash/OCR - zero image inspection, see `deductive_backfill.py`'s own module +docstring), its votes are sound (a 15-card sample checked all correct), and excluding those cards +from Stage D would strand ~27,819 sound-but-UNRESOLVED cards outside the new pipeline for no +protective benefit. Re-evaluating them is safe: the human-backed consensus gate +(`vote_consensus.resolve_weighted_consensus`, PROTECTED CORE, unmodified) still prevents any +machine-only vote accumulation from resolving a card by itself regardless of how many engines +vote on it, agreement between the backfill's vote and a fresh Stage D vote simply dedups (no harm +done), and a disagreement surfaces the card to human review (a genuine corroboration signal, not +noise). The pilot's own exclusion was a PERFORMANCE optimization (skip a card its own weaker +engines couldn't add anything to), not a soundness mechanism - restoring it here would trade real +coverage for a protection Stage D's vote-consensus layer already provides independently. +`_eligible_cards_queryset`'s pre-existing per-calculator stable-`anonymous_id` exclusion (its own +long-standing idempotence mechanism, entirely independent of this constant) is unaffected and +unchanged by this decision - see that function's own docstring. + +Constants #1 (`RESOLUTION_FLOOR_DPI`) and #2 (`EXCLUDED_RESOLVED_TAGS`) remain open pending +forward-impact sizing per the gate page, unaffected by this ruling on #3. This PR is CODE ONLY: it does not run the full-catalog Stage D fire, the targeted re-extraction of issue #340's 373-card cohort, or any other prod extraction/write - both remain separate, @@ -736,12 +743,15 @@ def _eligible_cards_queryset( instead, since the two calculators' own skip vocabularies mean different things by the same "transient, re-selectable" concept. - CONSTANT #3 (module docstring's PIECE 2, `docs/pipeline-fidelity-gate.md` SS3 item 3): - also excludes any card already carrying a `VoteSource.DEDUCTION` printing vote - see the - module docstring's PIECE 2 section for the full reasoning (why `source=DEDUCTION` rather than - the literal `deductive_backfill.DEDUCTIVE_BACKFILL_ANONYMOUS_ID`). Shared by BOTH calculators - that call this helper (join-key and fallback) - the guard that makes a repeated multi-pass - Stage D fire idempotent against the backfill population, independent of which calculator runs. + Idempotence for a repeated multi-pass Stage D fire comes entirely from the stable, per- + calculator `anonymous_id` exclusion above (`.exclude(printing_tags__anonymous_id=anonymous_id)`) + - deliberately the ONLY vote-population exclusion here. An earlier draft of this module also + excluded any card already carrying a `VoteSource.DEDUCTION` printing vote (the pilot's own + "don't re-vote a card the deductive backfill already covered" behavior, `docs/pipeline- + fidelity-gate.md` SS3 item 3) - owner-ruled OUT (2026-07-22, see module docstring's own + "CONSTANT #3" section for the full reasoning): that exclusion would strand ~27,819 + sound-but-UNRESOLVED cards outside Stage D for no protective benefit, since the human-backed + consensus gate already makes re-evaluating them safe. Do not re-add it without a fresh ruling. """ non_rescannable_scanned_card_ids = ( CardScanLog.objects.filter(anonymous_id=anonymous_id) @@ -755,7 +765,6 @@ def _eligible_cards_queryset( card_type=CardTypes.CARD, ) .exclude(printing_tags__anonymous_id=anonymous_id) - .exclude(printing_tags__source=VoteSource.DEDUCTION) .exclude(pk__in=non_rescannable_scanned_card_ids) .distinct() .select_related("source") @@ -1040,8 +1049,8 @@ def _fallback_eligible_cards_queryset() -> "QuerySet[Card]": `_slow_path_eligible_cards_queryset` below selects from (a real `is_no_match` vote, or a non-rescannable skip in `JOIN_KEY_NO_HIT_SKIP_REASONS`) - that this calculator's own `STAGE_D_FALLBACK_ANONYMOUS_ID` hasn't already processed (scanned OR voted), via the shared - `_eligible_cards_queryset` helper (which also carries PIECE 2's own deduction-vote exclusion, - applied identically to both calculators). + `_eligible_cards_queryset` helper (idempotence mechanism only - see that function's own + docstring for why a deduction-vote exclusion was considered and deliberately not added). """ join_key_no_match_card_ids = CardPrintingTag.objects.filter( anonymous_id=JOIN_KEY_ANONYMOUS_ID, is_no_match=True diff --git a/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py b/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py index 1e5c73dbb..a83ad7c5a 100644 --- a/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py +++ b/MPCAutofill/cardpicker/tests/test_local_calculate_verdicts.py @@ -1363,105 +1363,3 @@ def test_a_card_the_fallback_calculator_only_scanned_is_still_routed(self, db): assert result.cards_considered == 1 assert CardScanLog.objects.filter(anonymous_id=SLOW_PATH_ANONYMOUS_ID, card=card).count() == 1 - - -class TestDeductionVoteExclusion: - """PIECE 2 (module docstring) - constant #3, docs/pipeline-fidelity-gate.md SS3 item 3 / - docs/reports/2026-07-22-knowledge-inventory.md's MISSING item 3. Exercised against BOTH - calculators, since `_eligible_cards_queryset` is shared by both.""" - - def test_join_key_calculator_excludes_a_card_with_a_prior_deduction_vote(self, db): - card = CardFactory(name="Some Card", content_phash=42) - printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") - _evidence(card, collector_line_set_code="mom", collector_line_collector_number="158") - CardPrintingTag.objects.create( - card=card, - printing=printing, - is_no_match=False, - anonymous_id="deductive-backfill-v1", - source=VoteSource.DEDUCTION, - ) - - result = run_join_key_calculator(dry_run=False) - - assert result.cards_considered == 0 - assert CardPrintingTag.objects.filter(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 0 - - def test_join_key_calculator_still_processes_a_card_without_a_deduction_vote(self, db): - card = CardFactory(name="Some Card", content_phash=42) - CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") - _evidence(card, collector_line_set_code="mom", collector_line_collector_number="158") - - result = run_join_key_calculator(dry_run=False) - - assert result.votes_written == 1 - - def test_fallback_calculator_excludes_a_card_with_a_prior_deduction_vote(self, db): - card = CardFactory(name="Some Card", content_phash=42) - printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") - CanonicalPrintingMetadataFactory(canonical_card=printing, border_color="black") - _evidence(card, layout_class="black") - CardScanLog.objects.create(card=card, anonymous_id=JOIN_KEY_ANONYMOUS_ID, skip_reason="no-text") - CardPrintingTag.objects.create( - card=card, - printing=printing, - is_no_match=False, - anonymous_id="deductive-backfill-v1", - source=VoteSource.DEDUCTION, - ) - - result = run_fallback_calculator(dry_run=False) - - assert result.cards_considered == 0 - assert CardPrintingTag.objects.filter(anonymous_id=STAGE_D_FALLBACK_ANONYMOUS_ID).count() == 0 - - def test_exclusion_generalizes_to_any_deduction_source_vote_not_just_the_literal_identity(self, db): - """Deliberate generalization (module docstring's PIECE 2): filtered by - source=VoteSource.DEDUCTION, not the literal 'deductive-backfill-v1' anonymous_id - a - differently-named future deduction engine is excluded the exact same way.""" - card = CardFactory(name="Some Card", content_phash=42) - printing = CanonicalCardFactory(name="Some Card", expansion__code="mom", collector_number="158") - _evidence(card, collector_line_set_code="mom", collector_line_collector_number="158") - CardPrintingTag.objects.create( - card=card, - printing=printing, - is_no_match=False, - anonymous_id="some-future-deduction-engine-v1", - source=VoteSource.DEDUCTION, - ) - - result = run_join_key_calculator(dry_run=False) - - assert result.cards_considered == 0 - - def test_running_the_fire_twice_does_not_double_vote_via_either_mechanism(self, db): - """Exercises both idempotence mechanisms a repeated multi-pass Stage D fire relies on, in - one test: the calculator's own pre-existing anonymous_id self-exclusion (card_a, voted by - THIS calculator's own first pass) and PIECE 2's new deduction-source exclusion (card_b, - already voted by a prior deductive-backfill pass before this calculator ever saw it).""" - card_a = CardFactory(name="Card A", content_phash=1) - CanonicalCardFactory(name="Card A", expansion__code="mom", collector_number="158") - _evidence(card_a, collector_line_set_code="mom", collector_line_collector_number="158") - - card_b = CardFactory(name="Card B", content_phash=2) - printing_b = CanonicalCardFactory(name="Card B", expansion__code="vow", collector_number="200") - _evidence(card_b, collector_line_set_code="vow", collector_line_collector_number="200") - CardPrintingTag.objects.create( - card=card_b, - printing=printing_b, - is_no_match=False, - anonymous_id="deductive-backfill-v1", - source=VoteSource.DEDUCTION, - ) - - first = run_join_key_calculator(dry_run=False) - assert first.cards_considered == 1 # only card_a - card_b already excluded via PIECE 2 - assert CardPrintingTag.objects.filter(card=card_a, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 1 - assert CardPrintingTag.objects.filter(card=card_b, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 0 - - second = run_join_key_calculator(dry_run=False) - # card_a: excluded by its own anonymous_id now carrying a vote; card_b: still - # deduction-excluded, exactly as on the first pass. - assert second.cards_considered == 0 - assert CardPrintingTag.objects.filter(card=card_a, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 1 - assert CardPrintingTag.objects.filter(card=card_b, anonymous_id=JOIN_KEY_ANONYMOUS_ID).count() == 0 diff --git a/docs/features/catalog-completion-plan.md b/docs/features/catalog-completion-plan.md index cca00a573..eb931f7ab 100644 --- a/docs/features/catalog-completion-plan.md +++ b/docs/features/catalog-completion-plan.md @@ -2624,55 +2624,63 @@ clean (no model change — every field these checks read already existed on votes, no backup needed, since nothing is persisted. Production cutover is the only phase that uses `--write --run-id `. -**Pre-fire prep — the fallback channel calculator + constant #3 (built -2026-07-22, bundled ahead of the full-catalog fire per owner ruling; see -[[../pipeline-fidelity-gate.md]] §3 for the gate context)**: two pieces, -both code-only — neither runs the full-catalog fire, the targeted -re-extraction of issue #340's 373-card cohort, or any other prod -extraction/write, both of which remain separate, owner-gated prod steps. - -1. **`calculate_fallback_verdict`/`run_fallback_calculator`** (own - `anonymous_id="stage-d-fallback-v1"`) — Stage D's own port of - `local_fallback.py`'s pilot "Pass 2" evidence-combination model (that - module's own docstring: fires only when pass 1 yields no accepted vote - for a card), run over exactly the cards `run_join_key_calculator` - already concluded have no confident hit — the SAME population - `run_slow_path_calculator` routes to human review. Unlike the pilot's - own `run_fallback_for_card` (which crops/scans a LIVE image), this - calculator operates entirely off already-persisted `ImageEvidence` - fields: `layout_class`/`artist_ocr_name` feed straight into - `local_fallback.filter_by_border_color`/`match_artist` (PROTECTED CORE, - called not modified), and a new `_filter_by_symbol_phash` compares - `symbol_phash` against each candidate's rendered keyrune glyph - (`local_fallback.render_set_symbol`, PROTECTED CORE) via the same - pure-Hamming-distance-arithmetic reimplementation - `_symbol_phash_tiebreak` already established for the join-key - calculator's own symbol tie-break. A vote is cast ONLY when the - intersection across every sub-check that produced a reading narrows to - EXACTLY ONE candidate — `local_fallback.py`'s own documented rule, - reproduced exactly, with no added agreement/corroboration layer (that - layer doesn't exist in `local_fallback.py`, so this is a faithful port, - not an augmented one). `source=VoteSource.OCR` (not `DEDUCTION`) — the - enum's own docstring in `models.py` explicitly names "the - border/artist/symbol evidence-combination fallback" as part of OCR's - own umbrella definition. Wired into the management command between the - join-key and slow-path calculators (its own gate check via - `verify_zero_resolutions`, same as the join-key calculator's own), and - `_slow_path_eligible_cards_queryset` now also excludes any card this - calculator successfully voted on, so a card it resolves is not also - routed to human review in the same invocation. -2. **Constant #3** (`docs/pipeline-fidelity-gate.md` §3 item 3 / - `docs/reports/2026-07-22-knowledge-inventory.md`'s MISSING item 3): the - shared `_eligible_cards_queryset` helper (both calculators call it) now - also excludes any card already carrying a `VoteSource.DEDUCTION` - printing vote — the pilot never re-voted a card - `deductive_backfill.py`'s own `deductive-backfill-v1` pass had already - voted for (28,112 live production votes). Filtered by - `source=VoteSource.DEDUCTION` rather than the literal `anonymous_id` - value — deliberately generalized so any future deduction-class engine - is excluded the same way. Constants #1 (`RESOLUTION_FLOOR_DPI`) and #2 - (`EXCLUDED_RESOLVED_TAGS`) are explicitly NOT part of this bundle — - their forward-impact sizing is still open per the gate page. +**Pre-fire prep — the fallback channel calculator (built 2026-07-22, +bundled ahead of the full-catalog fire per owner ruling; see +[[../pipeline-fidelity-gate.md]] §3 for the gate context)**: code-only — +does not run the full-catalog fire, the targeted re-extraction of issue +#340's 373-card cohort, or any other prod extraction/write, both of which +remain separate, owner-gated prod steps. + +**`calculate_fallback_verdict`/`run_fallback_calculator`** (own +`anonymous_id="stage-d-fallback-v1"`) — Stage D's own port of +`local_fallback.py`'s pilot "Pass 2" evidence-combination model (that +module's own docstring: fires only when pass 1 yields no accepted vote +for a card), run over exactly the cards `run_join_key_calculator` +already concluded have no confident hit — the SAME population +`run_slow_path_calculator` routes to human review. Unlike the pilot's +own `run_fallback_for_card` (which crops/scans a LIVE image), this +calculator operates entirely off already-persisted `ImageEvidence` +fields: `layout_class`/`artist_ocr_name` feed straight into +`local_fallback.filter_by_border_color`/`match_artist` (PROTECTED CORE, +called not modified), and a new `_filter_by_symbol_phash` compares +`symbol_phash` against each candidate's rendered keyrune glyph +(`local_fallback.render_set_symbol`, PROTECTED CORE) via the same +pure-Hamming-distance-arithmetic reimplementation +`_symbol_phash_tiebreak` already established for the join-key +calculator's own symbol tie-break. A vote is cast ONLY when the +intersection across every sub-check that produced a reading narrows to +EXACTLY ONE candidate — `local_fallback.py`'s own documented rule, +reproduced exactly, with no added agreement/corroboration layer (that +layer doesn't exist in `local_fallback.py`, so this is a faithful port, +not an augmented one). `source=VoteSource.OCR` (not `DEDUCTION`) — the +enum's own docstring in `models.py` explicitly names "the +border/artist/symbol evidence-combination fallback" as part of OCR's +own umbrella definition. Wired into the management command between the +join-key and slow-path calculators (its own gate check via +`verify_zero_resolutions`, same as the join-key calculator's own), and +`_slow_path_eligible_cards_queryset` now also excludes any card this +calculator successfully voted on, so a card it resolves is not also +routed to human review in the same invocation. Idempotence across a +repeated fire comes entirely from `_eligible_cards_queryset`'s own +per-calculator stable-`anonymous_id` exclusion (unchanged, pre-existing). + +**Constant #3 considered and NOT restored (owner ruling, 2026-07-22)**: an +earlier same-day revision of this PR also added a +`.exclude(printing_tags__source=VoteSource.DEDUCTION)` clause to the +shared `_eligible_cards_queryset` helper, addressing `docs/pipeline- fidelity-gate.md` §3 item 3 / `docs/reports/2026-07-22-knowledge- inventory.md`'s former MISSING item 3 (the pilot never re-voted a card +`deductive_backfill.py`'s own `deductive-backfill-v1` pass had already +voted for). A read-only investigation of that 2026-07-14 backfill found +this would be a net-negative single-cohort carve-out, not a worthwhile +restoration: the backfill is pure name/metadata deduction (never +phash/OCR), its votes check out sound (15-card sample all correct), and +excluding those cards would strand ~27,819 sound-but-UNRESOLVED cards +outside Stage D for no protective benefit — re-evaluating them is safe +under the human-backed consensus gate (agreement dedups, disagreement +surfaces to human review). The pilot's own exclusion was a performance +optimization, not a soundness mechanism. Removed before merge; see +`docs/pipeline-fidelity-gate.md` §3 item 3 for the canonical record. +Constants #1 (`RESOLUTION_FLOOR_DPI`) and #2 (`EXCLUDED_RESOLVED_TAGS`) +remain open pending forward-impact sizing, unaffected by this ruling. **Stage E resume contract (owner directive, 2026-07-19 — full spec on task #147, acceptance test folded into task #156's soak gate)**: diff --git a/docs/pipeline-fidelity-gate.md b/docs/pipeline-fidelity-gate.md index d343edc8e..053c5a6f7 100644 --- a/docs/pipeline-fidelity-gate.md +++ b/docs/pipeline-fidelity-gate.md @@ -69,19 +69,29 @@ The knowledge-inventory sweep confirmed three pilot-era constants have live: all 28,112 carry `run_id=None`, `anonymous_id="deductive-backfill-v1"`, `created_at` between 2026-07-14T18:21:49Z and 2026-07-14T18:22:05Z. See `journal/2026-07-14-deductive-printing-tag-backfill.md` (gitignored, - machine-local) for that run's own narrative. **ADDRESSED IN CODE - (2026-07-22, pre-fire prep PR)**: `local_calculate_verdicts._eligible_cards_queryset` - now excludes `source=VoteSource.DEDUCTION` (generalized past the - literal `anonymous_id`, so a future second deduction engine is covered - too), applied to both the join-key AND new fallback calculators — code - only, not yet fired against production. + machine-local) for that run's own narrative. **INTENTIONALLY NOT + RESTORED (owner ruling, 2026-07-22)**, superseding an earlier same-day + revision of this page that marked it "addressed in code": a read-only + investigation of the 2026-07-14 backfill found it is pure name/metadata + deduction (never phash/OCR — zero image inspection) whose votes check + out sound (a 15-card sample all correct), and that excluding those + cards would strand ~27,819 sound-but-UNRESOLVED cards outside Stage D + for no protective benefit — re-evaluating them is safe under the + human-backed consensus gate (agreement dedups, disagreement surfaces to + human review). The pilot's own exclusion was a performance + optimization (skip a card its weaker engines couldn't add to), not a + soundness mechanism, so restoring it here would trade real coverage for + a protection the vote-consensus layer already provides independently. + See `local_calculate_verdicts._eligible_cards_queryset`'s own docstring + for the in-code record of this decision. None of these three are soundness violations — the human-backed consensus gate still applies to every vote Stage D casts regardless. -**Owner ruling still needed on #1/#2** (item #3 above is now addressed in -code, pending the fire): are the remaining two must-fix-before-fire, or an -accepted gap the gate can clear without them? Full detail, plus 3 lower -grade "open items" that are separate from these 3 MISSING findings: +**Owner ruling still needed on #1/#2** (item #3 above is now resolved — +deliberately not restored, per the ruling above): are the remaining two +must-fix-before-fire, or an accepted gap the gate can clear without them? +Full detail, plus 3 lower grade "open items" that are separate from these +3 MISSING findings: [`reports/2026-07-22-knowledge-inventory.md`](reports/2026-07-22-knowledge-inventory.md). ## 4. Open decision (b) — the corrected parity-replay methodology diff --git a/docs/reports/2026-07-22-knowledge-inventory.md b/docs/reports/2026-07-22-knowledge-inventory.md index 7989c5fb8..c981d4509 100644 --- a/docs/reports/2026-07-22-knowledge-inventory.md +++ b/docs/reports/2026-07-22-knowledge-inventory.md @@ -167,7 +167,7 @@ Three items, confirmed absent by direct `grep`/read of `local_calculate_verdicts 2. **`EXCLUDED_RESOLVED_TAGS = ["custom-art", "non-english"]`** (pilot). A card already carrying a resolved custom-art or non-english tag has its printing-identification precondition (an authentic depiction of a real printing) already falsified — the pilot excluded these from selection entirely. Stage D's `_eligible_cards_queryset` in `local_calculate_verdicts.py` has no equivalent `.exclude(tags__contains=...)` clause. Confirmed by `grep` for `custom-art`/`non-english`/`EXCLUDED_RESOLVED_TAGS`/`tags__contains` across all three Stage C/D files: zero hits. Checked for a same-effect-different-mechanism explanation before calling this MISSING (not just absence-of-the-literal-clause): `tag_consensus.py`'s `resolve_and_persist_tag_votes` only ever writes `card.tags`/`card.tag_vote_statuses` — it never touches `printing_tag_status` or `canonical_card`, so a resolved custom-art/non-english tag does NOT get a card excluded through Stage D's `printing_tag_status=UNRESOLVED`/`canonical_card__isnull=True` filters either; there is no other path that produces the pilot's exclusion effect. This does not create a soundness violation (the human-backed consensus gate still applies to any vote cast), but it does mean Stage D would spend real OCR/phash effort — and potentially cast a real `is_no_match` or match vote — on a card whose own tags already say the exercise is moot, which the pilot's own design explicitly avoided. Not previously tracked anywhere found in `docs/features/catalog-completion-plan.md`/`docs/theory.md` — appears to be a genuinely new finding from this sweep, not a known, already-accepted gap. -3. **`.exclude(printing_tags__anonymous_id=DEDUCTIVE_BACKFILL_ANONYMOUS_ID)`** (pilot). Lower severity than the two above, and narrower than it first looks: verified against `printing_consensus.py` directly — a card deductive backfill has actually _resolved_ (`resolve_and_persist_printing` sets `inferred_canonical_card` + `printing_tag_status=RESOLVED`) is already excluded from Stage D's query by its own pre-existing `printing_tag_status=UNRESOLVED` filter, no separate exclusion needed. This gap therefore only bites the narrower "deductive backfill already cast a vote for this card, but it's still sitting UNRESOLVED (pending human confirmation, or the vote alone wasn't sufficient)" subset — the pilot deliberately didn't let its own weaker engines pile a redundant vote onto exactly that subset. Since Stage D's own vote is still gated by the same human-backed consensus rule, this is a redundancy/efficiency gap on that narrower subset, not a soundness one — included for completeness, not alarm. **ADDRESSED IN CODE (2026-07-22, pre-fire prep PR)**: `_eligible_cards_queryset` now excludes `source=VoteSource.DEDUCTION` (deliberately generalized past the literal `anonymous_id`), shared by both the join-key and new fallback calculators — code only, not yet fired against production. +3. **`.exclude(printing_tags__anonymous_id=DEDUCTIVE_BACKFILL_ANONYMOUS_ID)`** (pilot). Lower severity than the two above, and narrower than it first looks: verified against `printing_consensus.py` directly — a card deductive backfill has actually _resolved_ (`resolve_and_persist_printing` sets `inferred_canonical_card` + `printing_tag_status=RESOLVED`) is already excluded from Stage D's query by its own pre-existing `printing_tag_status=UNRESOLVED` filter, no separate exclusion needed. This gap therefore only bites the narrower "deductive backfill already cast a vote for this card, but it's still sitting UNRESOLVED (pending human confirmation, or the vote alone wasn't sufficient)" subset — the pilot deliberately didn't let its own weaker engines pile a redundant vote onto exactly that subset. Since Stage D's own vote is still gated by the same human-backed consensus rule, this is a redundancy/efficiency gap on that narrower subset, not a soundness one — included for completeness, not alarm. **INTENTIONALLY NOT RESTORED (owner ruling, 2026-07-22)** — a same-day read-only investigation of the 2026-07-14 backfill found it is pure name/metadata deduction (never phash/OCR), its votes check out sound (a 15-card sample all correct), and restoring this exclusion would strand ~27,819 sound-but-UNRESOLVED cards outside Stage D for no protective benefit — re-evaluating them is safe under the human-backed consensus gate (agreement dedups, disagreement surfaces to human review). The pilot's own exclusion was a performance optimization, not a soundness mechanism. See `docs/pipeline-fidelity-gate.md` §3 item 3 and `local_calculate_verdicts._eligible_cards_queryset`'s own docstring for the full record. No other pilot-era constant, threshold, crop box, regex, or skip-reason was found with genuinely no current home; every other divergence found a place in one of the non-MISSING buckets above, including several where the current home is the SAME code, literally imported and called (not reimplemented) by the new pipeline.