perf(image-evidence): collapse Stage C OCR attempt ladder from 3 tiers to 2 (issue #677) - #698
Merged
WilfordGrimley merged 3 commits intoAug 5, 2026
Conversation
…s to 2 (issue #677) _collector_line_ocr_attempts drops tier 3 (a PSM-11 re-try of tier 1's own variants) - measured, not guessed. Two fresh real-production probes (450 forced-escalation cards combined) walked the old 3-tier ladder to full completion and recorded, per card, the first tier at which a candidate-validated genuine match appeared: tier 2 produced 2 genuine matches across the sample, tier 3 produced zero - only more lexicon-valid -but-uncorroborated noise, consistent with the 2026-07-23 preprocessing probes' own ~99% hopeless-art-noise finding for this population. Cross-checked against all 30 golden_set.GOLDEN_CARD_IDS cards: none ever resolved uniquely at tier 3, so this collapse changes zero golden-set expectations. Worst case drops from 8 tesseract calls to 6 (down from #259-#677's own 4x pre-#259 cost to 3x). collector_line_ocr/collector_line_tsv/artist_ocr extractor versions bumped v2->v3 (all three read through the same escalation loop, per EXTRACTOR_OWNERSHIP's own 'bump every listed key together' convention); legal_line unchanged. Investigated and explicitly did NOT ship the companion 'unify the collector-line/legal-line OCR reads into one full-width pass' idea named in this issue's own brief: a real-image accuracy probe found it regresses 46/150 (30.7%) of currently-correct collector-number reads (mostly proxy/ watermark text getting picked up by the set-code regex in place of the real code, or the wider crop causing tesseract to drop the collector- number line outright), with zero improvements. Reported as a legitimate negative finding rather than shipped anyway. Lifts the EXTRACTOR_OWNERSHIP EXCLUDED_HELPERS entry for the OCR ladder (check_extractor_ownership_totality.py) that PR #690 left in place while this branch was in flight - _collector_line_ocr_attempts and preprocess_fallback_variants now carry real ownership entries. Verification: full backend suite (3598 passed, 8 skipped, 0 failed), ownership-totality + manifest-sync scripts clean with their own unit tests green (40/40), docs_lint --strict clean, pre-commit clean on all tracked files.
…'s v3 ARTIST_OCR_EXTRACTOR_VERSION was set to "artist-ocr-v3" on this branch for issue #677's ladder-collapse dedup, but PR #685 (merged to master separately) already claims "artist-ocr-v3" for an unrelated change (wiring modern_artist_credit's crop fallback into live extraction). Two different behaviours under one version string would defeat the staleness filter MANIFEST_EXTRACTOR_CURRENT_VERSIONS exists to enforce. Bumps artist_ocr straight to v4, updates the matching manifest entry in run_image_evidence_cohort.py, and documents the collision in both the version constant's own history comment and catalog-completion-plan.md's #677 section. collector_line_ocr/collector_line_tsv stay at v3 (master still has both at v2, no collision). Also fixes golden_set.py's #677 re-verification comment, which cited MPCAutofill/scripts/experiments/ocr_ladder_tier_attribution.py - a worktree-only analysis script never committed to this repo, per the established convention (see docs/reports/2026-07-23-ocr-preprocessing- probe.md's own LIVE STATE) of not committing one-off probes. Points at catalog-completion-plan.md's already-committed #677 section instead, which documents the same 450-card probe's methodology and results.
Both scripts are deliberately uncommitted (worktree-only analysis tooling, same convention as the 2026-07-23 probe reports). Backticked citations to their paths tripped docs_lint.py's path-reference rule since MPCAutofill/scripts/ doesn't exist in the repo. Rephrase as prose so the provenance stays legible without a machine-checkable path reference to a file that was never meant to be committed.
WilfordGrimley
merged commit Aug 5, 2026
c5eee1c
into
feat/extractor-ownership-visibility
14 checks passed
8 tasks
WilfordGrimley
added a commit
that referenced
this pull request
Aug 5, 2026
…s to 2 (issue #677) (#698) (#702) * perf(image-evidence): collapse Stage C OCR attempt ladder from 3 tiers to 2 (issue #677) _collector_line_ocr_attempts drops tier 3 (a PSM-11 re-try of tier 1's own variants) - measured, not guessed. Two fresh real-production probes (450 forced-escalation cards combined) walked the old 3-tier ladder to full completion and recorded, per card, the first tier at which a candidate-validated genuine match appeared: tier 2 produced 2 genuine matches across the sample, tier 3 produced zero - only more lexicon-valid -but-uncorroborated noise, consistent with the 2026-07-23 preprocessing probes' own ~99% hopeless-art-noise finding for this population. Cross-checked against all 30 golden_set.GOLDEN_CARD_IDS cards: none ever resolved uniquely at tier 3, so this collapse changes zero golden-set expectations. Worst case drops from 8 tesseract calls to 6 (down from #259-#677's own 4x pre-#259 cost to 3x). collector_line_ocr/collector_line_tsv/artist_ocr extractor versions bumped v2->v3 (all three read through the same escalation loop, per EXTRACTOR_OWNERSHIP's own 'bump every listed key together' convention); legal_line unchanged. Investigated and explicitly did NOT ship the companion 'unify the collector-line/legal-line OCR reads into one full-width pass' idea named in this issue's own brief: a real-image accuracy probe found it regresses 46/150 (30.7%) of currently-correct collector-number reads (mostly proxy/ watermark text getting picked up by the set-code regex in place of the real code, or the wider crop causing tesseract to drop the collector- number line outright), with zero improvements. Reported as a legitimate negative finding rather than shipped anyway. Lifts the EXTRACTOR_OWNERSHIP EXCLUDED_HELPERS entry for the OCR ladder (check_extractor_ownership_totality.py) that PR #690 left in place while this branch was in flight - _collector_line_ocr_attempts and preprocess_fallback_variants now carry real ownership entries. Verification: full backend suite (3598 passed, 8 skipped, 0 failed), ownership-totality + manifest-sync scripts clean with their own unit tests green (40/40), docs_lint --strict clean, pre-commit clean on all tracked files. * fix(image-evidence): bump artist_ocr to v4 to avoid colliding with #685's v3 ARTIST_OCR_EXTRACTOR_VERSION was set to "artist-ocr-v3" on this branch for issue #677's ladder-collapse dedup, but PR #685 (merged to master separately) already claims "artist-ocr-v3" for an unrelated change (wiring modern_artist_credit's crop fallback into live extraction). Two different behaviours under one version string would defeat the staleness filter MANIFEST_EXTRACTOR_CURRENT_VERSIONS exists to enforce. Bumps artist_ocr straight to v4, updates the matching manifest entry in run_image_evidence_cohort.py, and documents the collision in both the version constant's own history comment and catalog-completion-plan.md's still has both at v2, no collision). Also fixes golden_set.py's #677 re-verification comment, which cited MPCAutofill/scripts/experiments/ocr_ladder_tier_attribution.py - a worktree-only analysis script never committed to this repo, per the established convention (see docs/reports/2026-07-23-ocr-preprocessing- probe.md's own LIVE STATE) of not committing one-off probes. Points at catalog-completion-plan.md's already-committed #677 section instead, which documents the same 450-card probe's methodology and results. * fix(docs): cite OCR ladder probe scripts as prose, not backticked paths Both scripts are deliberately uncommitted (worktree-only analysis tooling, same convention as the 2026-07-23 probe reports). Backticked citations to their paths tripped docs_lint.py's path-reference rule since MPCAutofill/scripts/ doesn't exist in the repo. Rephrase as prose so the provenance stays legible without a machine-checkable path reference to a file that was never meant to be committed. (cherry picked from commit c5eee1c)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_collector_line_ocr_attempts(issue Stage D no-text bucket: OCR preprocessing/crop recovery (88.8% of skips are fixable, not a coverage ceiling) #259's Stage C OCR attempt ladder) from 3 tiers/8 tesseract calls to 2 tiers/6 calls: tier 3 (a re-try of tier 1's own variants underALTERNATE_TESSERACT_CONFIG/PSM 11) is removed. Two fresh real-production probes (worktree-only analysis scripts, not committed to the repo — consistent with how this repo already cites other uncommitted probe tooling) walked the old 3-tier ladder to full completion over 450 forced-escalation cards (300 currently-blank + 150 currently-resolved, both freshly re-derived live 2026-08-05) and recorded, per card, the first tier at which a candidate-validated genuine match appeared (local_ocr.validate_against_candidatesagainst the card's own real name-scoped candidates, not merely a lexicon-shaped parse). Tier 2 produced 2 genuine matches across the 450-card sample; tier 3 produced zero — only more lexicon-valid-but-uncorroborated noise (6/300 in the blank pool).golden_set.GOLDEN_CARD_IDScards the same way: none of the 30 ever resolved uniquely at tier 3 (genuine match, lexicon-valid parse, or "best invalid" fallback) — this collapse changes zero golden-set expectations.COLLECTOR_LINE_OCR_EXTRACTOR_VERSION/COLLECTOR_LINE_TSV_EXTRACTOR_VERSION/ARTIST_OCR_EXTRACTOR_VERSIONbumped v2→v3 (all three read through the same escalation loop).ARTIST_OCR_EXTRACTOR_VERSIONjumps to v4 instead: fix(image_evidence): wire the artist-crop fallback into live extraction #685 (merged separately, unrelatedmodern_artist_creditfallback wiring) already claimed v3 for that extractor, and stamping two different behaviours under one version string would defeatMANIFEST_EXTRACTOR_CURRENT_VERSIONS's staleness filter.LEGAL_LINE_EXTRACTOR_VERSIONis unchanged.collector_line_ocr's narrow crop (DEFAULT_CROP_BOX) andlegal_line's full-width crop (LEGAL_LINE_CROP_BOX, same y-band) into one OCR pass. A real-image accuracy probe on 150 currently-successful production cards (same worktree-only, uncommitted convention as above) found that re-reading tier 1 from the wider crop with the same preprocessing regresses 46/150 (30.7%) of currently-correctcollector_line_set_code/collector_line_collector_numberreads — mostly proxy/watermark text and artist full names getting picked up by the "search after the number" set-code fallback in place of the real code, and in some cases the wider image causing tesseract to drop the collector-number line entirely. Zero improvements found (0/150).EXTRACTOR_OWNERSHIPEXCLUDED_HELPERSentry for the OCR ladder that feat(image-evidence): make evidence-affecting helpers version-visible + CI totality #690 left in place while this branch was in flight —_collector_line_ocr_attemptsandpreprocess_fallback_variantsnow carry real ownership entries ({"collector_line_ocr", "collector_line_tsv", "artist_ocr"}).docs/features/catalog-completion-plan.md: replaces the now-stale "OCR ladder exclusion" paragraph feat(image-evidence): make evidence-affecting helpers version-visible + CI totality #690 wrote with a dated entry carrying the full measured evidence above, including the crop-unification negative finding.golden_set.py'scollector_line_ocrcomment gets a matching re-verification note (2026-08-05); noGOLDEN_EXPECTATIONSvalue changed.Stacking
Cut from, and based on,
feat/extractor-ownership-visibility(#690) — this PR is stacked on top of it and should be reviewed/merged in that order. The diff shown is scoped to this PR's own changes.Verification
cardpicker/tests/test_image_evidence.py(111 tests, 7 updated for the new 6-attempt/2-tier ladder shape), together withtest_golden_set.py+test_run_image_evidence_cohort.py(207 tests combined) — green.cardpicker/tests/test_local_ocr.py+test_ocr_engine_ab.py(65 tests, untouched files) — green..github/scripts/tests/test_check_extractor_ownership_totality.py+test_check_extractor_manifest_sync.py(40 tests, 4 updated for the lifted exclusion + 3 new) — green.python3 .github/scripts/docs_lint.py— clean. This caught a real finding after the initial push: two backticked citations to the two probe scripts above (catalog-completion-plan.mdlines 1921/1945) resolved to paths that don't exist in the repo, since those scripts are deliberately uncommitted. Fixed by rephrasing both citations as prose rather than backticked paths, matching how this repo already cites its other uncommitted probe scripts (e.g.docs/reports/2026-07-23-ocr-preprocessing-probe.md) — the measured evidence itself is unchanged, only the machine-checkable path reference is gone.