From 5cbb9759ba7bfba4b3c325b102fece90b880a7ad Mon Sep 17 00:00:00 2001 From: wilfordgrimley <2397930+WilfordGrimley@users.noreply.github.com> Date: Thu, 23 Jul 2026 18:44:32 +0000 Subject: [PATCH] Add parse_legal_line to the OCR-helpers ledger row (PR #384) Co-Authored-By: Claude Sonnet 5 --- docs/upstreaming/extractable-primitives.md | 36 +++++++++++++--------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/upstreaming/extractable-primitives.md b/docs/upstreaming/extractable-primitives.md index 1a0ef9915..ef169adab 100644 --- a/docs/upstreaming/extractable-primitives.md +++ b/docs/upstreaming/extractable-primitives.md @@ -108,20 +108,20 @@ coupling to the vote system is. ## Backend -| Primitive | File(s) | Problem solved | Candidate consumers | Entanglement | License note | -| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | -| Outbound rate limiter ("lh4 rate limiter") | `MPCAutofill/cardpicker/local_phash.py` (`_RateLimiter`, `run_content_phash_backfill`) | Paces a threaded worker pool to a strict `<= N req/sec` ceiling against Google's `lh4.googleusercontent.com` image-resize endpoint | upstream, proxies-at-home (as a copy-paste, not an import — see note) | entangled-with-CanonicalPrinting (colocation) | — | -| Perceptual-hash storage utility | `MPCAutofill/cardpicker/local_phash.py` (`_hash_to_int`, `_int_to_hash`, `compute_card_art_hash`, `find_best_match`) | Encodes/decodes an `imagehash.ImageHash` as a signed 64-bit DB int; threshold+margin best-match selection | upstream, proxies-at-home | entangled-with-CanonicalPrinting (colocation) | — | -| Search-query sanitisation | `MPCAutofill/cardpicker/search/sanitisation.py` | Normalizes free-text queries/names (lowercase, strip bracketed text/punctuation/digits, collapse whitespace) for consistent matching | upstream, proxies-at-home | CLEAN | — | -| Scryfall-style search-operator parser | `MPCAutofill/cardpicker/search/operator_parser.py` (`parse_query`) | Parses a raw query string into residual free text + structured `operator:value`/`-operator:value` tokens (quoted values, case-insensitive operator names, unknown-operator errors) - pure string-in/structure-out, no knowledge of what an operator name maps to downstream | upstream, proxies-at-home | CLEAN | — | -| OCR crop/preprocessing helpers | `MPCAutofill/cardpicker/local_ocr.py` (`crop_collector_line`, `preprocess_variants`, `preprocess_fallback_variants`, `run_tesseract`, `run_tesseract_text_and_words`, `parse_collector_line`, `_normalize_collector_number`, `_median_from_histogram`) | Fractional-bbox crop, grayscale/upscale/threshold-both-polarities preprocessing (plus a heavier-upscale/sharpen + percentile-threshold fallback tier, issue #259) and regex parse of an OCR'd collector-number line | upstream, proxies-at-home | CLEAN | — | -| Image color/quality-signal math | `MPCAutofill/cardpicker/local_image_quality.py` (`is_image_truncated`, `compute_blur_variance`, `compute_entropy`, `compute_color_profile`) | Truncation check, Laplacian-kernel blur variance, grayscale entropy, and per-channel RGB mean/stddev, all pure `PIL.ImageStat`/`ImageFilter` calls against an already-fetched image | upstream, proxies-at-home, federation peers | CLEAN (zero `cardpicker.*` imports at all — only `PIL`) | — | -| Bleed/border geometry helpers | `MPCAutofill/cardpicker/local_fallback.py` (`normalize_crop_box`, `classify_bleed_edge`) | Pure crop-box remapping (bleed vs. trim) and aspect-ratio-based border classification | upstream, proxies-at-home (as a copy-paste, not an import — see note) | entangled-with-vote-consensus (colocation) | — | -| Generic backend utilities | `MPCAutofill/cardpicker/utils.py` (`get_json_endpoint_rate_limited`, `twos_complement`, `section_timer`, `time_to_hours_minutes_seconds`, `log_hours_minutes_seconds_elapsed`) | Rate-limited JSON GET wrapper, signed-int bit-twiddling, timing decorator/formatter | upstream, proxies-at-home | CLEAN | — | -| Batch-flush checkpoint pattern | `MPCAutofill/cardpicker/local_phash.py` (`run_content_phash_backfill`), `deductive_backfill.py` (`run_backfill`), `local_identify_printing_tags.py` (`run_pilot`) | Sliding-window worker pool + periodic bulk-flush + NULL-filter-as-checkpoint for resumable backfill jobs | upstream, proxies-at-home (needs generalizing first — see note) | entangled — no clean instance exists yet | — | -| Elasticsearch connection helpers | `MPCAutofill/cardpicker/search/search_functions.py` (`get_elasticsearch_connection`, `ping_elasticsearch`, `elastic_connection`, `SearchExceptions`) | Thread-local ES client + a decorator translating raw ES connection errors into app exceptions | upstream, proxies-at-home | entangled-with-consensus (colocation) | — | -| Back-face name lookup (issue #199) | `MPCAutofill/cardpicker/printing_metadata_import.py` (`get_back_face_names`, `is_back_face`, `DOUBLE_FACED_LAYOUTS`) | Deterministic name → "is this a known DFC back face" lookup from Scryfall's on-disk `card_faces` bulk data, no network fetch | upstream, proxies-at-home | entangled-with-CanonicalPrinting (colocation) | — | -| Self-recording, forced-dry-run-gated command lifecycle (issue #362) | `MPCAutofill/cardpicker/pilot_run_lifecycle.py` (`resilient_terminal_output`, `enforce_dry_run_precondition`, `add_dry_run_guard_arguments`, `scope_hash`, `initial_counters`, `merge_counters`) | Generic pattern for a long-running write management command: a RUNNING→COMPLETED/FAILED audit-row lifecycle with JSON counters, a broken-pipe-safe terminal-output wrapper, and a forced-dry-run precondition gate refusing `--write`/`--apply` without a matching recent dry-run | upstream, proxies-at-home (any Django project with long-running write management commands) | entangled-with-vote-consensus (colocation) - the one model this file depends on, `PilotRunLedger`, lives in `cardpicker/models.py` alongside the vote system, even though this file itself imports nothing from `vote_consensus`/`printing_consensus`/`tag_consensus`/`artist_consensus`/auth directly | — | +| Primitive | File(s) | Problem solved | Candidate consumers | Entanglement | License note | +| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | +| Outbound rate limiter ("lh4 rate limiter") | `MPCAutofill/cardpicker/local_phash.py` (`_RateLimiter`, `run_content_phash_backfill`) | Paces a threaded worker pool to a strict `<= N req/sec` ceiling against Google's `lh4.googleusercontent.com` image-resize endpoint | upstream, proxies-at-home (as a copy-paste, not an import — see note) | entangled-with-CanonicalPrinting (colocation) | — | +| Perceptual-hash storage utility | `MPCAutofill/cardpicker/local_phash.py` (`_hash_to_int`, `_int_to_hash`, `compute_card_art_hash`, `find_best_match`) | Encodes/decodes an `imagehash.ImageHash` as a signed 64-bit DB int; threshold+margin best-match selection | upstream, proxies-at-home | entangled-with-CanonicalPrinting (colocation) | — | +| Search-query sanitisation | `MPCAutofill/cardpicker/search/sanitisation.py` | Normalizes free-text queries/names (lowercase, strip bracketed text/punctuation/digits, collapse whitespace) for consistent matching | upstream, proxies-at-home | CLEAN | — | +| Scryfall-style search-operator parser | `MPCAutofill/cardpicker/search/operator_parser.py` (`parse_query`) | Parses a raw query string into residual free text + structured `operator:value`/`-operator:value` tokens (quoted values, case-insensitive operator names, unknown-operator errors) - pure string-in/structure-out, no knowledge of what an operator name maps to downstream | upstream, proxies-at-home | CLEAN | — | +| OCR crop/preprocessing helpers | `MPCAutofill/cardpicker/local_ocr.py` (`crop_collector_line`, `preprocess_variants`, `preprocess_fallback_variants`, `run_tesseract`, `run_tesseract_text_and_words`, `parse_collector_line`, `parse_legal_line`, `_normalize_collector_number`, `_median_from_histogram`) | Fractional-bbox crop, grayscale/upscale/threshold-both-polarities preprocessing (plus a heavier-upscale/sharpen + percentile-threshold fallback tier, issue #259), regex parse of an OCR'd collector-number line, and regex-based "not for sale"/proxy-marker/playtest/copyright-year detection over an OCR'd legal line (2026-07-23, PR #384's marker-detection expansion - widened to catch maker-brand-glued forms like "JestaProxy") | upstream, proxies-at-home | CLEAN | — | +| Image color/quality-signal math | `MPCAutofill/cardpicker/local_image_quality.py` (`is_image_truncated`, `compute_blur_variance`, `compute_entropy`, `compute_color_profile`) | Truncation check, Laplacian-kernel blur variance, grayscale entropy, and per-channel RGB mean/stddev, all pure `PIL.ImageStat`/`ImageFilter` calls against an already-fetched image | upstream, proxies-at-home, federation peers | CLEAN (zero `cardpicker.*` imports at all — only `PIL`) | — | +| Bleed/border geometry helpers | `MPCAutofill/cardpicker/local_fallback.py` (`normalize_crop_box`, `classify_bleed_edge`) | Pure crop-box remapping (bleed vs. trim) and aspect-ratio-based border classification | upstream, proxies-at-home (as a copy-paste, not an import — see note) | entangled-with-vote-consensus (colocation) | — | +| Generic backend utilities | `MPCAutofill/cardpicker/utils.py` (`get_json_endpoint_rate_limited`, `twos_complement`, `section_timer`, `time_to_hours_minutes_seconds`, `log_hours_minutes_seconds_elapsed`) | Rate-limited JSON GET wrapper, signed-int bit-twiddling, timing decorator/formatter | upstream, proxies-at-home | CLEAN | — | +| Batch-flush checkpoint pattern | `MPCAutofill/cardpicker/local_phash.py` (`run_content_phash_backfill`), `deductive_backfill.py` (`run_backfill`), `local_identify_printing_tags.py` (`run_pilot`) | Sliding-window worker pool + periodic bulk-flush + NULL-filter-as-checkpoint for resumable backfill jobs | upstream, proxies-at-home (needs generalizing first — see note) | entangled — no clean instance exists yet | — | +| Elasticsearch connection helpers | `MPCAutofill/cardpicker/search/search_functions.py` (`get_elasticsearch_connection`, `ping_elasticsearch`, `elastic_connection`, `SearchExceptions`) | Thread-local ES client + a decorator translating raw ES connection errors into app exceptions | upstream, proxies-at-home | entangled-with-consensus (colocation) | — | +| Back-face name lookup (issue #199) | `MPCAutofill/cardpicker/printing_metadata_import.py` (`get_back_face_names`, `is_back_face`, `DOUBLE_FACED_LAYOUTS`) | Deterministic name → "is this a known DFC back face" lookup from Scryfall's on-disk `card_faces` bulk data, no network fetch | upstream, proxies-at-home | entangled-with-CanonicalPrinting (colocation) | — | +| Self-recording, forced-dry-run-gated command lifecycle (issue #362) | `MPCAutofill/cardpicker/pilot_run_lifecycle.py` (`resilient_terminal_output`, `enforce_dry_run_precondition`, `add_dry_run_guard_arguments`, `scope_hash`, `initial_counters`, `merge_counters`) | Generic pattern for a long-running write management command: a RUNNING→COMPLETED/FAILED audit-row lifecycle with JSON counters, a broken-pipe-safe terminal-output wrapper, and a forced-dry-run precondition gate refusing `--write`/`--apply` without a matching recent dry-run | upstream, proxies-at-home (any Django project with long-running write management commands) | entangled-with-vote-consensus (colocation) - the one model this file depends on, `PilotRunLedger`, lives in `cardpicker/models.py` alongside the vote system, even though this file itself imports nothing from `vote_consensus`/`printing_consensus`/`tag_consensus`/`artist_consensus`/auth directly | — | ## Docs tooling & federation @@ -198,7 +198,13 @@ filter `validate_against_candidates` already computed internally, extracted so a caller with independent tie-break evidence can inspect the ambiguous match set directly) is excluded for the identical reason — it's the same printing-identification matching logic, just under a new name, not a -separate primitive. +separate primitive. `parse_legal_line` (present since before this ledger's +2026-07-19 sweep, widened 2026-07-23 by PR #384) is the opposite case and +IS included: it takes raw OCR text and returns a copyright year plus a +proxy/not-for-sale/playtest marker boolean via `_PROXY_MARKER_RE` — a +generic "is this legal line suspicious" text classifier with no reference +to any candidate printing, canonical card, or consensus state, unlike +`validate_against_candidates`/`find_matching_candidates` above. **Federation hash tool** — the _code artifact_ has zero fork dependencies (confirmed: its only two mentions of `cardpicker` are provenance comments,