Skip to content

Deferred: cross-source phash reference corpus backfill (PR #694) — supplanted by #508's self-referential exemplar index #697

Description

@WilfordGrimley

Status: deferred (PR #694 closed unmerged)

PR #694 implemented a resumable, batched backfill for CanonicalCard.image_hash
— the cross-source phash reference corpus, whose values are computed by
fetching Scryfall's art_crop image for each printing and hashing it. 86,073 of
113,224 printings are populated; the PR would have completed the remaining
27,151.

It is deferred rather than merged, by owner decision (2026-08-05), for reasons
that were already recorded in #508 on 2026-07-28 and which this work
inadvertently contradicted:

Exemplars come from our own DB, never Scryfall images. […] phash
comparability requires identical crop geometry/preprocessing — our artbox
extractor is self-consistent, while Scryfall's art_crop framing differs and
would make cross-source Hamming distances unreliable. Also keeps
index-not-store intact (hashes only, no fetched pixels).

Two additional facts established while reviewing #694:

  1. It is not a local operation. _fetch_and_hash calls
    rate_limited_get(SCRYFALL_CDN, …) and downloads the image bytes. The
    2026-07-19 "local-first" change removed the per-printing REST metadata call
    (the 93.6%-of-wall-clock cost), not the image fetch. The limiter's own comment
    says so: it governs "the CDN image fetch itself (still needed for every hash,
    cached art-crop URL or not)". Completing the corpus means 27,151 image
    downloads
    at rate_per_sec=10.0, max_concurrency=5. Beyond our own
    bandwidth, this is not a pattern to ship to instance operators.

  2. The loose thresholds are the cross-source penalty.
    find_best_match uses distance=20, margin=5, and its own comment explains
    why: sampled true matches ranged 14-22, so "a threshold of 10 would reject
    every single real candidate." Artbox-phash illustration matching: resolution-seeded exemplar index (second deduction path) #508's self-referential discipline is d=0 for
    identity and 0<d≤2 to narrow, an order of magnitude tighter — because it
    compares like with like.

What is NOT resolved by deferring this

identify_land_printing and recover_frame_mismatch_printing_via_phash already
consume CanonicalCard.image_hash and abstain with no-hashable-candidates when
a candidate printing has none. With 24% of printings unhashed, those two paths
abstain today for reasons unrelated to the card being scanned. That cost is real
but bounded and unmeasured; sizing it is the prerequisite to revisiting this.

Revisit conditions

The code is preserved on branch feat/local-backfill-canonical-hash — batching,
resumability, idempotence, --dry-run and its tests are all sound work and
directly reusable for the self-referential corpus, which is the follow-up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions