feat(local_phash): local-only resumable backfill for CanonicalCard.image_hash - #694
feat(local_phash): local-only resumable backfill for CanonicalCard.image_hash#694WilfordGrimley wants to merge 1 commit into
Conversation
…age_hash Closes the reference-corpus gap (27,151/113,224 canonical printings still at the unset image_hash=0 sentinel, confirmed via a read-only dry-run count against production 2026-08-05). CanonicalPrintingMetadata.art_crop_url is populated on 113,224/113,224 printings (issue #339), so the entire backlog is computable with zero Scryfall REST calls - only the image CDN fetch that produces the hash itself. - run_canonical_hash_backfill (local_phash.py): same pipelined, checkpoint-per-batch shape as run_content_phash_backfill. Selects on image_hash=0 (idempotent/resumable, no separate --resume flag). Local-only by default - a printing with no local art_crop_url is counted in skipped_no_local_url and left alone, not silently routed to a live REST call. --allow-remote is an explicit opt-in escape hatch, default off. No separate rate-limit parameter: _fetch_and_hash already goes through harvest_fetch_limiter's SCRYFALL_CDN destination limiter. - local_backfill_canonical_hash management command: --dry-run (default off - explicit flag required to avoid writing), --batch-size, --workers (default 5, matching SCRYFALL_CDN.max_concurrency), --queue-depth-batches, --limit, --allow-remote, --nice. Dry-run reports selected/backlog, hashed/skipped/failed counts, and an elapsed-rate-based wall-clock estimate for the full backlog. - Tests mirror TestContentPhashBackfill/TestBackfillCommandCLI/ TestPipelinedBackfillOutOfOrder from test_local_identify_printing_tags.py, adapted for CanonicalCard + the missing-local-url skip path. Does not run the backfill or touch find_best_match's matching behaviour - populates the corpus, doesn't consume it.
|
Closing unmerged and deferring to #697, per owner decision 2026-08-05. This builds the cross-source corpus (Scryfall Also corrected in the process: this backfill is not a local operation. The branch is preserved. The batching, resumability, idempotence, |
…ndex (issue #508 phase 1) (#701) Builds ArtboxPhashExemplar, the second illustration-deduction path's reference index, and its read-only backfill/retraction commands. Reuses the batched/resumable/idempotent shape from PR #694 (feat/local-backfill-canonical-hash) - only the source changes, from Scryfall CDN fetches to a scan of our own ImageEvidence; this backfill needs none of that branch's threaded fetch pipeline since every input is already a local DB read. Seeds from two sources, per the owner's 2026-08-05 extension of #508's original human-only design: human-backed printing resolutions (printing_tag_status == RESOLVED, always human-backed by the consensus gate) and unopposed join-key machine votes at 0.75/0.85 confidence. The artist-disagreement tier (0.65) and the no-match tier (0.6) are both excluded from seeding. Every exemplar records its own seed kind, confidence, and a seed_group_key that lets a bad seed retract together with everything it seeded in one filtered delete. Never fetches Scryfall images - phash comparability requires the same crop geometry our own extractor guarantees. Phase 1 scope only: no matching calculator, no vote, no change to consensus or the human-backed resolution gate. Documents the mechanism in docs/identification-pipeline.md's Parallel detectors section, stating explicitly what it does not do.
Summary
CanonicalCard.image_hashis the reference corpus every phash comparisonscores against (
local_phash.find_best_matchviaidentify_land_printing/recover_frame_mismatch_printing_via_phash, andany future illustration matching). It's populated lazily by
get_or_compute_canonical_hash, which caches forever on the premise aprinting's art crop never changes.
Measured against production (2026-08-05, read-only count, confirmed below):
27,151/113,224 canonical printings are still at the unset
image_hash=0sentinel.
CanonicalPrintingMetadata.art_crop_url(issue #339's local-firstfix) is populated on all 113,224/113,224 printings, so this backlog is
computable with zero Scryfall REST calls - only the image-CDN fetch that
produces the hash itself, which the same measurement below confirms fires
for all 27,151 rows (
skipped_no_local_urlwould be 0).This PR adds
run_canonical_hash_backfill(local_phash.py) and itsmanagement command,
local_backfill_canonical_hash, following the existingrun_content_phash_backfill/local_backfill_content_phashsibling's shape:image_hash=0, noseparate
--resumeflag - a plain re-invocation after a kill just picksup what's left.
submission window (
batch_size * queue_depth_batches), checkpoint-flushper batch as fetches complete - identical shape/out-of-order-completion
safety argument as the content-phash sibling.
art_crop_urliscounted in
skipped_no_local_urland left alone, never silently routedto a live REST call.
--allow-remoteis an explicit opt-in escape hatchfor deliberately closing a residual gap later (default off).
--rate-limit-per-sec,_fetch_and_hashalready goes throughharvest_fetch_limiter.rate_limited_get(SCRYFALL_CDN, ...), which pacesand bounds concurrency (10 req/s, max_concurrency=5) at the destination-
limiter layer - a second uncoordinated pacer on top would only add a
redundant ceiling.
--workersdefaults to 5, matchingSCRYFALL_CDN.max_concurrency(loaddiscipline per the brief - a wider pool just queues behind that
destination's own semaphore, it doesn't raise real throughput or host
load).
--dry-run(off by default - must be passed explicitly) reports theselected/full-backlog counts, hashed/skipped/failed counts, and an
elapsed-rate-based wall-clock estimate for the full backlog.
Does not touch
find_best_match, its 20/5 distance/margin calibration, orany matching behaviour (issue #419's scope) - this populates the corpus, it
does not consume it. Does not run the backfill itself - writing 27,151 rows
is an owner-authorised operation, out of scope for this PR.
Deviations from the brief
was verified exactly via a read-only production count (see Verification).
--allow-remotewas added (the brief called it a judgement call,default off either way): kept because it's a near-zero-cost wire-through
of the REST fallback
get_or_compute_canonical_hashalready has, and themeasurement below shows the local URL currently covers 100% of the
backlog - so the flag is a deliberate, rarely-needed escape hatch rather
than a routine one.
PilotRunLedgerintegration (unlikebackfill_md5_checksums'sforced-dry-run-guard family): followed the
local_backfill_content_phashsibling's simpler shape instead, since this command lives in the same
module and the brief pointed at that command as a convention model for
"CLI shape,
--dry-run, and progress reporting."Test plan
pytest cardpicker/tests/test_local_phash.py- 33 passed(individually)
pytest cardpicker/tests/test_local_phash.py cardpicker/tests/test_local_identify_printing_tags.py- 252 passed(together, per issue Test suite is order-dependent: leaked fetch-failure window trips the envelope across files (8 failures on master) #679's order-dependence concern)
(
TestCanonicalHashBackfill/TestCanonicalHashBackfillCommandCLI/TestCanonicalHashBackfillPipelineOutOfOrder) - all pass in isolationpre-commit runon all three touched files (ruff, isort, black,mypy, prettier) - all green
--dry-runselection-count check executed directly againstthe live production DB (
docker exec mpcautofill_django python manage.py shell, ORM.count()only, no writes, no mutating command run):CanonicalCard.objects.filter(image_hash=0).count()= 27,151, exactlymatching the brief's figure. Also confirmed
.filter(image_hash=0, printing_metadata__art_crop_url='').count()= 0and
.filter(image_hash=0, printing_metadata__isnull=True).count()= 0 -every backlog row has a local art-crop URL, so
skipped_no_local_urlwill be 0 on the real run and
--allow-remotecurrently has nothing todo.
lsp_diagnostics- attempted once, timed out (known best-effortlimitation on this box, per PR fix(image_evidence): wire the artist-crop fallback into live extraction #685's precedent); not retried - mypy +
pytest substitute.
Task-end checks (CLAUDE.md)
docs/file documents this backfill's internalsbeyond the code's own docstrings (matching the content-phash sibling,
which also has none); no blocker exceeded 15 minutes, so no
docs/troubleshooting.mdentry needed.not user- or admin-facing product behaviour.
independent of the vote/consensus system was created or removed.
(
DEFAULT_CANONICAL_HASH_BACKFILL_BATCH_SIZE/_WORKERS) are newly added,nothing existing was renamed/extracted/retired.
(
feat/local-backfill-canonical-hash, cut fresh fromorigin/master)and this PR opened per this task's explicit deliverable instructions, not
the solo-work default. Not merging - merges are owner-only, and running
the backfill itself is explicitly out of scope for this PR.