Hash-at-ingest + two-threshold clustering: dedup as a free DB read

[WilfordGrimley]

Summary

Coherent follow-on to the local printing-ID pilot's research findings (docs/features/printing-tags.md's prior-art read and phash-at-small-CDN-size sections): hash once at ingest, store, never recompute corpus-wide. Absorbs deferred item 2b. Built while the fast-follow-enabled full-catalog run (PR #26, merged) continues unattended — this PR does not touch that run, it's the architecture for the next one. Held as draft per instruction.

• Schema: Card.image_hash turned out to be a dead field (migration 0046 — always written 0, never read anywhere). Renamed/repurposed to content_phash (nullable, indexed) instead of adding a confusing second hash column. Doubles as federation-v1's reserved content_hash.
• Hash at ingest: update_database.hash_newly_created_cards — a real, threaded, best-effort fetch+hash for newly-created cards at a small CDN size. Correction to the brief's premise: update_database never touched image bytes before this, so this is genuine new cost, not a free byproduct. Scoped to CREATED cards only (content_phash was never in bulk_update's field list, so there's nothing to persist for updated cards anyway) — flagged as a deliberate narrowing of "new/changed cards."
• Backfill command: local_backfill_content_phash — idempotent via a NULL filter (no separate --resume state), batched, threaded.
• Two-threshold clustering (cardpicker/local_clustering.py): replaces the disabled fetch-based pre-pass entirely. d=0 propagates votes (unchanged semantics, restores the representative-filtering the disabled no-op had dropped). d<=2 is a narrowing prior — computed and tested, not yet wired into _compute_card's live candidate matching (flagged explicitly as a scoped-out fast-follow, not silently half-built).
• Benchmarked before trusting it: d=0 grouping is 0.13s at N=166,422; the d<=2 vectorized scan is ~2-3 min at the same N (vs. the disabled pre-pass's ~21.6h) — computed as two independent steps so a bug/slowdown in the newer tier can never take down the proven d=0 propagation.
• Validated against real production data: harvested 300 same-printing / 300 different-printing card pairs from the live full-catalog run via a read-only query (not the earlier n=2 sample). Ground-truth correction made before trusting the numbers: "voted for the same printing" ≠ "same uploaded image" — partitioned by full-resolution distance-0 to find 79 real true duplicates. Result: 0 false splits among those 79 (100% land at d<=2), 0 false merges among the 269 resolvable different-printing pairs (min distance 6).
• mypy caught a real gap: the new import chain (update_database → local_phash → imagehash/pytesseract) wasn't in the mypy pre-commit hook's isolated dependency list — fixed in .pre-commit-config.yaml.

Projected next-run wall-clock: ~0.52-0.54 days (12.6-12.9h), vs. this run's own ~0.73 days, once a one-time ~2.8h backfill completes.

Out of scope (logged, not built): art-region second hash, multi-hash ensembles, deep-embedding dedup, hash_size re-tuning, and wiring the d<=2 narrowing hint into live candidate matching.

Test plan

• pytest cardpicker/tests/ — 759 passed (2 pre-existing, unrelated moxfield failures)
• pre-commit run (ruff, isort, black, mypy, prettier) — all clean
• makemigrations --check — migration matches model state exactly
• Validated against 600 real card pairs harvested from the live full-catalog run

🤖 Generated with Claude Code

https://claude.ai/code/session_016i9S7LQsCL3FGaih3ZTRBJ