|
| 1 | +""" |
| 2 | +Two-threshold clustering over `Card.content_phash` (docs/features/printing-tags.md's |
| 3 | +hash-at-ingest architecture, 2026-07-16): replaces the disabled, fetch-based |
| 4 | +`compute_own_image_clusters` pre-pass entirely (see local_identify_printing_tags.py's own |
| 5 | +"DISABLED" comment at its old call site). Since content_phash is now persisted at ingest/backfill |
| 6 | +time (see local_phash.compute_content_phash_for_card, sources.update_database. |
| 7 | +hash_newly_created_cards, the local_backfill_content_phash management command), clustering a |
| 8 | +run's selected pool is a pure DB-column read + in-memory compute step - no network fetch, no |
| 9 | +sequential pre-pass, no observability gap. The pre-pass's own fixed ~21.6h sequential cost (the |
| 10 | +reason it was disabled) simply doesn't exist in this design. |
| 11 | +
|
| 12 | +Two tiers, two different trust levels - NOT the same operation at two thresholds, genuinely |
| 13 | +different semantics: |
| 14 | +
|
| 15 | +- d=0 (exact 64-bit hash match): sound entailment, identical reasoning to the original pre-pass |
| 16 | + - a distance-0 match among OUR OWN uploaded images most plausibly means a duplicate/ |
| 17 | + shared-source image, not independent depictions that coincidentally look alike (that's the |
| 18 | + *candidate* art-crop matching problem local_phash.find_best_match already handles separately, |
| 19 | + via a real DEFAULT_DISTANCE_THRESHOLD=20). An accepted vote on the representative propagates |
| 20 | + identically to every distance-0 member - see local_identify_printing_tags.run_pilot's write |
| 21 | + loop, unchanged from before. |
| 22 | +
|
| 23 | +- 0 < d <= 2: NOT entailment - a prior only, used to NARROW (never auto-vote) a member's own |
| 24 | + candidate list toward printings its near-duplicate cluster-mates are already candidates for. |
| 25 | + Same safety line as _narrow_candidates_by_expansion_hint (never narrows to empty, never |
| 26 | + touches select_candidates' ordering or the uncovered-printings-closed metric). Threshold |
| 27 | + justified two ways: (1) the LAION-scale precedent of d<=2 as the standard near-duplicate |
| 28 | + cutoff (see "Prior-art read" in docs/features/printing-tags.md); (2) this repo's own measured |
| 29 | + small-size-hashing drift test, where a real true-duplicate pair landed at exactly d=2 - see |
| 30 | + "Phash accuracy at small CDN sizes" in the same doc. d<=2 is REQUIRED, not optional, if |
| 31 | + small-size hashing is used for content_phash (which it is - see local_phash. |
| 32 | + INGEST_HASH_FETCH_DPI) - a d=0-only design would silently miss that class of true duplicate. |
| 33 | +
|
| 34 | +Neighbor search: chunked numpy XOR + popcount (numpy.bitwise_count, numpy>=2.0), NOT a Python |
| 35 | +pairwise loop and NOT one all-at-once O(N^2) memory allocation - prior MTG card-detector |
| 36 | +projects hit exactly the O(N*M) wall a naive implementation would (see "Prior-art read"'s |
| 37 | +brute-force-linear-scan note). Chosen over a BK-tree because the access pattern here is BATCH, |
| 38 | +not incremental: `run_pilot` computes clusters once per invocation for its whole selected pool, |
| 39 | +not one query at a time as new cards trickle in - a BK-tree earns its keep for the opposite |
| 40 | +pattern (repeated single-point nearest-neighbor lookups against a mostly-static corpus), which |
| 41 | +isn't what this call site does. Chunking over ROWS (not columns) keeps peak memory at |
| 42 | +chunk_size x N instead of N x N - at N~166k (the full-catalog run's own selected-pool size), |
| 43 | +N x N as even 1-byte-per-distance would be ~27GB; chunk_size=2000 keeps each chunk under 350MB. |
| 44 | +
|
| 45 | +**d=0 and d<=2 are computed as two INDEPENDENT steps, not one pass split afterward** - a |
| 46 | +deliberate robustness choice (advisor review, 2026-07-16), not an accident of implementation |
| 47 | +order. d=0 exact-match grouping is a plain dict grouping (measured: 0.13s at N=166,422 real |
| 48 | +scale, no numpy needed at all - equality doesn't need a distance computation). The d<=2 |
| 49 | +near-duplicate scan is the genuinely expensive part (measured: ~2-3 minutes at the same N, |
| 50 | +contended with this box's own concurrently-running full-catalog job at benchmark time - still a |
| 51 | +~500-650x win over the disabled pre-pass's ~21.6h, and it's pure in-memory compute, not network, |
| 52 | +so it doesn't compete for the same shared CDN request budget the old pre-pass did). Keeping |
| 53 | +these two steps independent means the safety-critical, already-proven vote-propagation tier |
| 54 | +(d=0) can NEVER be slowed down, blocked, or made incorrect by a bug or performance regression in |
| 55 | +the newer, less-battle-tested near-duplicate narrowing tier (d<=2) - if the latter ever needs to |
| 56 | +be disabled or reworked, the former is untouched. |
| 57 | +""" |
| 58 | + |
| 59 | +import collections |
| 60 | +import logging |
| 61 | +from dataclasses import dataclass |
| 62 | +from typing import TYPE_CHECKING, Any |
| 63 | + |
| 64 | +import numpy as np |
| 65 | + |
| 66 | +if TYPE_CHECKING: |
| 67 | + from cardpicker.local_identify_printing_tags import SelectedCard |
| 68 | + |
| 69 | +logger = logging.getLogger(__name__) |
| 70 | + |
| 71 | +_HASH_BITS = 64 |
| 72 | +_UNSIGNED_MASK = (1 << _HASH_BITS) - 1 |
| 73 | + |
| 74 | +# Same value local_clustering's own docstring justifies above (LAION-scale precedent + this |
| 75 | +# repo's own d=2 true-duplicate observation) - a module-level constant so both tiers below stay |
| 76 | +# in sync with the same number, and so a future re-tune has one place to change. |
| 77 | +NEAR_DUPLICATE_MAX_DISTANCE = 2 |
| 78 | + |
| 79 | +# Chunk over rows to bound peak memory - see module docstring's sizing note. Not tuned beyond |
| 80 | +# "keeps a chunk comfortably under 1GB at full-catalog scale" - a real profiling pass could |
| 81 | +# probably push this higher. |
| 82 | +DEFAULT_CHUNK_SIZE = 2000 |
| 83 | + |
| 84 | + |
| 85 | +@dataclass(frozen=True) |
| 86 | +class TwoThresholdClusterResult: |
| 87 | + # representative card_id -> the OTHER card_ids (never including the representative itself) |
| 88 | + # at EXACTLY distance 0 - an accepted vote on the representative should propagate to them. |
| 89 | + # Same shape as the old (now-removed) ClusterResult.members_by_representative, so |
| 90 | + # run_pilot's existing propagation/absorption logic needs no changes to consume this. |
| 91 | + members_by_representative: dict[int, list[int]] |
| 92 | + # card_id -> the set of OTHER card_ids within NEAR_DUPLICATE_MAX_DISTANCE (INCLUDING the |
| 93 | + # distance-0 members above, since a d<=2 narrowing prior is still valid information even for |
| 94 | + # a card that's also an exact-match representative) - a prior for candidate narrowing only, |
| 95 | + # never auto-voted. Absent key means "no near-duplicates found" (empty prior, not narrowed). |
| 96 | + near_duplicate_ids_by_card_id: dict[int, set[int]] |
| 97 | + |
| 98 | + |
| 99 | +def _unsigned_hash_array( |
| 100 | + card_ids: list[int], hash_by_card_id: dict[int, int] |
| 101 | +) -> "np.ndarray[Any, np.dtype[np.uint64]]": |
| 102 | + return np.array([hash_by_card_id[c] & _UNSIGNED_MASK for c in card_ids], dtype=np.uint64) |
| 103 | + |
| 104 | + |
| 105 | +def _find_pairs_within_distance( |
| 106 | + hashes: "np.ndarray[Any, np.dtype[np.uint64]]", max_distance: int, chunk_size: int = DEFAULT_CHUNK_SIZE |
| 107 | +) -> list[tuple[int, int, int]]: |
| 108 | + """ |
| 109 | + Returns (i, j, distance) triples with i < j (array indices into `hashes`, not card ids) and |
| 110 | + distance <= max_distance. Chunks over rows: for each block of `chunk_size` hashes, XORs the |
| 111 | + whole block against the FULL array in one vectorized op, then popcounts the result - avoids |
| 112 | + both a Python-level O(N^2) loop and an O(N^2) all-at-once allocation (see module docstring). |
| 113 | + """ |
| 114 | + n = len(hashes) |
| 115 | + pairs: list[tuple[int, int, int]] = [] |
| 116 | + for start in range(0, n, chunk_size): |
| 117 | + end = min(start + chunk_size, n) |
| 118 | + block = hashes[start:end] # shape (b,) |
| 119 | + xor = block[:, None] ^ hashes[None, :] # shape (b, n) |
| 120 | + distances = np.bitwise_count(xor) # shape (b, n) |
| 121 | + for local_i in range(end - start): |
| 122 | + i = start + local_i |
| 123 | + # only j > i: avoids self-pairs (distance 0 to itself) and double-counting (i, j) |
| 124 | + # and (j, i) as separate entries. |
| 125 | + row = distances[local_i, i + 1 :] |
| 126 | + close_local_js = np.nonzero(row <= max_distance)[0] |
| 127 | + for local_j in close_local_js: |
| 128 | + j = i + 1 + int(local_j) |
| 129 | + pairs.append((i, j, int(row[local_j]))) |
| 130 | + return pairs |
| 131 | + |
| 132 | + |
| 133 | +def _compute_exact_match_clusters(hash_by_card_id: dict[int, int]) -> dict[int, list[int]]: |
| 134 | + """d=0 tier: group by exact hash value - a direct dict grouping, not the vectorized scan |
| 135 | + below (equality doesn't need a distance computation at all). Measured 0.13s at N=166,422 - |
| 136 | + effectively free, independent of how expensive the d<=2 tier is or ever becomes.""" |
| 137 | + card_ids_by_hash: dict[int, list[int]] = collections.defaultdict(list) |
| 138 | + for card_id, h in hash_by_card_id.items(): |
| 139 | + card_ids_by_hash[h].append(card_id) |
| 140 | + |
| 141 | + members_by_representative: dict[int, list[int]] = {} |
| 142 | + for same_hash_card_ids in card_ids_by_hash.values(): |
| 143 | + if len(same_hash_card_ids) < 2: |
| 144 | + continue |
| 145 | + representative_id = min(same_hash_card_ids) |
| 146 | + others = [c for c in same_hash_card_ids if c != representative_id] |
| 147 | + members_by_representative[representative_id] = others |
| 148 | + return members_by_representative |
| 149 | + |
| 150 | + |
| 151 | +def _compute_near_duplicate_hints(hash_by_card_id: dict[int, int]) -> dict[int, set[int]]: |
| 152 | + """d<=2 tier: the vectorized chunked scan - the genuinely expensive part (~2-3 minutes at |
| 153 | + N=166,422 real scale, see module docstring). A deliberately separate function from the d=0 |
| 154 | + tier above, called independently by compute_two_threshold_clusters - see module docstring's |
| 155 | + "computed as two INDEPENDENT steps" note for why that separation matters.""" |
| 156 | + card_ids = list(hash_by_card_id.keys()) |
| 157 | + hashes = _unsigned_hash_array(card_ids, hash_by_card_id) |
| 158 | + pairs = _find_pairs_within_distance(hashes, NEAR_DUPLICATE_MAX_DISTANCE) |
| 159 | + |
| 160 | + near_duplicate_ids_by_card_id: dict[int, set[int]] = collections.defaultdict(set) |
| 161 | + for i, j, _distance in pairs: |
| 162 | + card_i, card_j = card_ids[i], card_ids[j] |
| 163 | + near_duplicate_ids_by_card_id[card_i].add(card_j) |
| 164 | + near_duplicate_ids_by_card_id[card_j].add(card_i) |
| 165 | + return dict(near_duplicate_ids_by_card_id) |
| 166 | + |
| 167 | + |
| 168 | +def compute_two_threshold_clusters(selected: list["SelectedCard"]) -> TwoThresholdClusterResult: |
| 169 | + """ |
| 170 | + The stored-hash replacement for the old fetch-based compute_own_image_clusters - see module |
| 171 | + docstring for the full d=0/d<=2 semantics. Cards with a NULL content_phash (not yet |
| 172 | + hashed - see local_phash.compute_content_phash_for_card and the backfill command) are |
| 173 | + excluded from clustering entirely, treated as always-singleton (same safe fallback the |
| 174 | + disabled pre-pass had for a failed fetch) - this function does no fetching or hashing of its |
| 175 | + own, purely reads whatever's already on `s.card.content_phash`. |
| 176 | + """ |
| 177 | + hash_by_card_id: dict[int, int] = { |
| 178 | + s.card.pk: s.card.content_phash for s in selected if s.card.content_phash is not None |
| 179 | + } |
| 180 | + if len(hash_by_card_id) < 2: |
| 181 | + return TwoThresholdClusterResult(members_by_representative={}, near_duplicate_ids_by_card_id={}) |
| 182 | + |
| 183 | + members_by_representative = _compute_exact_match_clusters(hash_by_card_id) |
| 184 | + try: |
| 185 | + near_duplicate_ids_by_card_id = _compute_near_duplicate_hints(hash_by_card_id) |
| 186 | + except Exception: |
| 187 | + # The d<=2 tier is a NARROWING PRIOR, never entailment (see module docstring) - a |
| 188 | + # failure here must never take down d=0's already-proven vote propagation. Falls back |
| 189 | + # to "no near-duplicate hints available this run", not a crash. |
| 190 | + logger.exception("Near-duplicate (d<=2) scan failed - continuing with d=0 clusters only") |
| 191 | + near_duplicate_ids_by_card_id = {} |
| 192 | + |
| 193 | + return TwoThresholdClusterResult( |
| 194 | + members_by_representative=members_by_representative, |
| 195 | + near_duplicate_ids_by_card_id=near_duplicate_ids_by_card_id, |
| 196 | + ) |
| 197 | + |
| 198 | + |
| 199 | +__all__ = [ |
| 200 | + "NEAR_DUPLICATE_MAX_DISTANCE", |
| 201 | + "DEFAULT_CHUNK_SIZE", |
| 202 | + "TwoThresholdClusterResult", |
| 203 | + "compute_two_threshold_clusters", |
| 204 | +] |
0 commit comments