Skip to content

Two pilot fast-follows: expansion_hint narrowing, name-frequency elimination - #26

Merged
WilfordGrimley merged 5 commits into
masterfrom
worktree-pilot-fastfollows
Jul 16, 2026
Merged

Two pilot fast-follows: expansion_hint narrowing, name-frequency elimination#26
WilfordGrimley merged 5 commits into
masterfrom
worktree-pilot-fastfollows

Conversation

@WilfordGrimley

@WilfordGrimley WilfordGrimley commented Jul 16, 2026

Copy link
Copy Markdown

Summary

Two fast-follows on top of the pilot's coverage-gap/ordering work (already merged, full-catalog run in progress), both researched and sized before building - neither needed a schema change.

expansion_hint candidate narrowing: Card.expansion_hint already exists and is already populated at import time (cardpicker.tags.Tags.extract) - deductive_backfill's own D2 tier already trusts it. Newly wired into the pilot: narrows the candidate list run_ocr_for_card/run_phash_for_card/run_fallback_for_card consider, scoped strictly to engine-matching (never touches select_candidates's ordering or coverage queries). Never narrows to empty - falls back to the full list if the hint matches nothing (~9% measured data-quality case).

  • Real yield: of 2,466 pilot-eligible cards with a real hint, 645 currently get skipped by phash outright (too-many-candidates) - narrowing brings 407 back under the cap. OCR's exact-match logic doesn't benefit from narrowing; this is a phash-only unlock in practice.

Name-frequency elimination: new run_name_frequency_elimination + management command local_name_frequency_elimination. For a name with exactly one uncovered printing AND exactly one unresolved pilot-eligible card, the match is deducible by elimination alone - no image fetch, no OCR/phash.

  • The safety gate is the whole point: a name with multiple unresolved cards and one uncovered printing doesn't tell you which card is the missing one (others could be redundant depictions of an already-covered printing). The naive "one uncovered printing" version (no card-count gate) was the originally-researched number; adding "and exactly one unresolved card too" is what makes it sound, not a refinement.
  • Measured against the full catalog (not a sample): 2,076 names have exactly one uncovered printing; only 1,678 also have exactly one unresolved eligible card - the naive version would have voted incorrectly, on average, for the other ~400 names' cards.
  • Confidence deliberately modest (0.6 vs OCR/phash's 0.85/0.75/0.8) - a structural deduction is weaker evidence than an engine that looked at the image. Still just a vote, never a direct resolve - same consensus/gate-check discipline, same batch-flush checkpointing as run_pilot.

Test plan

  • pytest cardpicker/tests/test_local_identify_printing_tags.py - 100 passed, 4 skipped (pre-existing skips from the earlier cluster-dedup pre-pass disablement, unrelated to this PR)
  • mypy --config-file mypy.ini MPCAutofill/ - clean
  • black/ruff/prettier/pre-commit hooks passed
  • New tests cover: narrowing's empty-fallback and cap-crossing behavior; the 1:1 gate's positive case AND its specific negative case (multiple unresolved cards sharing a name); idempotence; dry-run; token/cardback exclusion; the management command's dry-run and real-run output

🤖 Generated with Claude Code

https://claude.ai/code/session_016i9S7LQsCL3FGaih3ZTRBJ

expansion_hint candidate narrowing: uses the already-populated
Card.expansion_hint field to narrow the candidate list OCR/phash/
fallback consider, unlocking phash for 407 cards currently skipped on
too-many-candidates. Scoped to engine-matching only - never touches
coverage/ordering queries.

Name-frequency elimination: new run_name_frequency_elimination +
management command, votes for a card when its name has exactly one
uncovered printing AND exactly one unresolved eligible card - the 1:1
gate is what makes the deduction sound (a name with multiple
unresolved cards and one uncovered printing doesn't tell you which
card is the missing one). 1,678 real cards qualify under the safe
gate, measured against the full catalog.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016i9S7LQsCL3FGaih3ZTRBJ
WilfordGrimley and others added 4 commits July 16, 2026 04:04
Django DB connections are thread-local; a pool recreated inside the
chunk loop leaked one Postgres connection per worker per chunk, since
nothing closes a connection when its owning thread is torn down. At
workers=7/batch_size=25 against max_connections=100 this crashed the
live full-catalog run in ~3 minutes with "too many clients already".
Production site traffic was unaffected - confirmed 200s on both
domains, connections recovered once the crashed process exited.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016i9S7LQsCL3FGaih3ZTRBJ
# Conflicts:
#	docs/features/printing-tags.md
tmikonen/magic_card_detector's threshold is a population z-score, not a
raw Hamming distance - not directly reusable for the pilot's d=0/d<=2
tiers. Neither prior-art repo has working art-region hash code, just an
unimplemented idea. Also logs the fork-context-compaction scope-drift
incident as a reusable lesson.
@WilfordGrimley
WilfordGrimley marked this pull request as ready for review July 16, 2026 09:43
@WilfordGrimley
WilfordGrimley merged commit 3b2b5b7 into master Jul 16, 2026
1 of 2 checks passed
WilfordGrimley added a commit that referenced this pull request Jul 16, 2026
get_baked_git_sha reads a GIT_SHA file baked into the image at build
time (Dockerfile ARG + docker-compose build.args, both now require the
documented GIT_SHA=$(git rev-parse --short HEAD) prefix on the rebuild
command) - best-effort visibility only, logged at startup, never the
gate itself.

find_stale_applied_migrations is the actual hard gate: compares what
this image's own migrations/ directory knows about against what the DB
reports as applied (MigrationLoader vs MigrationRecorder) - pure DB+code
introspection, automates the PR #24/#26 stale-image lesson instead of
relying on someone remembering to check docker images timestamps.

Caught a real bug via the new test: MigrationRecorder.applied_migrations()
returns a dict, not a set - the first draft's `applied - disk` would have
raised TypeError at the first real invocation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016i9S7LQsCL3FGaih3ZTRBJ
@WilfordGrimley
WilfordGrimley deleted the worktree-pilot-fastfollows branch July 19, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant