Skip to content

Add Card.md5_checksum substrate and a listings-rewalk backfill command - #477

Merged
WilfordGrimley merged 2 commits into
masterfrom
md5-checksum-substrate
Jul 25, 2026
Merged

Add Card.md5_checksum substrate and a listings-rewalk backfill command#477
WilfordGrimley merged 2 commits into
masterfrom
md5-checksum-substrate

Conversation

@WilfordGrimley

@WilfordGrimley WilfordGrimley commented Jul 25, 2026

Copy link
Copy Markdown

Summary

  • References issue md5 identity groups: checksum substrate, evidence transfer, and group-level vote pooling #473's PR-1 — checksum substrate section only (the four owner rulings recorded there are binding context; this PR does NOT implement evidence transfer/staleness (PR-2) or group-level vote pooling (PR-3, PROTECTED CORE) — the issue stays open until all three land).
  • Adds Card.md5_checksum (CharField(max_length=32, null=True, blank=True, db_index=True), migration 0084_card_md5_checksum) — copied verbatim from the Drive API's own md5Checksum listing field, never computed/derived locally (governing "we index, we do not store images" premise).
  • Stamps it at the existing crawler seam: GoogleDrive.get_all_images_inside_folder now requests md5Checksum in its listing fields and carries it through Image.md5_checksumtransform_image_into_object → new Card rows; bulk_sync_objects's change-detection now also refreshes it on an ordinary re-scan of an existing card (free metadata, no extra fetch, unlike content_phash). LocalFile.get_all_images_inside_folder never sets it — stays NULL, which per issue md5 identity groups: checksum substrate, evidence transfer, and group-level vote pooling #473 ruling 3 is the correct "group of one" state.
  • Adds manage.py backfill_md5_checksums: re-walks every GOOGLE_DRIVE source's listing (metadata only, zero image fetches, reuses explore_folder/the existing rate-limited Drive client — no new pacing hand-rolled), reconciles Card.md5_checksum against each listing, and prints matched_files/dupe_groups/dupe_files/dupe_factor for reconciliation against issue Ingest Drive file checksums; dedupe extraction via byte-identical equivalence classes #442's own sizing walk (18.87% dupe rate, 24,712/130,960 files, 12,275 groups) before any --write. Wired through the #373/pilot_run_lifecycle forced-dry-run-before-write convention (PilotRunLedger row either way, --skip-dryrun-check escape hatch). LOCAL_FILE sources are reported as skipped, never walked or touched.
  • Ledger update: added a CLEAN row for cardpicker/md5_backfill.py to docs/upstreaming/extractable-primitives.md (zero dependency on the vote system/CanonicalPrinting-consensus/auth — verified against the tether's own fork-only-module allowlist).

Test plan

  • cardpicker/tests/test_sources.pytransform_image_into_object stamps/leaves-null md5_checksum correctly; bulk_sync_objects stamps on create and refreshes on update from the md5 change alone (no other field changed); LOCAL_FILE all-sources run leaves every card's md5_checksum null.
  • cardpicker/tests/test_md5_backfill.py (new, 20 tests) — walk_source_checksums reachable/unreachable; run_md5_backfill dry-run vs --write, LOCAL_FILE sources skipped-not-walked, unreachable sources reported, already-correct checksums not double-counted, un-indexed listing entries never invented into a Card, dupe-group stats computed globally across sources, --source-key filtering; the management command's forced-dry-run guard (refused without a prior matching dry-run, succeeds after one, refused on scope mismatch, --skip-dryrun-check bypass recorded on the ledger).
  • Ran via /home/ubuntu/.venvs/mpcautofill-pilot against the documented testcontainers Postgres/Elasticsearch pattern (docs/troubleshooting.md): test_sources.py + test_md5_backfill.py together — 90 passed, 2 skipped (the two pre-existing Google-Drive-credential-gated snapshot tests, same named skip as before this change). test_local_file_source.py + test_import_sources.py — 20 passed, confirming no regression in adjacent source-handling code.
  • pre-commit run (ruff, isort, black, mypy across MPCAutofill/, prettier) — clean.
  • .github/scripts/docs_lint.py — clean, including the new extractable-primitives CLEAN row's mechanical tether check.
  • Not run: an actual --write against production data (out of scope for this PR — the command exists, the reconciliation-vs-Ingest Drive file checksums; dedupe extraction via byte-identical equivalence classes #442 check is the acceptance gate the issue names, and it's a normal deploy-time operational step, not something this PR performs).

🤖 Generated with Claude Code

2026-07-25 addendum: also adds Card.sha256_checksum to the same migration/listing walk/backfill pass (owner-approved, issue #473 comment thread) — PR-2's evidence-transfer premise ("identical bytes ⇒ identical evidence") needs a cryptographic match, since md5 collisions are constructible; groups still key on md5 only, sha256 is the transfer-safety pairing and future federation join key.

…llable

Adds Card.md5_checksum (nullable, from Drive listing md5Checksum), stamps
it going forward in update_database, and adds backfill_md5_checksums (a
listings re-walk, no image fetches) for existing rows. Issue #473 PR-1.
Owner-approved addition to issue #473 PR-1 (2026-07-25 evening): PR-2's
evidence-transfer premise needs a cryptographic match, not just md5
(collision-constructible), so sha256 rides the same listing walk,
migration, and backfill pass. Groups still key on md5 only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant