Add Card.md5_checksum substrate and a listings-rewalk backfill command - #477
Merged
Conversation
…llable Adds Card.md5_checksum (nullable, from Drive listing md5Checksum), stamps it going forward in update_database, and adds backfill_md5_checksums (a listings re-walk, no image fetches) for existing rows. Issue #473 PR-1.
4 tasks
Owner-approved addition to issue #473 PR-1 (2026-07-25 evening): PR-2's evidence-transfer premise needs a cryptographic match, not just md5 (collision-constructible), so sha256 rides the same listing walk, migration, and backfill pass. Groups still key on md5 only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 25, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Card.md5_checksum(CharField(max_length=32, null=True, blank=True, db_index=True), migration0084_card_md5_checksum) — copied verbatim from the Drive API's ownmd5Checksumlisting field, never computed/derived locally (governing "we index, we do not store images" premise).GoogleDrive.get_all_images_inside_foldernow requestsmd5Checksumin its listing fields and carries it throughImage.md5_checksum→transform_image_into_object→ newCardrows;bulk_sync_objects's change-detection now also refreshes it on an ordinary re-scan of an existing card (free metadata, no extra fetch, unlikecontent_phash).LocalFile.get_all_images_inside_foldernever sets it — staysNULL, which per issue md5 identity groups: checksum substrate, evidence transfer, and group-level vote pooling #473 ruling 3 is the correct "group of one" state.manage.py backfill_md5_checksums: re-walks everyGOOGLE_DRIVEsource's listing (metadata only, zero image fetches, reusesexplore_folder/the existing rate-limited Drive client — no new pacing hand-rolled), reconcilesCard.md5_checksumagainst each listing, and printsmatched_files/dupe_groups/dupe_files/dupe_factorfor reconciliation against issue Ingest Drive file checksums; dedupe extraction via byte-identical equivalence classes #442's own sizing walk (18.87% dupe rate, 24,712/130,960 files, 12,275 groups) before any--write. Wired through the#373/pilot_run_lifecycleforced-dry-run-before-write convention (PilotRunLedgerrow either way,--skip-dryrun-checkescape hatch).LOCAL_FILEsources are reported as skipped, never walked or touched.CLEANrow forcardpicker/md5_backfill.pytodocs/upstreaming/extractable-primitives.md(zero dependency on the vote system/CanonicalPrinting-consensus/auth — verified against the tether's own fork-only-module allowlist).Test plan
cardpicker/tests/test_sources.py—transform_image_into_objectstamps/leaves-nullmd5_checksumcorrectly;bulk_sync_objectsstamps on create and refreshes on update from the md5 change alone (no other field changed); LOCAL_FILE all-sources run leaves every card'smd5_checksumnull.cardpicker/tests/test_md5_backfill.py(new, 20 tests) —walk_source_checksumsreachable/unreachable;run_md5_backfilldry-run vs--write, LOCAL_FILE sources skipped-not-walked, unreachable sources reported, already-correct checksums not double-counted, un-indexed listing entries never invented into a Card, dupe-group stats computed globally across sources,--source-keyfiltering; the management command's forced-dry-run guard (refused without a prior matching dry-run, succeeds after one, refused on scope mismatch,--skip-dryrun-checkbypass recorded on the ledger)./home/ubuntu/.venvs/mpcautofill-pilotagainst the documented testcontainers Postgres/Elasticsearch pattern (docs/troubleshooting.md):test_sources.py+test_md5_backfill.pytogether — 90 passed, 2 skipped (the two pre-existing Google-Drive-credential-gated snapshot tests, same named skip as before this change).test_local_file_source.py+test_import_sources.py— 20 passed, confirming no regression in adjacent source-handling code.pre-commit run(ruff, isort, black, mypy acrossMPCAutofill/, prettier) — clean..github/scripts/docs_lint.py— clean, including the new extractable-primitivesCLEANrow's mechanical tether check.--writeagainst production data (out of scope for this PR — the command exists, the reconciliation-vs-Ingest Drive file checksums; dedupe extraction via byte-identical equivalence classes #442 check is the acceptance gate the issue names, and it's a normal deploy-time operational step, not something this PR performs).🤖 Generated with Claude Code
2026-07-25 addendum: also adds
Card.sha256_checksumto the same migration/listing walk/backfill pass (owner-approved, issue #473 comment thread) — PR-2's evidence-transfer premise ("identical bytes ⇒ identical evidence") needs a cryptographic match, since md5 collisions are constructible; groups still key on md5 only, sha256 is the transfer-safety pairing and future federation join key.