Persist scryfall_cache volume and fail loud on a missing cache - #412
Merged
Conversation
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #402. The Scryfall bulk-data cache (
MPCAutofill/scryfall_cache/default_cards.json, ~558MB) lived as a plain file inside the django/worker container filesystem with no persistent volume — every image rebuild silently destroyed it. Deploy-2 (2026-07-23) did exactly that, andlocal_calculate_verdictswent on to run toCOMPLETEDwith a silently degraded (empty) back-face lookup, no error anywhere.Two independent fixes, both requested by the issue:
docker/docker-compose.prod.ymlgains a namedscryfall_cachevolume (mirroring the existingpostgres_data/elasticsearch_datapattern), mounted on bothdjango(manualexec-run commands) andworker(the weeklyimport_canonical_card_data/update_dfcsqclusterschedule also reads/refreshes this same file — confirmed viamtg.py'sget_canonical_cards_and_artists). The dev compose (docker/docker-compose.yml) only defines thepostgres/elasticsearchservices, notdjango/worker, so it doesn't have this hazard and needed no change.printing_metadata_import.ensure_scryfall_cache_present()is a new existence-only check (deliberately independent of the existing weekly_is_stalestaleness/re-download logic, which is untouched), called at the very start oflocal_calculate_verdicts'sCommand.handle()before any card-by-card work. Raises aCommandErrornaming the missing path unless--allow-missing-scryfall-cacheis passed explicitly — a message distinct fromget_back_face_names's existing soft "empty set" per-card path, which stays as-is (still the right behaviour for a single lookup).Deploy-time activation note: the compose change only takes effect at the next
up --build -ddeploy — it does not retroactively attach to the currently-runningmpcautofill_django/mpcautofill_workercontainers. No containers were rebuilt or restarted from this session.Checklist
pre-commitand installed the hooks withpre-commit installbefore creating any commits.docker compose -f docker-compose.prod.yml config --quietvalidates the compose file cleanly (no volume/syntax errors).TestScryfallCacheGuard(5 tests) covering the guard directly (missing/present file) and wired through the management command (refuses without the flag, proceeds with the flag, proceeds when the cache is genuinely present). The 4 pre-existingTestCommandLedgerHardeningAndDryRunGuardtests were updated to pass--allow-missing-scryfall-cachesince the test environment has no real cache file on disk (they're testing the dry-run guard, not this one).1904 passed, 4 skipped(the 4 skips are the pre-existing documented named skips) viapytest cardpickerin/home/ubuntu/.venvs/mpcautofill-pilot.pre-commit run(ruff/isort/black/mypy/prettier) clean on the final commit;docs_lint.pyclean.docs/troubleshooting.md: new symptom-first entry for this incident (silently degraded back-face lookup after a rebuild), covering both the cause and the two-part fix, so a future occurrence (e.g. a box rebuild that skips the compose file) is diagnosable quickly.Notes for reviewer
import_scryfall_printing_metadata/import_canonical_card_datarun (already scheduled weekly viaworker'sqcluster) will populate it, same as any fresh bootstrap.docs/troubleshooting.md/docs/infrastructure.mdfor an active migration freeze per CLAUDE.md's process — none found, and N/A here regardless (no schema change).