docs(ops): runbook for releasing, and for refreshing the snapshot independently - #72
Merged
Merged
Conversation
…ependently Neither procedure was written down. The release process was scattered across CLAUDE.md, the rollout record and the workflow file; the refresh procedure did not exist at all, because until today there had never been a second artifact. The distinction the runbook opens with is the one that matters: a code release and a snapshot refresh are INDEPENDENT. A refresh publishes an artifact and moves one pointer -- no version bump, no deploy, no PyPI. Confusing them is the main way this goes wrong, and the table is there so the difference is visible before either is started. Records what was learned doing it for the first time today: * Publish a GitHub Release; never `fly deploy` locally. `fly deploy` ships the working directory, not a commit, and only `release.yml`'s checkout of the tag makes what shipped knowable. * Three version fields must agree with the tag or the reconcile job fails the release before deploying. `uv lock` after, which validate.sh checks first. * Tags here are lightweight, so `--notes-from-tag` publishes nothing. Notes have historically been the CHANGELOG entry body. * Publish order to Tigris, and why: bundle, read back and VERIFY, manifest, `current.json` last. It is the single control point every Machine reads on next boot, so stopping before that step changes nothing for anyone. * Check the key fingerprint first. A ReadOnly key fails the first PUT with a 403 indistinguishable from a signing error -- the one genuinely confusing failure. * Cadence: HMLR publishes monthly around the 28th; coverage end is the first of the publication month minus a day; FRESHNESS_WARNING_DAYS is 45, so a monthly refresh stays inside the threshold and a skipped month announces itself in every response. Registration lag is separate and far longer -- the most recent covered month held 29% of a typical month's sales when measured. Public rather than private, matching what docs/ops/ already does. Stated explicitly with the reasoning, because the choice is only safe while the line holds: the bucket is IsPublic=false so its name grants nothing, a key label is not a key, a sha256 prefix of an id is not the id -- and ids, secrets and tokens never appear in any repo file. Also records the coupling that currently breaks the separation the runbook describes: boot_only_verify.py pins EXPECTED_BUNDLE_BYTES and is deliberately not CLI-overridable, so a new artifact needs a constant edited and an image shipped before that verifier reports true. Out-of-band only, nothing blocked, but it should derive from the manifest.
The runbook said "PUT the bundle", which is exactly what failed on the first real publish. A single PUT of the 1.19 GB bundle was rejected with XAmzContentSHA256Mismatch after transferring the whole object -- the server received a complete request whose body hashed differently from the signed header. Small objects upload fine by either route, so the fault is in streaming a very large body as one request, not in the signing. Recorded with the reason, because a runbook that names the failure is worth more than one that just prescribes the fix: multipart is better here beyond merely working, since each part carries its own sha256 and is rejected on arrival, so corruption fails on the offending part in seconds rather than at the end of the whole object. Also records that the failure changed nothing in production, because it stopped at step 1 and current.json still named the previous manifest. That ordering is the safety property the section opens with, and it has now been tested by accident rather than only asserted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Neither procedure was written down. The release process was scattered across
CLAUDE.md, the rollout record and the workflow file; the refresh proceduredidn't exist at all, because until today there had never been a second artifact.
The distinction it opens with
Confusing them is the main way this goes wrong.
What it records, learned doing it today
fly deploylocally.fly deployshipsthe working directory, not a commit.
deploying.
uv lockafter —validate.shchecks it first, and did.--notes-from-tagpublishes nothing.current.jsonlast. Stopping before the last step changes nothing foranyone.
403 indistinguishable from a signing error.
publication month minus a day;
FRESHNESS_WARNING_DAYS = 45, so a monthlyrefresh stays inside the threshold and a skipped month announces itself in
every response. Registration lag is separate and far longer — the most recent
covered month held 29% of a typical month's sales.
Public vs private — public, stated explicitly
Matches what
docs/ops/already does, and the reasoning is written down becausethe choice is only safe while the line holds: the bucket is
IsPublic=falsesoits name grants nothing, a key label is not a key, a sha256 prefix of an id is
not the id. Ids, secrets and tokens never appear in any repo file — that
sentence is in the doc, and it's the only match a credential-shaped grep finds.
Also records a coupling worth removing
boot_only_verify.pypinsEXPECTED_BUNDLE_BYTESand is deliberately notCLI-overridable, so a new artifact needs a constant edited and an image shipped
before that verifier reports true — dragging a code release into every refresh,
exactly the separation the runbook opens with. Out-of-band only, nothing blocked.
./scripts/validate.sh→ 2220 passed, 28 skipped.