ci: retry a failed Fly deploy, and fail loudly when the two apps disagree

[paulieb89]

The incident this closes

On v1.15.1 the PyPI publish and the propertydata deploy succeeded while the
property-shared deploy failed twice, five minutes apart:

error releasing builder: deadline_exceeded: context deadline exceeded
timed out connecting to machine: failed to list workers:
  Unavailable: ... authentication handshake failed: EOF

The two deploys are independent leaf jobs, so one failing while the other
succeeds is a normal terminal state of this graph. Nothing observed the pair.
The apps ran different versions for about an hour — with the app that needed a
critical hotfix still serving the broken build — until a human noticed and
deployed manually with --depot=false.

Two changes: the cause, and the silence.

scripts/deploy_fly.py — the retry ladder

Depot first (the default and faster path; one intermittent incident is not a
reason to discard it permanently), then two --depot=false attempts — the one
causally relevant difference from the successful manual recovery.

The per-attempt 900s timeout is load-bearing. Both observed failures were
timeouts. Without a bound, one hung attempt consumes the job budget and the
fallback never runs, which would make the retry decorative.

--app and --config are now always explicit. CI previously relied on cwd
discovery of fly.toml, so the command that ran was not the command anyone had
written down.

scripts/verify_release.py — the job that would have noticed

needs: [deploy, deploy-propertydata] with if: ${{ !cancelled() }} — it must
run precisely when a deploy has failed. Not always(), so a cancelled run
isn't reported as drift.

Verified live (2026-09-04): both apps expose
/.well-known/mcp/server-card.json → serverInfo.version from the same
installed property-shared distribution. propertydata's /health carries no
version and cannot serve this.

It keeps two failures distinguishable:

• tag/pyproject mismatch — fails immediately with its own message. The
  version is baked into the image at build time, so tagging v1.18.3 without
  bumping pyproject.toml deploys two apps that both honestly report 1.18.2;
  no amount of polling fixes that.
• release drift — reported with both observed versions.

--require-consecutive 2 because Fly's proxy may front more than one Machine
and a single 200 cannot distinguish "propagated" from "hit the updated
Machine". Each app runs exactly one Machine today — an observation, not a
guarantee — so the defence stays. An unreachable target is a failure, not an
unknown.

Why scripts and not YAML

Retry logic in workflow YAML cannot be tested. The if: failure()
alternative needs continue-on-error, renders a failed first attempt as a
misleading green, and writes the flyctl arguments twice — and two copies
drifting apart is the exact failure class this change exists to close.

Both are stdlib-only operator scripts, loaded by path and driven through
injected runners, following scripts/fly_observability_snapshot.py. The suite
never shells out and never needs a Fly token.

Notable cases: exit 0 when the fallback succeeds (the bug the two-step form
would have had); a hung attempt is bounded and the fallback still runs; the
v1.15.1 incident replayed as test_one_target_behind_fails_and_names_both_versions;
a flapping version doesn't pass; a mismatch spends zero polling rounds.

Action pinning

setup-flyctl pinned @master → ed8efb3 (tag 1.6). That job holds a
production deploy token and @master runs whatever is on the branch at that
moment. The SHA is what @master already resolved to, so the pin changes
nothing but reproducibility. The flyctl binary stays unpinned — it talks to a
moving server API, so pinning it would turn a rare flake into certain rot.

One existing test updated, not relaxed

test_the_release_workflow_still_deploys_both_apps asserted the literal inline
flyctl deploy --remote-only --ha=false, which has moved into the script. The
fact it guards — every release still deploys propertydata, which the Stage 1
runbook's honesty depends on — is unchanged, so the assertions moved to where
each half now lives: the app/config pairing pinned in the workflow, the flags
pinned in the script.

Verification limit, stated

This cannot be verified end to end without a real release. The ladder and
the reconcile logic are unit-tested; the workflow wiring is only exercised on
the next release.

./scripts/validate.sh → 2040 passed, 27 skipped.