Publish every architecture to one bucket per cloud

[artemnikitin]

Publishes both architectures to one bucket per cloud, under an <arch>/ key prefix, instead of a separate bucket per architecture.

<images-bucket>/
  amd64/tenant-1-kibana-rootfs.ext4
  arm64/tenant-1-kibana-rootfs.ext4

The prefix uses the Go vocabulary (amd64, arm64) that TARGET_PLATFORM already carries — deliberately not the AWS x86_64 spelling, which the agent would never look under. An unrecognised TARGET_PLATFORM is rejected rather than published without a prefix.

What changed

• scripts/push-images.sh — derives TARGET_ARCH from TARGET_PLATFORM and prefixes the destination key
• Makefile — threads TARGET_PLATFORM through the push-* targets, and drops its own default so an unset push fails instead of assuming amd64
• .github/workflows/build-images.yaml — the per-arch bucket resolution step is gone; both legs upload unconditionally to S3_IMAGES_BUCKET / GCS_IMAGES_BUCKET
• scripts/test-push-images.sh — asserts prefixed destinations, arm64-prefix, bad-platform, unset-platform, and that the workflow passes the matrix expression (not merely the key) to both upload steps (13 checks, all passing)
• docs/ci-pipeline.md, AGENTS.md — rewritten bucket semantics plus a migration runbook

This reverts much of #28

S3_IMAGES_BUCKET_ARM64 / _AMD64 and the "no bucket configured" ::warning are removed. Two comments went with them because they became false — the arm64 leg now does get AWS credentials and does upload, and "publishing an architecture nobody consumes" is no longer a reachable state.

That mechanism existed because both architectures produced identical object names, making one bucket unsafe. Putting the architecture in the key removes the reason for it.

Why this is a correctness fix

Under per-arch buckets, pointing a node at the wrong bucket gave a silent wrong-architecture boot — a guest kernel panic at init, which is how the Aug 7 mismatch presented. With the architecture in the key, a node can only address its own images.

Scoped after review: a missing image fails loudly only on a node with no local copy — the agent falls back to a cached image at debug level, so an existing node stays quiet. So verify bucket contents directly rather than waiting for a node to report a gap. A freshly provisioned node has no cache and does fail loudly.

It also makes a mixed-architecture fleet free: node configs carry no architecture, so one desired state serves both.

Deploy order

No live environments, so there is no migration and merge order is unconstrained. The only ordering that matters operationally: publish images before deploying nodes, because a fresh node has no image cache to fall back on.

Companions: artemnikitin/firework#45 (agent resolves the prefix) and artemnikitin/firework-deployment-example#22 (node bootstrap picks its own prefix — a functional fix, found in review).

Storage impact: the arm64 leg starts uploading for the first time, so images-bucket usage roughly doubles. CI IAM is bucket-wide on both clouds, so no policy change is needed.

Restored safeguard (review finding)

Collapsing the per-arch bucket resolution also removed the warning that fired when a publishing run had no bucket configured — leaving builds green while uploading nothing. A publishing run now fails when neither bucket variable is set, and warns per provider when only one is.

Migration runbook removed (review finding)

It had four defects that only mattered if followed: no step to repoint nodes off a separate ARM64 bucket, verification covering only amd64 and only S3, a requirement that every flat object have a prefixed counterpart while acknowledging some are intentionally obsolete, and a listing that parsed PRE <prefix>/ rows into empty entries — making the pre-deletion "must be empty" check unreliable. With no deployments on the old layout, it's replaced by a short note on deleting inert root-level objects, using list-objects-v2 --delimiter / so those empty entries cannot occur.

The failure mode this guards against

Both architectures now publish to one bucket under identical object names. If an upload step loses its TARGET_PLATFORM env, the arm64 leg publishes under amd64/ and overwrites the amd64 images — silently, until a node boots a guest built for the wrong architecture.

The backend cases run the Makefile directly and never see the workflow, so that wiring had no coverage.

Review caught that the first version of the guard only checked the key was present — hardcoding TARGET_PLATFORM: linux/amd64 in all three steps passed the entire suite. It now matches ${{ matrix.target_platform }}, verified against exactly that bypass.

A second footgun, same shape: make build-arm64 sets TARGET_PLATFORM inside its own recipe while push-s3 read it from the make environment, so make build-arm64 && make push-s3 silently published arm64 images under amd64/. push-images.sh now requires an explicit TARGET_PLATFORM; make build is unchanged because build-images.sh defaults on its own.

Testing

bash scripts/test-push-images.sh — 13/13 pass. shellcheck clean on both scripts. Workflow YAML parses and the step list is as intended.