ci: publish amd64 rootfs images for AWS, and document the bucket semantics - #28
Merged
Conversation
The AWS data plane in firework-deployment-example is switching its default node type from bare-metal Graviton to x86_64 instances using nested virtualization, because AWS added nested virtualization on virtual EC2 instances and metal forced a 64 vCPU purchase. That makes S3_IMAGES_BUCKET_AMD64 a requirement for AWS deployments rather than an optional cross-backend extra, which the bucket configuration section did not say. No build or workflow changes are needed — CI already produces amd64 rootfs images and already supports this bucket variable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The AWS data plane in firework-deployment-example now defaults to x86_64 nodes using nested virtualization rather than bare-metal Graviton, and GCP has always been x86_64. Both providers therefore consume amd64 rootfs images, but the amd64 build resolved its S3 target from S3_IMAGES_BUCKET_AMD64 only. That variable is not configured, so the amd64 build silently skipped its S3 upload and the only images in S3 were arm64. An x86_64 node running an arm64 rootfs boots its kernel, mounts the root filesystem, then panics executing init with ENOEXEC, which presents as every service failing health checks rather than as a deploy-time error. Make the legacy S3_IMAGES_BUCKET mean the amd64 S3 bucket, matching how GCS_IMAGES_BUCKET already means the amd64 GCS bucket. The amd64 build now falls back to it and publishes into the existing bucket under the same object names, so no new bucket, node IAM change, or s3_images_bucket_id repoint is needed. Publishing arm64 becomes opt-in via S3_IMAGES_BUCKET_ARM64. Flip the local TARGET_PLATFORM default to linux/amd64 as well. It is what both providers run, and leaving it at arm64 would let a local make build && make push-s3 push arm64 images into what is now the amd64 bucket. Refs artemnikitin/firework-deployment-example#13 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
artemnikitin
changed the title
Removing the S3_IMAGES_BUCKET fallback from the arm64 branch means an arm64 build with no arm64 bucket configured now publishes nothing. That is the intended default, but a silent no-op is exactly the failure mode that let an architecture mismatch reach a running deployment, so emit a ::warning:: when a build resolves no bucket at all. Flip the remaining local arm64 defaults so the whole repository agrees on amd64: docker-to-rootfs.sh's own TARGET_PLATFORM fallback and its usage comment. build-images.sh always passes the platform explicitly, so this only affects direct invocation. Correct README.md and AGENTS.md, which both claimed the repository publishes ARM64 and amd64. It builds both, but publishes amd64 by default and ARM64 only when the *_ARM64 bucket variables are set. Refs artemnikitin/firework-deployment-example#13 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The GCP credentials step was already gated on the matrix architecture having a configured bucket, but the two upload steps were not. With arm64 publishing now opt-in, that left the arm64 leg running an "Upload images to S3" step on every main build, receiving AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for an upload that can only no-op, and an "Upload images to GCS" step that had never authenticated. Gate all three steps the same way. The inner empty-bucket guards stay as defence in depth. Refs artemnikitin/firework-deployment-example#13 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
push-images.sh inferred its backend from whichever bucket variable was set and checked GCS first. Until now exactly one was ever set per matrix leg, so the precedence never mattered. Making the amd64 build resolve both an S3 and a GCS bucket exposed it: `make push-s3` saw GCS_IMAGES_BUCKET in the environment and ran `gcloud storage cp` against the GCS bucket, before the GCP credentials step had even run. The S3 bucket would have been left untouched, so the preceding commits would not actually have fixed the architecture mismatch. Give push-images.sh an explicit `s3`/`gcs` backend argument and have the push-s3 and push-gcs targets pass it. Inference is kept for a bare `make push` when exactly one bucket is set, and is now an error when both are, so a publish cannot silently go to the wrong object store. Add scripts/test-push-images.sh, which runs the real Makefile targets with aws and gcloud stubbed and asserts the backend for both-buckets-set, single-bucket, and missing-bucket cases. It fails against the previous implementation. Wired into the validate-config job. Reported by codex in review of 8b7716c. Refs artemnikitin/firework-deployment-example#13 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
artemnikitin
marked this pull request as ready for review
August 7, 2026 18:41
artemnikitin
added a commit
that referenced
this pull request
Aug 14, 2026
Publishes both architectures to **one bucket per cloud**, under an `<arch>/` key prefix, instead of a separate bucket per architecture. ```text <images-bucket>/ amd64/tenant-1-kibana-rootfs.ext4 arm64/tenant-1-kibana-rootfs.ext4 ``` The prefix uses the Go vocabulary (`amd64`, `arm64`) that `TARGET_PLATFORM` already carries — deliberately not the AWS `x86_64` spelling, which the agent would never look under. An unrecognised `TARGET_PLATFORM` is rejected rather than published without a prefix. ## What changed - `scripts/push-images.sh` — derives `TARGET_ARCH` from `TARGET_PLATFORM` and prefixes the destination key - `Makefile` — threads `TARGET_PLATFORM` through the `push-*` targets, and drops its own default so an unset push fails instead of assuming amd64 - `.github/workflows/build-images.yaml` — the per-arch bucket resolution step is gone; both legs upload unconditionally to `S3_IMAGES_BUCKET` / `GCS_IMAGES_BUCKET` - `scripts/test-push-images.sh` — asserts prefixed destinations, arm64-prefix, bad-platform, unset-platform, and that the workflow passes the **matrix expression** (not merely the key) to both upload steps (13 checks, all passing) - `docs/ci-pipeline.md`, `AGENTS.md` — rewritten bucket semantics plus a migration runbook ## This reverts much of #28 `S3_IMAGES_BUCKET_ARM64` / `_AMD64` and the "no bucket configured" `::warning` are removed. Two comments went with them because they became false — the arm64 leg now *does* get AWS credentials and *does* upload, and "publishing an architecture nobody consumes" is no longer a reachable state. That mechanism existed because both architectures produced identical object names, making one bucket unsafe. Putting the architecture in the key removes the reason for it. ## Why this is a correctness fix Under per-arch buckets, pointing a node at the wrong bucket gave a silent wrong-architecture boot — a guest kernel panic at init, which is how [the Aug 7 mismatch](https://github.com/artemnikitin/firework-deployment-example) presented. With the architecture in the key, a node can only address its own images. **Scoped after review:** a missing image fails loudly only on a node with no local copy — the agent falls back to a cached image at debug level, so an existing node stays quiet. So verify bucket contents directly rather than waiting for a node to report a gap. A freshly provisioned node has no cache and does fail loudly. It also makes a mixed-architecture fleet free: node configs carry no architecture, so one desired state serves both. ## Deploy order No live environments, so there is no migration and merge order is unconstrained. The only ordering that matters operationally: **publish images before deploying nodes**, because a fresh node has no image cache to fall back on. Companions: artemnikitin/firework#45 (agent resolves the prefix) and artemnikitin/firework-deployment-example#22 (node bootstrap picks its own prefix — a functional fix, found in review). **Storage impact:** the arm64 leg starts uploading for the first time, so images-bucket usage roughly doubles. CI IAM is bucket-wide on both clouds, so no policy change is needed. ## Restored safeguard (review finding) Collapsing the per-arch bucket resolution also removed the warning that fired when a publishing run had no bucket configured — leaving builds green while uploading nothing. A publishing run now **fails** when neither bucket variable is set, and warns per provider when only one is. ## Migration runbook removed (review finding) It had four defects that only mattered if followed: no step to repoint nodes off a separate ARM64 bucket, verification covering only amd64 and only S3, a requirement that every flat object have a prefixed counterpart while acknowledging some are intentionally obsolete, and a listing that parsed `PRE <prefix>/` rows into empty entries — making the pre-deletion "must be empty" check unreliable. With no deployments on the old layout, it's replaced by a short note on deleting inert root-level objects, using `list-objects-v2 --delimiter /` so those empty entries cannot occur. ## The failure mode this guards against Both architectures now publish to one bucket under **identical object names**. If an upload step loses its `TARGET_PLATFORM` env, the arm64 leg publishes under `amd64/` and overwrites the amd64 images — silently, until a node boots a guest built for the wrong architecture. The backend cases run the Makefile directly and never see the workflow, so that wiring had no coverage. Review caught that the first version of the guard only checked the *key* was present — hardcoding `TARGET_PLATFORM: linux/amd64` in all three steps passed the entire suite. It now matches `${{ matrix.target_platform }}`, verified against exactly that bypass. **A second footgun, same shape:** `make build-arm64` sets `TARGET_PLATFORM` inside its own recipe while `push-s3` read it from the make environment, so `make build-arm64 && make push-s3` silently published arm64 images under `amd64/`. `push-images.sh` now requires an explicit `TARGET_PLATFORM`; `make build` is unchanged because `build-images.sh` defaults on its own. ## Testing `bash scripts/test-push-images.sh` — 13/13 pass. `shellcheck` clean on both scripts. Workflow YAML parses and the step list is as intended. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Switches AWS image publishing to x86_64 and brings the whole repository in line with it.
Refs artemnikitin/firework-deployment-example#13 and #14.
Why
firework-deployment-exampleswitched the AWS data plane default from bare-metal Graviton (c6g.metal) to x86_64 instances using nested virtualization. GCP has always been x86_64. Both providers therefore consume amd64 rootfs images.CI already built amd64. What it did not do was publish it to S3: the amd64 build resolved its S3 target from
S3_IMAGES_BUCKET_AMD64only, that variable is not configured, so the amd64 build silently loggedSkipping S3 upload; no bucket configuredand exited 0. The only images ever uploaded to S3 were arm64.That combination is live right now and it takes the whole AWS deployment down. An x86_64 node running an arm64 rootfs boots its kernel and mounts the root filesystem fine, then panics executing init:
-8isENOEXEC. Firecracker then exitsexit_code=0, so nothing on the host looks failed — nodes, agent, Traefik and ALB targets are all healthy — and it surfaces only as every tenant service failing its health check.What changed
Bucket resolution (
.github/workflows/build-images.yaml) — the legacyS3_IMAGES_BUCKETnow means the amd64 S3 bucket, mirroring howGCS_IMAGES_BUCKETalready means the amd64 GCS bucket. The amd64 build falls back to it; arm64 publishing becomes opt-in viaS3_IMAGES_BUCKET_ARM64.With the repository variables exactly as they are today:
artemnikitin-firework-imagesfirework-imagesThe amd64 images land in the existing bucket under the same object names, replacing the arm64 objects in place. No new bucket, no node IAM change, no
s3_images_bucket_idrepoint.Explicit push backend —
push-images.shinferred its backend from whichever bucket variable was set, checking GCS first. Until now exactly one was ever set per matrix leg, so the precedence never mattered. Resolving both buckets for the amd64 build exposed it:make push-s3would have rungcloud storage cpagainst the GCS bucket and left S3 untouched, so the change above would not actually have fixed anything. The script now takes an explicits3/gcsargument; inference errors out when both buckets are set. Caught by codex in review — thank you.Regression check —
scripts/test-push-images.shruns the real Makefile targets withawsandgcloudstubbed and asserts the chosen backend for both-buckets-set, single-bucket, and missing-bucket cases. It fails against the previous implementation. Runs invalidate-config.Loud no-op — a build that resolves no bucket at all now emits a
::warning::. A silent skip is precisely what let this mismatch reach a running deployment.Architecture-gated publish steps — the GCP credentials step was already gated on the matrix arch having a configured bucket; the two upload steps were not. The arm64 leg was running an
Upload images to S3step on every main build and receivingAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEYfor an upload that could only no-op. All three steps are now gated identically.Local defaults —
TARGET_PLATFORMflipped tolinux/amd64inMakefile,scripts/build-images.sh, andscripts/docker-to-rootfs.sh. Local pushes bypass the workflow's arch resolution, so leaving these at arm64 would letmake build && make push-s3push arm64 images into what is now the amd64 bucket and reproduce the outage.Docs —
docs/ci-pipeline.mdrewritten for the new semantics with a resolution table, the backend contract, and an explicit migration note thatS3_IMAGES_BUCKETchanged meaning.README.mdandAGENTS.mdcorrected: they claimed the repo publishes both architectures, when it builds both and publishes amd64 by default.Trade-off
Anyone still on Graviton nodes must set
S3_IMAGES_BUCKET_ARM64and points3_images_bucket_idat it. Documented, and the build now warns rather than skipping silently. This is the right default because both provider data planes are x86_64.Validation
shellcheck scripts/*.shcleanbash scripts/test-push-images.shpasses, and fails against the pre-fix scriptMerging is enough to republish: this PR touches four
GLOBAL_PIPELINE_PATHSentries, which forces a full rebuild of every tenant image rather than letting change detection skip them.Not yet done: the images in S3 are still arm64 until this merges and the build workflow runs on
main.🤖 Generated with Claude Code