Stop prefetching rootfs images during node bootstrap

iam-policies/aws/README.md

@@ -4,7 +4,7 @@ Replace `IMAGES_BUCKET_NAME` and `ACCOUNT_ID` in the JSON examples before use.
 
 | File | Principal | Purpose |
 |---|---|---|
-| `01-ci-images-s3.json` | GitHub Actions CI | Upload ARM64 rootfs images to S3 |
+| `01-ci-images-s3.json` | GitHub Actions CI | Upload rootfs images for every architecture to S3 |
 | `02-terraform-deploy.json` | Terraform operator | Deploy AWS control and data planes |
 | `03-packer-build.json` | Packer operator | Build the ARM64 AMI |
 

iam-policies/gcp/01-ci-images-gcs.md

@@ -1,7 +1,9 @@
 # CI image uploader
 
 Grant the GitHub Actions service account `roles/storage.objectAdmin` on the
-amd64 images bucket only. Do not grant the role at project level.
+images bucket only. Do not grant the role at project level. One bucket holds
+every architecture under an `<arch>/` key prefix, so a single bucket-scoped
+binding covers all of them.
 
 Use Workload Identity Federation and grant the repository principal
 `roles/iam.workloadIdentityUser` on that service account. No JSON service
@@ -12,7 +14,7 @@ account key is required.
 Bucket-level upload access (note: bucket-scoped, not project-scoped):
 
 ```bash
-gcloud storage buckets add-iam-policy-binding gs://AMD64_IMAGES_BUCKET \
+gcloud storage buckets add-iam-policy-binding gs://IMAGES_BUCKET \
   --member="serviceAccount:CI_SA@PROJECT_ID.iam.gserviceaccount.com" \
   --role="roles/storage.objectAdmin"
 ```
@@ -28,7 +30,7 @@ gcloud iam service-accounts add-iam-policy-binding \
 
 ## Granting via Console
 
-- Bucket access: **Cloud Storage → Buckets →** select the amd64 images bucket
+- Bucket access: **Cloud Storage → Buckets →** select the images bucket
   **→ Permissions → Grant access**, add the CI service account, role
   **Storage Object Admin**.
 - WIF binding: **IAM & Admin → Service Accounts →** select the CI service

packer/aws/README.md

@@ -18,8 +18,10 @@ The architecture must be consistent across three places:
 | `node_ami_architecture` and `node_instance_type` | data-plane stack |
 | rootfs image set | S3 images bucket (`firework-gitops-example` builds both) |
 
-Host and guest architecture must match, so pointing an x86_64 node at an arm64
-rootfs bucket fails at microVM start, not at deploy time.
+Host and guest architecture must match. One bucket holds both, and the agent
+resolves the `<arch>/` prefix from its own architecture, so a node cannot be
+pointed at the wrong rootfs set. An architecture that has not been published yet
+fails at image sync naming the missing key.
 
 All provisioning scripts in `packer/scripts/` resolve their downloads from
 `uname -m`, so they need no changes between architectures.

terraform/data-plane/aws/README.md

@@ -182,7 +182,7 @@ performance sensitive or have strict latency requirements, so **do not use this
 stack to benchmark microVM boot latency.**
 
 To go back to bare-metal Graviton nodes, three settings must change together,
-plus the AMI and the image bucket:
+plus the AMI:
 
 ```hcl
 node_instance_type         = "c6g.metal"
@@ -191,9 +191,12 @@ node_ami_architecture      = "arm64"
 ```
 
 and rebuild the AMI with `architecture = "arm64"` (see
-[packer/aws/README.md](../../../packer/aws/README.md)) and point
-`s3_images_bucket_id` at the arm64 rootfs bucket. Host and guest architecture
-must match; a mismatch fails at microVM start, not at deploy time.
+[packer/aws/README.md](../../../packer/aws/README.md)). `s3_images_bucket_id`
+stays as it is: the bucket holds every architecture under an `<arch>/` key
+prefix, and the agent resolves the prefix from its own binary's architecture, so
+host and guest architecture cannot diverge. If the arm64 images have not been
+published yet, the node fails at image sync naming the missing key, rather than
+at microVM start.
 
 **Delete or rebuild `packer/aws/manifest.json` when switching architecture.**
 `use_packer_manifest_ami` defaults to `true`, so a stale manifest left over from
@@ -239,8 +242,9 @@ node_ami_architecture      = "arm64"
 That reproduces the previous topology exactly, so the upgrade is a no-op. Change
 them one at a time when you are ready, and expect nodes to be replaced.
 
-Switching architecture additionally requires rebuilding the AMI and repointing
-`s3_images_bucket_id` at a matching rootfs image set, as described above.
+Switching architecture additionally requires rebuilding the AMI, as described
+above. `s3_images_bucket_id` does not change: one bucket holds every
+architecture.
 
 #### Subnet keys changed from index to availability zone
 

terraform/data-plane/aws/templates/user-data.sh.tpl

@@ -316,13 +316,14 @@ if [ -x /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl ]; then
     -s || true
 fi
 
-# --- 1. Download rootfs images from S3 ---
-echo "==> Downloading rootfs images from s3://$S3_IMAGES_BUCKET/"
-retry "download rootfs images" 20 5 \
-  aws s3 sync "s3://$S3_IMAGES_BUCKET/" "$IMAGES_DIR/" \
-    --region "$S3_REGION" \
-    --exclude "*" --include "*.ext4"
-echo "==> Images downloaded"
+# --- 1. Rootfs images ---
+# Not prefetched here. The agent syncs images itself from s3_images_bucket, and
+# it is the better place to do it: it downloads only the images assigned to this
+# node rather than every image for this architecture, and it maintains the
+# write-token sidecars that let a restart skip unchanged objects.
+#
+# A prefetch here cannot write those sidecars, so every prefetched image was
+# downloaded a second time by the agent moments later.
 
 # --- 2. Write agent config ---
 # Detect the primary network interface for masquerade (agent needs this).

terraform/data-plane/aws/terraform.tfvars.example

@@ -60,9 +60,10 @@ node_key_name = "my-key-pair"
 node_instance_type         = "c8i.2xlarge"
 node_nested_virtualization = true
 
-# To use bare-metal Graviton nodes instead, all three must change together, and
-# the AMI must be rebuilt with architecture = "arm64" and s3_images_bucket_id
-# pointed at the arm64 rootfs bucket:
+# To use bare-metal Graviton nodes instead, all three must change together and
+# the AMI must be rebuilt with architecture = "arm64". s3_images_bucket_id is
+# unchanged: the bucket holds every architecture under an <arch>/ key prefix and
+# the agent reads the one matching its node.
 # node_instance_type         = "c6g.metal"
 # node_nested_virtualization = false
 # node_ami_architecture      = "arm64"

terraform/data-plane/aws/variables.tf

@@ -93,7 +93,7 @@ variable "nat_gateway_mode" {
 # --- S3 (pre-existing, managed outside this stack) ---
 
 variable "s3_images_bucket_id" {
-  description = "Name/ID of the pre-existing S3 images bucket (managed by CI, not Terraform)"
+  description = "Name/ID of the pre-existing S3 images bucket (managed by CI, not Terraform). Holds every architecture under an <arch>/ key prefix; nodes read the prefix matching their own architecture."
   type        = string
 }
 
@@ -148,7 +148,7 @@ variable "node_ami_owners" {
 }
 
 variable "node_ami_architecture" {
-  description = "Architecture filter used when resolving AMI by name pattern. Must match the architecture the node AMI was built for and the rootfs images in s3_images_bucket_id."
+  description = "Architecture filter used when resolving AMI by name pattern. Must match the architecture the node AMI was built for. The images bucket holds every architecture, so nodes resolve their rootfs images automatically and this does not need to be matched against a bucket."
   type        = string
   default     = "x86_64"
 

terraform/data-plane/gcp/templates/startup-script.sh.tpl

@@ -96,7 +96,16 @@ fi
 read_secret "${registry_ca_secret}" > /etc/firework/pki/node-ca.crt
 REGISTRY_BOOTSTRAP_TOKEN=$(read_secret "${registry_bootstrap_secret}")
 
-gcloud storage cp --no-clobber "gs://${gcs_images_bucket}/*.ext4" /var/lib/images/
+# Rootfs images are not prefetched here. The agent syncs them itself from
+# gcs_images_bucket, and it is the better place to do it: it downloads only the
+# images assigned to this node rather than every image for this architecture,
+# and it maintains the write-token sidecars that let a restart skip unchanged
+# objects.
+#
+# A prefetch here cannot write those sidecars, so every prefetched image was
+# downloaded a second time by the agent moments later. It was also a startup
+# hazard: under `set -e`, a copy that matched nothing aborted this script before
+# the agent config below was written, leaving a node that never enrolled.
 
 cat > /etc/firework/agent.yaml <<EOF
 node_id: "$INSTANCE_NAME"

terraform/data-plane/gcp/terraform.tfvars.example

@@ -8,7 +8,7 @@ dns_zone_name = "firework-gcp"
 # Single source of truth for wildcard DNS, the wildcard TLS certificate, and the
 # agent ingress_domain. Services route as <subdomain>.<base_domain>.
 base_domain       = "gcp.example.com"
-images_bucket_name = "firework-images-amd64-my-gcp-project"
+images_bucket_name = "firework-images-my-gcp-project"
 
 # Default false: Private Google Access and registry peering are enough for the
 # demo. Enable only when guest workloads or node debugging need public egress.

terraform/data-plane/gcp/variables.tf

@@ -226,7 +226,7 @@ variable "config_prefix" {
 
 variable "images_bucket_name" {
   type        = string
-  description = "Globally unique amd64 images bucket name (created by the images-infra stack)"
+  description = "Globally unique images bucket name (created by the images-infra stack). Holds every architecture under an <arch>/ key prefix."
 }
 
 variable "registry_url" {

terraform/images-infra/gcp/variables.tf

@@ -11,7 +11,7 @@ variable "gcp_region" {
 
 variable "images_bucket_name" {
   type        = string
-  description = "Globally unique name for the amd64 node images bucket. Used by Packer to upload images and by data-plane nodes to rsync images at boot."
+  description = "Globally unique name for the node images bucket. Holds every architecture under an <arch>/ key prefix; nodes read the prefix matching their own architecture. Used by Packer to upload images and by data-plane nodes to rsync images at boot."
 }
 
 variable "packer_service_account" {