Test packer AMI provisioning locally in a VM (no AWS build, no snapshot wait)

[rustyrazorblade]

Problem

Every change to the packer AMI build — provisioning scripts and the .pkr.hcl provisioner block (file uploads, step ordering, cache save/restore, the new prune/fstrim) — currently can only be validated end-to-end by running a full amazon-ebs build in AWS. That is slow (~8 min, dominated by the EBS "AMI becoming ready" snapshot floor) and expensive, so tiny provisioning changes cost a full cloud build each. This has repeatedly bitten us: the arm64 update-java-alternatives bugs (#785/#786) and the inert fstrim -av (#792) all had to be discovered via AWS builds.

The existing local harness does not close this gap:

• packer/docker-compose.yml + packer/Dockerfile run only a subset of scripts per service (test-cassandra runs prepare_instance.sh + install_cassandra_easy_stress.sh only — not install_cassandra.sh, install_axon.sh, install_maac, install_pyroscope), and none of the .pkr.hcl provisioner block.
• It is amd64-only — the exact arm64 bug class (install_cassandra.sh hardcodes amd64 Java alternative — breaks ARM64/Graviton cassandra image builds #785/fix(packer): ARM64 arch fixes in cassandra provisioning #786) can never surface. The test Dockerfile even hardcodes update-java-alternatives -s java-1.11.0-openjdk-amd64 || true, masking that class of failure.
• It is a container, not a VM — no systemd, no real block device, so fstrim/snapshot behavior (the packer: prune build cruft before baking the AMI #792 story) cannot be tested at all.
• OS drift: the test image is ubuntu:26.04 while the AMI base is 24.04 noble.

Goal

Make packer AMI provisioning changes fully testable locally, without an AWS build and without waiting on any EBS snapshot — by running the same provisioner sequence the production amazon-ebs build runs, in a local VM, and asserting the result.

Owner's framing: "make sure changes to the packer ami building are done in a way that we can try them locally … run the whole thing in a virtual machine … if snapshot is the problem, then the most logical solution is to stop waiting for snapshots in testing."

Central design tension — arch matrix across Mac (dev) and CI (GitHub)

• The owner's dev machine is Apple Silicon (arm64). Native-speed local VMs there are arm64 (Apple Virtualization.framework via Tart/Lima). Testing the amd64 path on the Mac requires x86 emulation (slow).
• GitHub Actions standard hosted runners are x86_64 (amd64) — amd64 VMs run fast (KVM); arm64 needs emulation or arm64 runners.
• AMIs are baked for both amd64 and arm64.

A viable answer is likely a matrix: Mac covers arm64 natively (which would have caught #785/#786), CI covers amd64 (fast on x86 runners) — sharing one provisioner block so tests cannot drift from production. The tool choice and exact wiring is the design question.

Design questions for the architect

1. Local builder choice to run the full provisioner sequence with no snapshot: packer Tart builder (native Virtualization.framework, arm64, ships a Packer plugin) vs Lima vs packer qemu builder (cross-arch via emulation) vs extending the Docker harness. Trade-offs on fidelity (systemd? real block device for fstrim?), speed, and cross-arch.
2. How to share the provisioner block so the local source and amazon-ebs run byte-identical steps (single build block, multiple sources; -only= selection). What has to be abstracted (AMI-specific bits: source_ami_filter, run_tags, IMDS options) vs shared (all provisioner steps).
3. Stubbing cloud-only dependencies the provisioners assume — the S3 apt/tarball cache (edl-cache.sh / save_s3_cache.sh / cached_fetch) and IMDS — so a local run degrades cleanly instead of failing.
4. Arch matrix: what runs where (Mac arm64 dev loop; CI amd64, optionally arm64 on arm64 runners), and how CI wires it (the repo already has "Test … Provisioning Sequence" jobs to evolve).
5. What "assert success" means locally without an AMI — exit status of the provisioner run, plus targeted post-provision checks (systemd units present, java alternatives resolve per-arch, axon jars present, fstrim -v / reports bytes trimmed on a real block device).
6. Scope / phasing — MVP that unblocks the dev loop vs full CI matrix.

First validation target once this exists

The #792 fstrim -av → fstrim -v / correction (needs a real block device to verify) should be the first change validated on the new local VM harness instead of another AWS build.

Out of scope

• Changing the production amazon-ebs build's output or the AMI contract.
• The packer performance work in Improve packer performance #569 (separate).