A conceptual overview of the toolchain for building machine images and provisioning isolated, disposable VMs on AWS on demand. This describes what each tool does and how they fit together — not any particular implementation.
The goal it serves: spin up a fresh, pre-configured VM when needed, let it be used, then tear it down completely — repeatably and programmatically.
Four tools, each with a primary responsibility. Keeping them roughly separate keeps the system understandable — but the lines can blur where it's pragmatic.
| Tool | Phase | Responsibility |
|---|---|---|
| Packer | build (occasional) | Bake a machine image (AMI) with everything pre-installed |
| Ansible | build (inside Packer) | Configure the image idempotently via reusable roles |
| Terraform | deploy (per VM) | Create and destroy instances from an image; owns their state |
| boto3 | deploy + runtime | Wait for readiness; query and monitor; by convention doesn't create resources |
Mental model: Packer and Ansible build an image once. Terraform stamps out many instances from it. boto3 watches them. These are defaults that keep the system legible — bend them where a pragmatic shortcut clearly wins.
Packer automates building a custom AMI. A build launches a temporary instance from a base OS image, hands it off to a provisioner to install and configure software, snapshots the result into an AMI, and terminates the temporary instance. The output is an AMI ID.
Packer runs occasionally — only when the contents of the image change. It is not idempotent: each build produces a brand-new image. A manifest post-processor can record the resulting AMI ID to a file, which is the clean way for downstream tooling to discover "the latest image" without copy-pasting IDs.
Rather than scripting image setup with raw shell commands, Packer can delegate configuration to Ansible. Ansible runs a playbook of roles against the builder instance during the Packer build.
Why reach for it over inline shell:
- Idempotent — re-running converges to the same state, so iterating on the config is safe.
- Reusable roles — a base role (OS updates, hardening, common tooling) can be shared across every image; more specific roles layer on top.
- Testable independently — the same playbook can run against a throwaway live instance during development, so configuration is debugged without paying the full image-bake cycle each time.
Terraform declares the desired infrastructure (an instance, its networking, its
tags) as code and reconciles reality to match. It creates resources on apply
and removes them on destroy.
Its defining feature is state: Terraform records what it created in a state file, and that record is what lets it cleanly destroy exactly what it made later. Without state, teardown is guesswork. Terraform takes the image ID as an input — it does not care where the image came from.
Terraform runs per VM, every time one is needed.
boto3 talks to the AWS API directly from Python. In this workflow it is primarily read-and-wait — it generally doesn't create or destroy instances. Letting Terraform own create/destroy keeps state authoritative, but that's a convention, not a hard rule; one-off glue can bend it. Its two main uses:
- Waiters — block until an instance reaches a desired condition. Critically, wait for the health-check-passing state, not merely running: "running" means the VM was started; "status OK" means the OS is actually reachable.
- Status queries — fetch the live state, IP, and metadata of an instance for monitoring during its lifetime.
BUILD TIME — run when the image contents change
packer build
├─ launch a temporary builder instance
├─ Ansible playbook configures it
├─ snapshot → AMI
├─ terminate the builder
└─ manifest records the new AMI ID
DEPLOY TIME — run whenever a fresh VM is needed
├─ discover the current AMI ID (from the manifest)
├─ terraform apply (AMI ID passed in as an input variable)
├─ boto3 waiter blocks until the instance is health-check-ready
└─ hand back the instance ID and IP
TEARDOWN — run when the VM is no longer needed
├─ terraform destroy (removes the instance; state knows exactly what to remove)
└─ image cleanup is separate — see below
The manifest is the handoff between build and deploy: the build phase writes the AMI ID, the deploy phase reads it. This keeps the deploy configuration generic — it just consumes "an image ID."
-
Build once, deploy many. Baking software into an image (minutes, occasional) is separated from launching instances (seconds, frequent). Don't install software at boot if it can be baked into the image.
-
State matters. Terraform's ability to tear down cleanly depends on its state record. Creating via Terraform and destroying via a side channel tends to orphan resources — so keep create/destroy on one side unless you have a reason not to.
-
Running ≠ ready. An instance reports "running" before its OS is reachable. Always wait on the health-check state before treating a VM as usable.
-
Isolate concurrent deployments. Each independent VM needs its own Terraform state — distinct state per instance so that creating or destroying one never disturbs another. This is what makes parallel provisioning safe.
-
Tag everything. Consistent tags (project, owner, purpose) are how resources are later found, filtered, and — when something escapes automation — cleaned up by hand.
Different resources are cleaned up by different owners:
- Instances are removed by
terraform destroy— automatic, driven by state. - Images and their snapshots are not managed by Terraform. A built AMI and its backing disk snapshot are separate resources that must be deregistered and deleted explicitly. Because image builds accumulate, a routine that prunes old images (keeping the most recent) prevents creeping storage cost.
- Idle instances cost money while running. A reaper that terminates VMs past a time-to-live is worth having once provisioning is automated.
Things commonly deferred while prototyping — worth weighing before real load, but tradeoffs to make deliberately, not prerequisites to gate on:
- Remote, locked state. Local Terraform state doesn't survive a restart or scale to multiple workers. Production keeps state in a shared backend with locking so concurrent operations are safe.
- Least-privilege access. Prefer a narrowly-scoped role over a broad, long-lived credential. Grant only the actions actually used, on only the resources actually touched.
- Access to the VM. Decide how users authenticate to a provisioned instance (injected key pairs, a managed session service, etc.) and whether any port is exposed at all.
- Networking. Deliberate choices around subnets, public vs private addressing, and firewall rules scoped to expected sources.
- Secrets. No long-lived credentials in source or environment files; use a role or a secrets manager.
- Async provisioning. Readiness waits take a minute or more, so provisioning belongs off any request's critical path — behind a background job or queue.
- An AWS account, with credentials available to the local toolchain (kept out of source control).
- The CLIs on PATH: Packer, Ansible, Terraform, and the AWS CLI.
- A successful image build (so an AMI and its manifest exist) before the first provision.