Skip to content

Root volume and the seal share one KMS key, so destroy can strand every snapshot #55

Description

@sethbergman

The last of the four findings from the terraform/ review. Left open
deliberately at the time because, unlike the other three, it is a design
question rather than a defect with an obvious fix.

What is shared

terraform/aws/compute.tf encrypts the root EBS volume with the
auto-unseal key:

kms_key_id = aws_kms_key.vault_autounseal.arn

terraform/aws/storage.tf uses the same key for the snapshot bucket's
SSE-KMS default encryption, and warns about exactly this:

a snapshot is encrypted under the auto-unseal KMS key, so this bucket
on its own is not a recoverable backup. The KMS key has to survive too,
and must not be scheduled for deletion when a cluster is torn down.

Why that warning is not enough

terraform destroy schedules the key for deletion. deletion_window_in_days
is 7, so after a week every archived snapshot becomes permanently
undecryptable — including snapshots taken from clusters that no longer
exist and were the reason for keeping the bucket.

The document says the key must not be deleted. The code deletes it. An
operator following the runbook and then tearing down a test cluster
destroys the backups of every other cluster that shared the key.

scripts/teardown-cloud.sh exists because terraform destroy fails
partway on both profiles, so this is the right place to reconcile it.

The second, quieter half

The key carries no aws_kms_key_policy, so it has the default policy,
and Auto Scaling reaches it through AWSServiceRoleForAutoScaling's
managed policy via IAM delegation. That works today. Replacing the
default policy with anything more restrictive — a plausible hardening
step — breaks instance launch with Client.InternalError, which names
neither KMS nor the policy.

Options

  1. Separate keys. One for the seal, one for data at rest. The seal
    key becomes the only thing that must outlive a teardown, and its
    lifecycle can be managed deliberately (prevent_destroy, or created
    outside the profile).
  2. Keep one key, protect it. lifecycle { prevent_destroy = true }
    plus a longer deletion window, and make teardown-cloud.sh say
    plainly what it is not removing and why.
  3. Document only, and accept that the warning in storage.tf is the
    whole mitigation. Currently what ships, and the reason this is filed.

Not reachable without an apply

Whether the ASG half actually bites depends on how AWS resolves the
service-linked role against the default key policy, which the emulated
apply cannot answer — moto does not enforce IAM. The destroy half needs
no apply to reason about and can be fixed now.


Found by the max-effort review of terraform/. The other three are
fixed in #46 and #54.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions