Default the health check to EC2, and say when to promote it - #54
Merged
Conversation
A bare `terraform apply` of this profile never converged. The group used health_check_type = "ELB". But the profile deliberately does not issue TLS certificates -- user-data says so and defers to the Ansible layer -- and Vault will not start without them, so the load balancer's check could never pass. Every instance was therefore marked unhealthy at the end of its grace period, terminated, and replaced by one that repeated it: a loop, billing EC2, NAT and EBS throughout, and presenting as a slow bootstrap rather than a configuration gap. EC2 health only asks whether the instance is running, which is exactly enough for the window before Ansible has converged and no more. It is now the default, and health_check_type is a variable so the profile is not trapped there -- a cluster left on EC2 health forever cannot notice a node that is up but sealed or wedged, which is the reason ELB was chosen originally and is still the right setting once Vault is serving. The old test asserted the value was "ELB", which is how a defect that makes the apply loop was held in place by a passing test. It now asserts both halves: that the default is EC2, so the apply terminates, and that ELB is reachable, so the promotion is possible. Reintroducing the "ELB" default turns the first red while the second still passes. docs/deployment.md sequences the promotion after the playbook converges, next to the certificates it depends on. Verified in WSL: terraform fmt clean, 28 assertions pass across the AWS suite, and the mutation above fails exactly one of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A bare
terraform applyof this profile never converged.The loop
The group used
health_check_type = "ELB". The profile deliberately does not issue TLS certificates — user-data says so and defers to Ansible — and Vault will not start without them, so the load balancer's check could never pass.Every instance was marked unhealthy at the end of its grace period, terminated, and replaced by one that repeated it. Billing EC2, NAT and EBS throughout, and presenting as a slow bootstrap rather than a configuration gap.
The fix
health_check_typeis now a variable defaulting toEC2. EC2 health only asks whether the instance is running — exactly enough for the window before Ansible has converged, and no more.It is a variable rather than a flat change because the original reasoning was right: a cluster left on EC2 health forever cannot notice a node that is up but sealed, wedged, or out of quorum.
docs/deployment.mdnow sequences the promotion right after the certificates it depends on:The test was holding the defect in place
The old assertion was
health_check_type == "ELB"— so a value that made the apply loop forever was defended by a passing test. That is the same shape as every other bug found in this repo: a check that agrees with the thing it is supposed to question.It now asserts both halves — the default is
EC2so the apply terminates, andELBis reachable so the promotion is possible. The second block also asserts the target group still accepts a standby's 429, because promoting to ELB health is pointless if it then ejects healthy standbys.Verified
Run in WSL rather than assumed:
terraform fmt -check -recursivecleanterraform test— 28 passed, 0 failed across the AWS suiteELBfailshealth_check_defaults_to_ec2_so_a_bare_apply_terminatesand nothing else — 10 passed, 1 failed🤖 Generated with Claude Code