Repository navigation
feat(deploy): one-command deploy from a reset lab, automatic state recovery, honest make down - #16
Merged
Merged
Conversation
… reset lab `make deploy` now runs scripts/bootstrap.sh: preflight, Terraform state, the permanent layer, DNS delegation + certificate, the ECR image mirrors, the platform, the application and a smoke test. On a running platform it is a redeploy. `make redeploy` keeps the old app-only behaviour (scripts/deploy.sh, what CI runs). - scripts/delegate-dns.sh: makes the Cloudflare NS records for the delegated subdomain match the new Route 53 zone (a lab reset changes its name servers). With CLOUDFLARE_API_TOKEN it uses the API (token sent on stdin, never printed); otherwise it prints the 4 names and waits on a terminal, or exits 3. Tested against a local mock (scripts/delegate-dns.test.sh). - bootstrap trusts the account, not Terraform's state, for "is the platform up"; recreates the uploads bucket (a local-exec that never re-runs) when the state remembers a bucket that is gone; runs sync-secrets only on a fresh clone (it deletes the tfvars of groups .env leaves empty, which would delete those Secrets Manager secrets). - docs: README "after a lab reset", CLAUDE.md "After a lab reset", aws.env.example gets the optional CLOUDFLARE_API_TOKEN. Verified: the delegation tests, bash -n, and a full run of `make deploy` against the live platform (the already-up path). The from-zero path as a whole needs a real lab reset. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…down no longer lies The lab cut the credentials half way through `make down` (explicit deny voc-cancel-cred). Terraform saved errored.tfstate and left a stale state lock, and teardown.sh still exited 0 because it never checked the apply and printed the AccessDenied errors of its own verification as if they were lists. - scripts/recover-state.sh, run by tf-init.sh (make up, make deploy) and by make down: removes a stale lock (no terraform process here, older than 5 min) and pushes errored.tfstate when it is this account's and newer than the remote state of the same lineage; archives it (*.foreign-, *.stale-) otherwise, never deletes; a lineage mismatch or a failed push stops with exit 1. - teardown.sh: runs tf-init (so the recovery) first; exits non-zero unless the apply succeeded and every list is empty. lib.sh verify_empty reports a list it cannot read as UNVERIFIED, never as empty. - tests with stand-ins for aws, terraform and Cloudflare, run in CI: recover-state (11 cases), teardown (7 checks; the old script exits 0 under the same failure, checked), delegate-dns. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflicts resolved by keeping both sides: - scripts/lib.sh: main's payment release gate and the teardown verify_empty are independent blocks. - .github/workflows/ci.yml: main's release and secret-scan steps plus the session-script tests. - CLAUDE.md: the "After a lab reset" section replaces the old manual blocker 4. README.md (rewritten on main) is updated for the production layout: overview diagram and embedded as-built image, component table (edge/egress node groups, RDS Proxy, ElastiCache, ALB + WAF, NAT, endpoints, CloudWatch), Terraform layout, the one-command session lifecycle and the recovery after an interrupted make down, the lab limits, the rules; the duplicate Deploy section is removed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
suwizx
pushed a commit
to suwizx/Cloud_Computing2026-Project
that referenced
this pull request
Oct 27, 2026
…on/off The same tests as the 2026-10-25 run, on the layout merged in PR Vnntnn#15/Vnntnn#16 (ElastiCache, RDS Proxy, ALB + WAF, private tiers), measured 2026-10-26: - lifecycle timings: make deploy from main took 32 min 45 s (per step and per resource), the old layout's make down 12 min 21 s, Redis off/on 1 min 39 s / 22 s; - Redis on vs off, in the cluster at 200 req/s: event detail p50 2.4 ms vs 7.3 ms and a ~33 s stall with 6.4% failures when the pods start without Redis; the session path fails 100% (registration goes down); the Postgres-only control did not move; - all services through the public host: 87,725 requests, 0 failed, p95 376 ms overall; - what is not comparable with the earlier results, one unexplained tail, and what was not measured. RESULTS.md now points to the new file instead of saying the layout was untested. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After a lab reset the only steps left are pasting the new credentials into
infra/aws.envandmake deploy. Also fixes the failure mode seen when the lab cut the credentials half way throughmake down.What changes
make deploy=scripts/bootstrap.sh: preflight → Terraform state → permanent layer → DNS delegation + wait for the ACM certificate → ECR image mirrors → platform → application (scripts/deploy.sh, so the payment release gate still applies) → smoke test. On a running platform it is a redeploy.make redeploykeeps the old app-only behaviour (what CI runs).scripts/delegate-dns.sh: a reset replaces the Route 53 zone, so the Cloudflare NS records go stale. With the optionalCLOUDFLARE_API_TOKENininfra/aws.envit replaces them through the API (token sent on stdin, never printed); without it, it prints the 4 names and waits on a terminal, or exits 3.scripts/recover-state.sh(run bytf-init.sh, somake up/make deploy, and bymake down): removes a stale state lock (no terraform process here, older than 5 min) and pusheserrored.tfstatewhen it is this account's and newer than the remote state of the same lineage; archives it otherwise, never deletes; a lineage mismatch or failed push stops with exit 1.make downno longer lies: it exited 0 after the lab cut the credentials mid-destroy (it never checked the apply, and printed the AccessDenied errors of its own verification as if they were lists). It now exits non-zero unless the apply succeeded and every list is empty; a list it cannot read isUNVERIFIED, never empty.CLAUDE.mdgets "After a lab reset".Tests (all run in CI, no AWS / network)
recover-state.test.sh(11 cases),teardown.test.sh(7 checks; the old script exits 0 under the same failure, verified),delegate-dns.test.sh(10 checks against a local Cloudflare mock). Also green on the merged tree:lib.test.sh,release.test.sh(25),scan-secretsand its test,terraform validate,helm templatefor AWS and k3d.Not verified
The from-zero path as a whole needs a real lab reset. Not exercised against real S3: the lock removal and the state push (the credentials were dead when this was written); the logic is covered by the stand-in tests. Not tried against real Cloudflare: no token yet.
Notes
scripts/lib.sh,ci.yml,CLAUDE.md).make down). The next session runsmake downfrom a checkout of this branch: itstf-initrecoverserrored.tfstateautomatically.🤖 Generated with Claude Code