From 213f6f1cb962c5d8c436813d301a52bb19cbdf1d Mon Sep 17 00:00:00 2001
From: Poojithvsc <poojithrokz@gmail.com>
Date: Sat, 6 Jun 2026 01:15:33 +0200
Subject: [PATCH 1/6] docs(observability): full-profile QA checklist for 18a
 metrics + 18b logs/Kafka UI

The end-to-end checks mvn verify can't make: Grafana panels light up,
{container="shopsphere-app"} |= "<orderId>" returns every module's line, and
Kafka UI shows topics/offsets. Companion to qa-walkthrough.md (which is
dev-mode); this one runs the full compose profile.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 docs/observability-qa.md | 152 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 152 insertions(+)
 create mode 100644 docs/observability-qa.md

diff --git a/docs/observability-qa.md b/docs/observability-qa.md
new file mode 100644
index 0000000..1e9950a
--- /dev/null
+++ b/docs/observability-qa.md
@@ -0,0 +1,152 @@
+# ShopSphere — Observability QA (Phases 18a + 18b)
+
+The end-to-end checks that `mvn verify` **cannot** make: that the Grafana dashboard panels light up with real data (18a), that a Loki search for an `orderId` returns every module's log line for that order (18b), and that Kafka UI shows the topics/partitions/offsets (18b).
+
+> Unlike [`qa-walkthrough.md`](./qa-walkthrough.md) — which runs the app on the host with backing services only — this checklist runs the **`full` compose profile**: the app is containerised so promtail can tail its logs and Prometheus can scrape it. Commands assume **Windows PowerShell**.
+
+| What | URL | Credentials |
+|---|---|---|
+| App | `http://localhost:8080` | Bearer token (below) |
+| Grafana | `http://localhost:3000` | `admin` / `admin` (first-login change prompt) |
+| Prometheus | `http://localhost:9090` | — |
+| Loki (via Grafana Explore) | `http://localhost:3100` | — |
+| Kafka UI | `http://localhost:8081` | — |
+
+---
+
+## 1. Bring up the full stack
+
+From `D:\shopsphere-project\code`, with Docker Desktop running:
+
+```powershell
+docker compose --profile full up -d --build
+```
+
+This builds the app image and starts: `postgres`, `kafka`, `localstack`, `app`, `prometheus`, `grafana`, `loki`, `promtail`, `kafka-ui`.
+
+Wait for the app to be healthy (it's the long pole — it waits on Postgres + Kafka + LocalStack):
+
+```powershell
+docker compose ps
+curl.exe -s http://localhost:8080/actuator/health | ConvertFrom-Json | ConvertTo-Json -Depth 5
+```
+
+- [ ] `docker compose ps` shows all nine containers `Up`; `shopsphere-app` is `healthy`.
+- [ ] `/actuator/health` returns `"status":"UP"` with `db` and `kafka` both `UP`.
+
+> If the build fails on a buildx snapshot error (`parent snapshot … not found`), the local build cache is corrupt: `docker builder prune -af` then re-run the `up` command. (Seen during Phase 18b verify.)
+
+---
+
+## 2. Generate an order (gives you an `orderId` to trace)
+
+Run the auth + checkout portion of the main walkthrough against the containerised app. Minimal happy-path version:
+
+```powershell
+$BASE = "http://localhost:8080"
+$EMAIL = "qa+$(Get-Date -Format yyyyMMddHHmmss)@shopsphere.test"
+$PASSWORD = "password1234"
+
+# register + login
+curl.exe -s -X POST "$BASE/api/v1/auth/register" -H "Content-Type: application/json" `
+  -d "{`"email`":`"$EMAIL`",`"password`":`"$PASSWORD`"}" | Out-Null
+$login = curl.exe -s -X POST "$BASE/api/v1/auth/login" -H "Content-Type: application/json" `
+  -d "{`"email`":`"$EMAIL`",`"password`":`"$PASSWORD`"}" | ConvertFrom-Json
+$AUTH = "Authorization: Bearer $($login.accessToken)"
+
+# add the keyboard, check out with the success card
+curl.exe -s -X POST "$BASE/api/v1/cart/items" -H $AUTH -H "Content-Type: application/json" `
+  -d '{"productId":"11111111-1111-1111-1111-111111111111","qty":1}' | Out-Null
+$order = curl.exe -s -X POST "$BASE/api/v1/orders" -H $AUTH -H "Content-Type: application/json" `
+  -d '{"shippingAddress":"1 Test Street, Bengaluru","cardNumber":"4242424242424242"}' | ConvertFrom-Json
+$ORDER_ID = $order.orderId
+"orderId = $ORDER_ID"
+
+# poll to PAID
+for ($i=1; $i -le 10; $i++) {
+  $o = curl.exe -s "$BASE/api/v1/orders/$ORDER_ID" -H $AUTH | ConvertFrom-Json
+  "$($i): $($o.status)"; if ($o.status -ne "PENDING_PAYMENT") { break }; Start-Sleep -Milliseconds 400
+}
+```
+
+- [ ] Checkout returns `202` + a `orderId`. **Copy the `orderId`** — you need it in §4.
+- [ ] The order reaches **`PAID`** within a few polls.
+
+> For a richer dashboard/log picture, also run the **declined** (`4000000000000002`) and **insufficient-funds** (`4000000000009995`) cards from `qa-walkthrough.md` §11–12. That gives you `payments_total{outcome=...}` and `reservations_total{status=RELEASED}` dimensions to see.
+
+---
+
+## 3. Phase 18a — metrics on the Grafana dashboard
+
+1. Open `http://localhost:3000`, log in `admin`/`admin` (skip or set a new password at the prompt).
+2. **Dashboards → ShopSphere Overview** (auto-provisioned).
+
+- [ ] **orders_placed_total** panel shows a non-zero count matching the number of checkouts you ran.
+- [ ] **payments_total** shows data, split by `outcome` (run declined/insufficient cards to see more than one series).
+- [ ] **reservations_total** shows `held` / `confirmed` (and `released` if you ran a failing card).
+- [ ] **checkout_latency_seconds** shows histogram data (count > 0).
+
+Sanity-check the source if a panel is empty:
+
+```powershell
+# Prometheus is scraping the app?  → "up" should be 1 for job "shopsphere"
+curl.exe -s "http://localhost:9090/api/v1/query?query=up" | ConvertFrom-Json | ConvertTo-Json -Depth 6
+```
+
+- [ ] Prometheus `up{job="shopsphere"}` is `1`. (If `0`, the app container isn't scrapeable — check `docker compose logs prometheus`.)
+
+---
+
+## 4. Phase 18b — search logs by `orderId` in Loki
+
+1. In Grafana: **Explore** (compass icon) → datasource dropdown → **Loki**.
+2. Query (paste your real `orderId`):
+
+   ```
+   {container="shopsphere-app"} |= "<ORDER_ID>"
+   ```
+
+3. Set the time range to **Last 15 minutes** and run.
+
+- [ ] Results include lines from **multiple modules** for that one order. For a PAID order you should see (logger in each line):
+  - [ ] **Ordering** — `Order placed with N line(s) …` (`OrderPlacement`)
+  - [ ] **Reservation** — `Reservation granted for N item(s)` and later `Reservation confirmed for N item(s)` (`CatalogImpl`)
+  - [ ] **Payment** — `Charge succeeded for order` (`PaymentOrderingConsumer`) and `Order marked PAID` (`PaymentEventsConsumer`)
+- [ ] Each matched line is JSON with `orderId` as a **top-level field** (not just substring in the message) — confirms the MDC stamping, not an accidental match.
+
+> Why a line filter and not a label: `orderId` is intentionally **not** a Loki label (a UUID label is unbounded cardinality). The raw JSON line is stored and `|=` substring-matched. See ADR-0018b.
+
+Quick CLI cross-check (optional), bypassing Grafana:
+
+```powershell
+$q = [uri]::EscapeDataString('{container="shopsphere-app"} |= "' + $ORDER_ID + '"')
+curl.exe -s "http://localhost:3100/loki/api/v1/query_range?query=$q" | ConvertFrom-Json `
+  | Select-Object -ExpandProperty data | Select-Object -ExpandProperty result | Measure-Object
+```
+
+- [ ] Count is > 0 (Loki has lines for that order).
+
+---
+
+## 5. Phase 18b — Kafka UI
+
+Open `http://localhost:8081`.
+
+- [ ] The **shopsphere** cluster is listed and online.
+- [ ] **Topics** include `ordering.events` and `payment.events`, each with a partition count.
+- [ ] **Consumers** lists the groups — `payment.orderplaced`, `ordering.payment-events`, `catalog.ordering-terminal-events` — each with committed offsets (lag ~0 after the order settles).
+
+---
+
+## 6. Teardown
+
+```powershell
+docker compose --profile full down        # keep the db volume
+docker compose --profile full down -v      # full clean slate (wipes pgdata)
+```
+
+---
+
+## Pass criteria
+
+All boxes in §3 (18a), §4 and §5 (18b) checked. If any panel is empty or the Loki search misses a module, that's a real gap — likely a module that didn't stamp `orderId` via `OrderLog`, or a scrape/ship misconfiguration. Don't mark the phase's manual-QA acceptance criterion done until this passes.

From 690050f9684f08b50e69b8092f34f0e273a2770e Mon Sep 17 00:00:00 2001
From: Poojithvsc <poojithrokz@gmail.com>
Date: Sat, 6 Jun 2026 01:52:02 +0200
Subject: [PATCH 2/6] build(deploy): pre-stage Phase 12 EC2 deploy
 (terraform/ec2 + compose.cloud.yml) [#58]
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

One-apply terraform/ec2 module: EC2 (AL2023, templated user-data installs Docker,
writes compose.cloud.yml, pulls poojithvsc/shopsphere:latest, brings up app+Kafka)
+ PRIVATE RDS (ingress only from the EC2 SG) + minimal IAM profile. compose.cloud.yml
runs app+Kafka against RDS, S3 dormant. ADR-0012 (self-hosted Kafka over MSK; Docker
Hub over ECR; RDS posture flip) + lab runbook. terraform validate passes; mvn verify
green. Lab-only ACs (apply, RDS-private timeout, QA over EC2, destroy) run in one
Whizlabs session — does not close #58 yet.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 compose.cloud.yml                             |  84 +++++++
 ...c2-deploy-self-hosted-kafka-rds-private.md |  34 +++
 docs/lab-runbook-ec2-https.md                 | 110 +++++++++
 terraform/ec2/.gitignore                      |   6 +
 terraform/ec2/README.md                       |  44 ++++
 terraform/ec2/main.tf                         | 218 ++++++++++++++++++
 terraform/ec2/outputs.tf                      |  24 ++
 terraform/ec2/terraform.tfvars.example        |  18 ++
 terraform/ec2/user-data.sh.tftpl              |  54 +++++
 terraform/ec2/variables.tf                    |  80 +++++++
 10 files changed, 672 insertions(+)
 create mode 100644 compose.cloud.yml
 create mode 100644 docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md
 create mode 100644 docs/lab-runbook-ec2-https.md
 create mode 100644 terraform/ec2/.gitignore
 create mode 100644 terraform/ec2/README.md
 create mode 100644 terraform/ec2/main.tf
 create mode 100644 terraform/ec2/outputs.tf
 create mode 100644 terraform/ec2/terraform.tfvars.example
 create mode 100644 terraform/ec2/user-data.sh.tftpl
 create mode 100644 terraform/ec2/variables.tf

diff --git a/compose.cloud.yml b/compose.cloud.yml
new file mode 100644
index 0000000..5174b35
--- /dev/null
+++ b/compose.cloud.yml
@@ -0,0 +1,84 @@
+# ShopSphere — cloud deploy compose (Phase 12 + Phase 20).
+#
+# This runs ON the EC2 instance (written there by Terraform user-data), NOT on your laptop. It differs
+# from the local docker-compose.yml in three deliberate ways:
+#   1. No `postgres` — the app connects to the PRIVATE RDS (DB_HOST is the RDS endpoint, injected via .env).
+#   2. No `localstack` — S3 image storage is dormant in the lab (S3_ENDPOINT blank → real-S3 resolution,
+#      never called by the core QA flow). Real-S3 wiring is deferred to own-AWS (ADR-0016).
+#   3. `app` is pulled from Docker Hub (APP_IMAGE), not built — the EC2 has no source tree.
+#
+# All ${...} below are docker-compose env interpolation, resolved from /opt/shopsphere/.env on the EC2.
+#
+# Bring up (Phase 12):           docker compose -f compose.cloud.yml up -d
+# Bring up with HTTPS (Phase 20): docker compose -f compose.cloud.yml --profile caddy up -d
+
+services:
+  kafka:
+    image: confluentinc/cp-kafka:7.6.1
+    container_name: shopsphere-kafka
+    environment:
+      KAFKA_NODE_ID: 1
+      KAFKA_PROCESS_ROLES: broker,controller
+      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
+      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+      # Only the in-cluster PLAINTEXT listener is needed — nothing off the EC2 talks to Kafka directly.
+      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
+      CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qg
+    healthcheck:
+      test: ["CMD-SHELL", "kafka-broker-api-versions --bootstrap-server localhost:29092 >/dev/null 2>&1 || exit 1"]
+      interval: 10s
+      timeout: 10s
+      retries: 15
+
+  app:
+    image: ${APP_IMAGE:-poojithvsc/shopsphere:latest}
+    container_name: shopsphere-app
+    depends_on:
+      kafka:
+        condition: service_healthy
+    environment:
+      DB_HOST: ${DB_HOST}
+      DB_PORT: ${DB_PORT:-5432}
+      DB_NAME: ${DB_NAME:-shopsphere}
+      DB_USER: ${DB_USER:-shopsphere}
+      DB_PASSWORD: ${DB_PASSWORD}
+      KAFKA_BOOTSTRAP_SERVERS: kafka:29092
+      JWT_SECRET: ${JWT_SECRET}
+      # S3 dormant: blank endpoint → SDK uses real-S3 resolution; the core QA flow never calls it.
+      S3_ENDPOINT: ""
+      AWS_REGION: ${AWS_REGION:-us-east-1}
+    ports:
+      # Direct HTTP (Phase 12). Caddy (Phase 20) also fronts this on 443.
+      - "8080:8080"
+    restart: unless-stopped
+
+  caddy:
+    # Phase 20 — HTTPS termination with a self-signed cert (`tls internal`), reverse-proxying to the
+    # app. Only started under the `caddy` profile (Terraform user-data passes --profile caddy when
+    # enable_https=true). Whizlabs ephemeral IPs preclude Let's Encrypt, so self-signed is the choice.
+    profiles: ["caddy"]
+    image: caddy:2.8
+    container_name: shopsphere-caddy
+    depends_on:
+      app:
+        condition: service_started
+    environment:
+      # user-data sets PUBLIC_IP in .env (read from IMDS) so Caddy mints an internal cert with the
+      # instance's IP in the SAN. Falls back to localhost if somehow unset.
+      CADDY_SITE_ADDRESS: ${PUBLIC_IP:-localhost}
+    volumes:
+      - ./Caddyfile:/etc/caddy/Caddyfile:ro
+      - caddy-data:/data
+    ports:
+      - "443:443"
+    restart: unless-stopped
+
+volumes:
+  caddy-data:
diff --git a/docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md b/docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md
new file mode 100644
index 0000000..26ab8a6
--- /dev/null
+++ b/docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md
@@ -0,0 +1,34 @@
+---
+status: accepted
+date: 2026-06-06
+cites: XP, PragProg, PoEAA
+---
+
+# 0012 — EC2 + self-hosted Kafka on one box; RDS goes private; Docker Hub over ECR
+
+Phase 11 stood up a *public* RDS reachable from the developer laptop — a deliberate learning step. Phase 12 is the production-posture deploy: one `terraform apply` provisions an **EC2** that runs the app + Kafka via `compose.cloud.yml` (image pulled from Docker Hub) and a **private RDS** whose only ingress is the EC2's security group. This is the first time ShopSphere runs as it would in front of a user, not on a laptop.
+
+## Self-hosted Kafka in compose on the box, not MSK / Confluent Cloud
+
+The EC2 runs Kafka as a compose service beside the app — the *same* `confluentinc/cp-kafka` image and KRaft config as local dev. **XP YAGNI + the Whizlabs-ephemeral constraint:** MSK and Confluent Cloud are managed, durable, multi-AZ brokers — everything an ephemeral 4-hour lab is not. Provisioning MSK would add a VPC/subnet/permission story, minutes of standup, and cost, to back a broker that's deleted at lab end. **PragProg dev/prod parity:** running the identical broker image locally and on the EC2 means the deploy exercises the exact Kafka the tests and dev loop use — no "works locally, breaks on MSK" surprises. The honest cost is recorded below: single-node, no HA, no durability beyond the box.
+
+## RDS flips to private — the network is the security control
+
+`publicly_accessible = false`, and the RDS security group's only ingress is the EC2's security group (not a laptop /32). The acceptance check is a *negative*: `psql -h <rds-endpoint>` from the laptop must **time out**. **PoEAA / PragProg — push the control to the boundary:** the database isn't protected by application logic or a password alone; it's unreachable off the VPC. The app reaches it because the app runs inside the trust boundary (on the EC2 whose SG is allow-listed). **XP incremental design:** Phase 11 used a public RDS to *learn* the RDS + Flyway path with the laptop as client; Phase 12 removes that affordance now that the EC2 is the client. The two postures are different scenarios, so they live as two self-contained Terraform configs (`terraform/rds/` historical, `terraform/ec2/` the deploy) rather than a migrated state — correct because the lab is throwaway and there's no shared state to preserve.
+
+## Docker Hub, not ECR
+
+The image is built locally and pushed to a public Docker Hub repo (`poojithvsc/shopsphere:latest`), pulled by EC2 user-data. **XP simplicity + the lab constraint:** ECR adds an extra Whizlabs permission scope and a registry that dies with the session; a public Docker Hub repo is a one-line `docker push` and a zero-auth pull. Phase 19 (deferred) would automate this push from CI. The push is manual in this phase and documented in the lab runbook.
+
+## Throwaway-lab posture, made explicit
+
+Default VPC (no bespoke network), t3.micro, no backups, no final snapshot, `apply_immediately`. **XP YAGNI:** a 4-hour box does not warrant private subnets + NAT or a backup plan. This is correct *only* because the lab is ephemeral and holds no real data — the same honesty as ADR-0011. The IAM instance profile is minimal (no SSM yet); ADR-0013 already documents what own-AWS would add.
+
+## Consequences
+
+`terraform apply` brings up the whole deploy; `terraform destroy` removes it. `mvn verify` is unaffected (this is infrastructure, no app code). Three honest limits, recorded so they don't surprise later:
+
+- **t3.micro is tight** — app and Kafka are two JVMs in 1 GiB. `instance_class` is a variable; bump to t3.small if the app OOMs on boot.
+- **S3 image storage is dormant in the lab** — `compose.cloud.yml` sets `S3_ENDPOINT` blank, so the SDK resolves real S3 but the core QA flow never calls it (no LocalStack on the box). Real-S3 wiring stays deferred to own-AWS (ADR-0016).
+- **Kafka is single-node, non-durable** — fine for the walkthrough, not a statement about production topology.
+- The acceptance criteria that *prove* this (app reachable, RDS-private timeout, QA over the EC2 endpoint, `terraform destroy`) are **manual lab steps** — they need a live Whizlabs session, like the other cloud phases. On graduation to own AWS the same module runs with a different `terraform.tfvars` and would add private subnets + NAT, ACM/ALB (ADR-0020), and SSM (ADR-0013). Phase 20 adds HTTPS in front of this same EC2.
diff --git a/docs/lab-runbook-ec2-https.md b/docs/lab-runbook-ec2-https.md
new file mode 100644
index 0000000..0f06cbe
--- /dev/null
+++ b/docs/lab-runbook-ec2-https.md
@@ -0,0 +1,110 @@
+# Lab Runbook — Phase 12 (EC2 deploy) + Phase 20 (HTTPS) in one Whizlabs session
+
+Both phases share the **same ephemeral EC2**, so they run **back-to-back in one 4-hour lab**. Everything below is pre-staged and committed; this sheet is just the live execution. Do it in order; the whole thing fits comfortably in the time box.
+
+> Why one session: the EC2 is destroyed at lab end. "Do Phase 12 now, Phase 20 later" would mean standing up the entire stack twice. See ADR-0012 / ADR-0020.
+
+## 0. Before the lab (on your laptop, no AWS needed)
+
+```powershell
+cd D:\shopsphere-project\code
+# Build and push the image Docker Hub (Phase 19 would automate this):
+docker build -t poojithvsc/shopsphere:latest .
+docker login
+docker push poojithvsc/shopsphere:latest
+```
+
+- [ ] `poojithvsc/shopsphere:latest` is on Docker Hub.
+
+## 1. Start the Whizlabs lab + credentials
+
+1. Launch the Whizlabs AWS sandbox; note the region (assume `us-east-1`).
+2. Export the lab credentials in your terminal (or `aws configure`):
+   ```powershell
+   $env:AWS_ACCESS_KEY_ID="..."; $env:AWS_SECRET_ACCESS_KEY="..."; $env:AWS_SESSION_TOKEN="..."
+   aws sts get-caller-identity        # sanity: returns the lab account
+   ```
+
+## 2. Configure Terraform
+
+```powershell
+cd D:\shopsphere-project\code\terraform\ec2
+copy terraform.tfvars.example terraform.tfvars
+# Edit terraform.tfvars: set my_ip_cidr (curl -s https://checkip.amazonaws.com -> add /32),
+#   db_password, jwt_secret (openssl rand -base64 48). Leave enable_https commented for now.
+```
+
+## 3. Phase 12 — apply (HTTP only)
+
+```powershell
+terraform init
+terraform apply        # type yes; ~5-8 min (RDS is the long pole)
+```
+
+Capture outputs: `ec2_public_ip`, `rds_endpoint`, `app_http_url`.
+
+- [ ] **App is up:** open `http://<ec2_public_ip>:8080/swagger-ui.html` in a browser. (If it's not up after ~3 min, SSH in or check: the app waits on Kafka health; user-data logs are in `/var/log/cloud-init-output.log`.)
+- [ ] **RDS is private (the negative test):** from your laptop,
+  ```powershell
+  psql "host=<rds-host> port=5432 dbname=shopsphere user=shopsphere"   # must TIME OUT
+  ```
+  A hang/timeout is the pass — the laptop is no longer allow-listed; only the EC2 SG can reach 5432.
+
+## 4. Phase 12 — QA walkthrough over HTTP
+
+Run `docs/qa-walkthrough.md` against the EC2, swapping the base URL:
+
+```powershell
+$BASE = "http://<ec2_public_ip>:8080"
+```
+
+- [ ] Register → login → add to cart → checkout (`4242…`) → order reaches **PAID**.
+- [ ] Declined card (`4000…0002`) → **CANCELLED**, stock released.
+- [ ] (Optional) the rest of the walkthrough (refresh-reuse detection, orders list).
+
+If all green, **Phase 12 is done.** Decide: enough time/energy for Phase 20? It's optional polish. If not, skip to §7 (destroy) — Phase 12 still ships.
+
+## 5. Phase 20 — turn on HTTPS
+
+```powershell
+terraform apply -var enable_https=true     # re-runs user-data, starts Caddy with the public IP
+```
+
+- [ ] Open `https://<ec2_public_ip>` in a browser → after accepting the self-signed-cert warning, the app loads.
+- [ ] `curl -k https://<ec2_public_ip>/actuator/health` returns `"status":"UP"`.
+
+## 6. Phase 20 — QA walkthrough over HTTPS
+
+```powershell
+$BASE = "https://<ec2_public_ip>"      # note: 443, no port suffix
+# add --insecure / -k to curl calls for the self-signed cert
+```
+
+- [ ] At least the happy-path checkout passes end-to-end over HTTPS.
+- [ ] Plain `http://<ec2_public_ip>:8080` still works (documented as open in ADR-0020).
+
+## 7. Teardown
+
+```powershell
+terraform destroy      # type yes; removes EC2 + RDS
+```
+
+- [ ] `terraform destroy` completes; nothing left running (the lab will also reap it, but destroy proves the IaC is clean).
+
+## 8. After the lab (back on your laptop)
+
+Tell me the results and I'll:
+- tick the lab-only acceptance boxes on #58 and #64,
+- finalize the Phase-12 and Phase-20 article drafts (claims now proven),
+- ship the release PR(s) `dev → main` closing #58 and #64,
+- record the lab outcome in the vault session log.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| App never comes up on :8080 | t3.micro OOM (app + Kafka in 1 GiB) | `terraform apply -var instance_class=t3.small` |
+| `terraform apply` AccessDenied | Lab creds expired / wrong scope | Re-export lab creds; `aws sts get-caller-identity` |
+| `psql` from laptop *connects* (should time out) | RDS SG still public, or you applied `terraform/rds/` not `terraform/ec2/` | Confirm you're in `terraform/ec2/`; check `aws_security_group.rds` ingress is the EC2 SG |
+| HTTPS handshake fails | Caddy didn't get the public IP | Check `/opt/shopsphere/.env` has `PUBLIC_IP=`; `docker logs shopsphere-caddy` |
+| Image pull fails on EC2 | Docker Hub repo private / typo | Ensure `poojithvsc/shopsphere:latest` is public; check `docker compose logs` |
diff --git a/terraform/ec2/.gitignore b/terraform/ec2/.gitignore
new file mode 100644
index 0000000..80ad845
--- /dev/null
+++ b/terraform/ec2/.gitignore
@@ -0,0 +1,6 @@
+# Never commit local Terraform state, provider binaries, or real tfvars.
+.terraform/
+.terraform.lock.hcl
+*.tfstate
+*.tfstate.backup
+terraform.tfvars
diff --git a/terraform/ec2/README.md b/terraform/ec2/README.md
new file mode 100644
index 0000000..c69effd
--- /dev/null
+++ b/terraform/ec2/README.md
@@ -0,0 +1,44 @@
+# Terraform — Phase 12 (EC2 deploy) + Phase 20 (HTTPS via Caddy)
+
+One `terraform apply` stands up the full cloud deploy in a Whizlabs sandbox:
+
+- an **EC2** instance (Amazon Linux 2023) that installs Docker, writes `compose.cloud.yml` + `Caddyfile`, pulls `poojithvsc/shopsphere:latest` from Docker Hub, and brings up **app + Kafka** (and **Caddy** when `enable_https = true`);
+- a **private RDS** Postgres whose only ingress is the EC2's security group.
+
+> **Relationship to `../rds/`:** the Phase-11 `rds/` module stood up a *public* RDS reachable from your laptop — a learning step. This module is the production-posture deploy and **supersedes it**. Do **not** apply both at once (you'd get two RDS instances). For Phase 12+, use this directory.
+
+## Prerequisites
+
+1. A Whizlabs AWS sandbox; credentials exported (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`) or `aws configure` done in the lab.
+2. The app image pushed to Docker Hub (Phase 12 does this manually — see the lab runbook `docs/lab-runbook-ec2-https.md`):
+   ```
+   docker build -t poojithvsc/shopsphere:latest .
+   docker login
+   docker push poojithvsc/shopsphere:latest
+   ```
+3. `cp terraform.tfvars.example terraform.tfvars` and fill in `my_ip_cidr`, `db_password`, `jwt_secret`.
+
+## Apply
+
+```
+terraform init
+terraform apply                 # Phase 12: app over HTTP at :8080
+terraform apply -var enable_https=true   # Phase 20: also start Caddy (HTTPS :443, self-signed)
+```
+
+Outputs give you `app_http_url`, `app_https_url`, `ec2_public_ip`, and `rds_endpoint`.
+
+## Acceptance checks (run in the lab)
+
+- App: open `http://<ec2_public_ip>:8080/swagger-ui.html`.
+- **RDS is private:** `psql -h <rds-endpoint> -U shopsphere` **from your laptop must time out** (only the EC2 SG can reach 5432).
+- HTTPS (Phase 20): open `https://<ec2_public_ip>` — the app loads after you accept the self-signed-cert warning.
+- Full QA walkthrough end-to-end against the EC2 (see `docs/qa-walkthrough.md`, swapping the base URL).
+
+## Teardown
+
+```
+terraform destroy
+```
+
+Throwaway-lab posture (no backups, no final snapshot, default VPC, t3.micro) is correct **only** because the lab is ephemeral and holds no real data. See ADR-0011 / ADR-0012. On graduation to your own AWS, the same module runs with a different `terraform.tfvars` and would add: private subnets + NAT, ACM/ALB for TLS (replacing self-signed Caddy — ADR-0020), and SSM for secrets (ADR-0013).
diff --git a/terraform/ec2/main.tf b/terraform/ec2/main.tf
new file mode 100644
index 0000000..1595896
--- /dev/null
+++ b/terraform/ec2/main.tf
@@ -0,0 +1,218 @@
+# ShopSphere — Phase 12: EC2 deploy + self-hosted Kafka; RDS goes private.
+#
+# This is the Phase-11 RDS module's grown-up sibling. Phase 11 stood up a *public* RDS reachable
+# from the developer laptop (a deliberate learning step). Phase 12 is the production-posture deploy:
+# one `terraform apply` provisions BOTH an EC2 instance running the app + Kafka AND a PRIVATE RDS whose
+# only ingress is the EC2's security group. The app talks to RDS over the VPC; the laptop cannot.
+#
+# Throwaway-lab posture (see ADR-0011/0012): default VPC, no backups, no final snapshot, t3.micro.
+# Correct ONLY because the Whizlabs lab is ephemeral (4h) and holds no real data.
+#
+# Supersedes terraform/rds/ for the deploy scenario — do not apply both at once (they would create
+# two RDS instances). See terraform/ec2/README.md.
+
+terraform {
+  required_version = ">= 1.5"
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 5.0"
+    }
+  }
+}
+
+provider "aws" {
+  region = var.aws_region
+}
+
+# Reuse the lab's default VPC/subnets rather than provisioning a network (XP YAGNI; the lab is
+# throwaway). Same choice as the Phase-11 rds module.
+data "aws_vpc" "default" {
+  default = true
+}
+
+data "aws_subnets" "default" {
+  filter {
+    name   = "vpc-id"
+    values = [data.aws_vpc.default.id]
+  }
+}
+
+# Latest Amazon Linux 2023 AMI — owned by Amazon, resolved at apply time so the runbook never pins a
+# stale AMI id that rots between lab sessions.
+data "aws_ami" "al2023" {
+  most_recent = true
+  owners      = ["amazon"]
+
+  filter {
+    name   = "name"
+    values = ["al2023-ami-2023.*-x86_64"]
+  }
+  filter {
+    name   = "architecture"
+    values = ["x86_64"]
+  }
+}
+
+# ---------------------------------------------------------------------------
+# Security groups
+# ---------------------------------------------------------------------------
+
+# EC2: HTTP (8080) and HTTPS (443) open to the world for the QA walkthrough; SSH (22) only from the
+# developer /32. 443 is here from the start so Phase 20 (Caddy) needs no SG change — it just turns on.
+resource "aws_security_group" "ec2" {
+  name        = "${var.name_prefix}-ec2"
+  description = "ShopSphere EC2 - app 8080, HTTPS 443 from anywhere; SSH from developer /32"
+  vpc_id      = data.aws_vpc.default.id
+
+  ingress {
+    description = "App HTTP (direct, Phase 12)"
+    from_port   = 8080
+    to_port     = 8080
+    protocol    = "tcp"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  ingress {
+    description = "HTTPS via Caddy (Phase 20)"
+    from_port   = 443
+    to_port     = 443
+    protocol    = "tcp"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  ingress {
+    description = "SSH from the developer public /32"
+    from_port   = 22
+    to_port     = 22
+    protocol    = "tcp"
+    cidr_blocks = [var.my_ip_cidr]
+  }
+
+  egress {
+    description = "Unrestricted egress (Docker Hub pull, RDS, OS updates)"
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+}
+
+# RDS: PRIVATE. The posture flip from Phase 11 — ingress is ONLY the EC2 security group, never a
+# laptop /32. Proving this is an acceptance criterion: `psql -h <rds-endpoint>` from the laptop must
+# time out. See ADR-0012.
+resource "aws_security_group" "rds" {
+  name        = "${var.name_prefix}-rds"
+  description = "ShopSphere RDS - Postgres 5432 from the EC2 security group only"
+  vpc_id      = data.aws_vpc.default.id
+
+  ingress {
+    description     = "Postgres from the EC2 instances only"
+    from_port       = 5432
+    to_port         = 5432
+    protocol        = "tcp"
+    security_groups = [aws_security_group.ec2.id]
+  }
+
+  egress {
+    description = "Unrestricted egress"
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+}
+
+# ---------------------------------------------------------------------------
+# RDS (private)
+# ---------------------------------------------------------------------------
+
+resource "aws_db_subnet_group" "this" {
+  name       = "${var.name_prefix}-rds"
+  subnet_ids = data.aws_subnets.default.ids
+}
+
+resource "aws_db_instance" "this" {
+  identifier     = "${var.name_prefix}-postgres"
+  engine         = "postgres"
+  engine_version = var.engine_version
+  instance_class = var.db_instance_class
+
+  allocated_storage = 20
+  storage_type      = "gp3"
+
+  db_name  = var.db_name
+  username = var.db_username
+  password = var.db_password
+  port     = 5432
+
+  db_subnet_group_name   = aws_db_subnet_group.this.name
+  vpc_security_group_ids = [aws_security_group.rds.id]
+  publicly_accessible    = false # the Phase-12 flip — see ADR-0012
+
+  # Throwaway-lab posture — see ADR-0011/0012.
+  skip_final_snapshot     = true
+  backup_retention_period = 0
+  apply_immediately       = true
+  deletion_protection     = false
+}
+
+# ---------------------------------------------------------------------------
+# IAM instance profile (minimal — no SSM yet; Phase 13 documents what own-AWS would add)
+# ---------------------------------------------------------------------------
+
+resource "aws_iam_role" "ec2" {
+  name = "${var.name_prefix}-ec2"
+
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Action    = "sts:AssumeRole"
+      Effect    = "Allow"
+      Principal = { Service = "ec2.amazonaws.com" }
+    }]
+  })
+}
+
+resource "aws_iam_instance_profile" "ec2" {
+  name = "${var.name_prefix}-ec2"
+  role = aws_iam_role.ec2.name
+}
+
+# ---------------------------------------------------------------------------
+# EC2 instance — user-data installs Docker, writes compose.cloud.yml, brings up the stack
+# ---------------------------------------------------------------------------
+
+resource "aws_instance" "app" {
+  ami                    = data.aws_ami.al2023.id
+  instance_type          = var.instance_class
+  subnet_id              = data.aws_subnets.default.ids[0]
+  vpc_security_group_ids = [aws_security_group.ec2.id]
+  iam_instance_profile   = aws_iam_instance_profile.ec2.name
+  key_name               = var.key_name != "" ? var.key_name : null
+
+  # The compose file and Caddyfile are baked into user-data from the repo copies so the running EC2
+  # matches what is committed (no drift between the lab box and version control).
+  user_data = templatefile("${path.module}/user-data.sh.tftpl", {
+    image         = var.app_image
+    db_host       = aws_db_instance.this.address
+    db_port       = "5432"
+    db_name       = var.db_name
+    db_user       = var.db_username
+    db_password   = var.db_password
+    jwt_secret    = var.jwt_secret
+    aws_region    = var.aws_region
+    enable_https  = var.enable_https
+    compose_cloud = file("${path.module}/../../compose.cloud.yml")
+    caddyfile     = file("${path.module}/../../Caddyfile")
+  })
+
+  # Re-run user-data if any of its inputs change.
+  user_data_replace_on_change = true
+
+  tags = {
+    Name = "${var.name_prefix}-app"
+  }
+
+  depends_on = [aws_db_instance.this]
+}
diff --git a/terraform/ec2/outputs.tf b/terraform/ec2/outputs.tf
new file mode 100644
index 0000000..8d15c6e
--- /dev/null
+++ b/terraform/ec2/outputs.tf
@@ -0,0 +1,24 @@
+output "ec2_public_ip" {
+  description = "Public IP of the app EC2. The app is at http://<ip>:8080/swagger-ui.html (and https://<ip> when enable_https=true)."
+  value       = aws_instance.app.public_ip
+}
+
+output "app_http_url" {
+  description = "Direct HTTP entry point (Phase 12)."
+  value       = "http://${aws_instance.app.public_ip}:8080"
+}
+
+output "app_https_url" {
+  description = "HTTPS entry point via Caddy (Phase 20; only serves once enable_https=true). Self-signed cert — expect a browser warning."
+  value       = "https://${aws_instance.app.public_ip}"
+}
+
+output "rds_endpoint" {
+  description = "host:port of the PRIVATE RDS. Reachable from the EC2 only — a psql from your laptop should time out (that is the Phase-12 acceptance check)."
+  value       = aws_db_instance.this.endpoint
+}
+
+output "rds_host" {
+  description = "RDS hostname only — what the EC2's compose.cloud.yml uses as DB_HOST."
+  value       = aws_db_instance.this.address
+}
diff --git a/terraform/ec2/terraform.tfvars.example b/terraform/ec2/terraform.tfvars.example
new file mode 100644
index 0000000..fdbd436
--- /dev/null
+++ b/terraform/ec2/terraform.tfvars.example
@@ -0,0 +1,18 @@
+# Copy to terraform.tfvars (gitignored) and fill in before `terraform apply`.
+# Find your public /32 with: curl -s https://checkip.amazonaws.com
+
+my_ip_cidr  = "203.0.113.4/32"            # only used for SSH (22); RDS is now private
+db_password = "set-a-strong-password-in-your-lab-session"
+jwt_secret  = "generate-with: openssl rand -base64 48"
+
+# Phase 20: set true to also start Caddy (HTTPS on :443, self-signed). Leave false for Phase 12 only.
+# enable_https = true
+
+# Optional EC2 key pair (for SSH debugging). Leave unset to launch without SSH.
+# key_name = "my-lab-keypair"
+
+# Override only if your Whizlabs sandbox is not in us-east-1:
+# aws_region = "us-west-2"
+
+# If the app OOMs on t3.micro (app + Kafka are two JVMs in 1 GiB), bump to t3.small:
+# instance_class = "t3.small"
diff --git a/terraform/ec2/user-data.sh.tftpl b/terraform/ec2/user-data.sh.tftpl
new file mode 100644
index 0000000..acf6fd8
--- /dev/null
+++ b/terraform/ec2/user-data.sh.tftpl
@@ -0,0 +1,54 @@
+#!/bin/bash
+# ShopSphere EC2 bootstrap (Phase 12 + Phase 20). Rendered by Terraform `templatefile`: the
+# dollar-brace placeholders below are TERRAFORM template variables, not shell variables. This script
+# deliberately avoids shell variables with braces so it needs no escaping (bare $TOKEN/$PUBIP are
+# fine; templatefile only touches the brace form). Logs to /var/log/cloud-init-output.log.
+set -euxo pipefail
+
+# --- Docker + compose plugin (Amazon Linux 2023) ---
+dnf install -y docker
+systemctl enable --now docker
+usermod -aG docker ec2-user
+mkdir -p /usr/local/lib/docker/cli-plugins
+curl -SL https://github.com/docker/compose/releases/download/v2.29.7/docker-compose-linux-x86_64 \
+  -o /usr/local/lib/docker/cli-plugins/docker-compose
+chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
+
+# --- App working dir ---
+mkdir -p /opt/shopsphere
+cd /opt/shopsphere
+
+# --- Environment for docker compose interpolation (.env is read automatically) ---
+# These are real values substituted by Terraform at apply time; the app reads them via compose.
+cat > /opt/shopsphere/.env <<'ENVEOF'
+APP_IMAGE=${image}
+DB_HOST=${db_host}
+DB_PORT=${db_port}
+DB_NAME=${db_name}
+DB_USER=${db_user}
+DB_PASSWORD=${db_password}
+JWT_SECRET=${jwt_secret}
+AWS_REGION=${aws_region}
+ENVEOF
+
+# Discover this instance's public IP from IMDSv2 and append it so Caddy mints a cert with the IP in
+# the SAN (browsers don't send SNI for bare-IP URLs, so the cert must carry the IP). Done at runtime
+# rather than templated to avoid the launch-time chicken-and-egg with the public IP.
+TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
+PUBIP=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/public-ipv4)
+echo "PUBLIC_IP=$PUBIP" >> /opt/shopsphere/.env
+chmod 600 /opt/shopsphere/.env
+
+# --- compose.cloud.yml (verbatim copy of the repo file, baked in so the box matches version control) ---
+cat > /opt/shopsphere/compose.cloud.yml <<'COMPOSEEOF'
+${compose_cloud}
+COMPOSEEOF
+
+# --- Caddyfile (used only when HTTPS is enabled) ---
+cat > /opt/shopsphere/Caddyfile <<'CADDYEOF'
+${caddyfile}
+CADDYEOF
+
+# --- Bring up the stack ---
+docker compose -f compose.cloud.yml pull
+docker compose -f compose.cloud.yml %{ if enable_https }--profile caddy %{ endif }up -d
diff --git a/terraform/ec2/variables.tf b/terraform/ec2/variables.tf
new file mode 100644
index 0000000..2ada35c
--- /dev/null
+++ b/terraform/ec2/variables.tf
@@ -0,0 +1,80 @@
+variable "aws_region" {
+  description = "AWS region for the Whizlabs sandbox. Override if your lab is not us-east-1."
+  type        = string
+  default     = "us-east-1"
+}
+
+variable "name_prefix" {
+  description = "Prefix for resource names/identifiers."
+  type        = string
+  default     = "shopsphere"
+}
+
+# --- EC2 ---
+
+variable "instance_class" {
+  description = "EC2 instance type. t3.micro (1 GiB) per the plan, but app + Kafka are two JVMs — if the app OOMs on boot, bump to t3.small (2 GiB). One-line change, no other edits."
+  type        = string
+  default     = "t3.micro"
+}
+
+variable "app_image" {
+  description = "Docker Hub image the EC2 pulls. Built and pushed manually in Phase 12 (Phase 19 would automate)."
+  type        = string
+  default     = "poojithvsc/shopsphere:latest"
+}
+
+variable "key_name" {
+  description = "Optional EC2 key pair name for SSH debugging. Leave blank to launch without SSH access (user-data still runs)."
+  type        = string
+  default     = ""
+}
+
+variable "enable_https" {
+  description = "Phase 20 toggle: when true, user-data also starts the Caddy container (HTTPS via tls internal on :443). Leave false for a Phase-12-only deploy."
+  type        = bool
+  default     = false
+}
+
+variable "jwt_secret" {
+  description = "JWT signing secret for the deployed app (>= 32 bytes). Set in terraform.tfvars (gitignored); never committed. Generate: openssl rand -base64 48"
+  type        = string
+  sensitive   = true
+}
+
+# --- RDS (now private) ---
+
+variable "engine_version" {
+  description = "Postgres engine version. Major-only ('16') matches local docker-compose postgres:16."
+  type        = string
+  default     = "16"
+}
+
+variable "db_instance_class" {
+  description = "RDS instance class. db.t4g.micro is the cheapest Graviton burstable — plenty for a lab."
+  type        = string
+  default     = "db.t4g.micro"
+}
+
+variable "db_name" {
+  description = "Initial database name. Mirrors the local dev DB so Flyway runs identically."
+  type        = string
+  default     = "shopsphere"
+}
+
+variable "db_username" {
+  description = "Master username. Mirrors local dev."
+  type        = string
+  default     = "shopsphere"
+}
+
+variable "db_password" {
+  description = "RDS master password. Set in terraform.tfvars (gitignored); never committed."
+  type        = string
+  sensitive   = true
+}
+
+variable "my_ip_cidr" {
+  description = "The developer's public IP as a /32 — the only source allowed to reach SSH (22). Find it with: curl -s https://checkip.amazonaws.com . NOTE: this is NO LONGER allowed to reach RDS (5432) — that is the Phase-12 privacy flip."
+  type        = string
+}

From 3d045ce7f158e57f327409061b135e4bcf810b7c Mon Sep 17 00:00:00 2001
From: Poojithvsc <poojithrokz@gmail.com>
Date: Sat, 6 Jun 2026 01:52:02 +0200
Subject: [PATCH 3/6] build(deploy): pre-stage Phase 20 HTTPS via Caddy (tls
 internal) [#64]
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Caddyfile reverse-proxies :443 -> app:8080 with a self-signed tls-internal cert;
site address via {$CADDY_SITE_ADDRESS:localhost} so the same file serves localhost
locally and the EC2 public IP in the lab. caddy added to docker-compose.yml (full
profile) and compose.cloud.yml (caddy profile). TLS termination PROVEN locally:
https://localhost/actuator/health returns the app JSON through Caddy, /api/v1/products
-> 401. ADR-0020 (self-signed over Let's Encrypt-impossible; Caddy over nginx;
ACM/ALB as own-AWS replacement). Lab-only ACs run with #58 — does not close #64 yet.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 Caddyfile                                     | 23 +++++++++++++++
 docker-compose.yml                            | 18 ++++++++++++
 docs/adr/0020-https-via-caddy-tls-internal.md | 29 +++++++++++++++++++
 3 files changed, 70 insertions(+)
 create mode 100644 Caddyfile
 create mode 100644 docs/adr/0020-https-via-caddy-tls-internal.md

diff --git a/Caddyfile b/Caddyfile
new file mode 100644
index 0000000..100c2fa
--- /dev/null
+++ b/Caddyfile
@@ -0,0 +1,23 @@
+# ShopSphere — Phase 20: HTTPS termination in front of the app.
+#
+# `tls internal` makes Caddy mint a self-signed cert from its own local CA — no ACME, because
+# Whizlabs ephemeral public IPs can't pass Let's Encrypt's domain validation (and there's no domain).
+#
+# The site address is taken from $CADDY_SITE_ADDRESS so the SAME file works everywhere:
+#   - local testing: env unset -> defaults to `localhost`, so `curl -k https://localhost` works;
+#   - EC2 in the lab: user-data sets CADDY_SITE_ADDRESS to the instance's public IP (read from IMDS),
+#     so Caddy mints an internal cert with that IP in the SAN and `https://<ec2-public-ip>` works.
+# A bare `:443` (no host) is intentionally NOT used — Caddy then has no name to issue a cert for and
+# the TLS handshake fails. Caddy needs a concrete host/IP in the site address.
+#
+# Expect a browser cert warning — that's the untrusted self-signed CA, acceptable for this iteration.
+# On graduation to own AWS, this proxy is replaced by ACM + ALB (a managed, trusted cert) — ADR-0020.
+
+{
+	auto_https disable_redirects
+}
+
+{$CADDY_SITE_ADDRESS:localhost} {
+	tls internal
+	reverse_proxy app:8080
+}
diff --git a/docker-compose.yml b/docker-compose.yml
index 9b61b97..4dc2e82 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -199,5 +199,23 @@ services:
     ports:
       - "8081:8080"
 
+  caddy:
+    # Phase 20 — HTTPS reverse proxy in front of the app, self-signed via `tls internal`. Under the
+    # `full` profile so a local `docker compose --profile full up -d` lets you prove TLS works:
+    # `curl -k https://localhost/actuator/health` reaches the app. The same Caddyfile fronts the EC2
+    # in the cloud deploy (compose.cloud.yml). See ADR-0020.
+    profiles: ["full"]
+    image: caddy:2.8
+    container_name: shopsphere-caddy
+    depends_on:
+      app:
+        condition: service_started
+    volumes:
+      - ./Caddyfile:/etc/caddy/Caddyfile:ro
+      - caddy-data:/data
+    ports:
+      - "443:443"
+
 volumes:
   shopsphere-pgdata:
+  caddy-data:
diff --git a/docs/adr/0020-https-via-caddy-tls-internal.md b/docs/adr/0020-https-via-caddy-tls-internal.md
new file mode 100644
index 0000000..fa2de79
--- /dev/null
+++ b/docs/adr/0020-https-via-caddy-tls-internal.md
@@ -0,0 +1,29 @@
+---
+status: accepted
+date: 2026-06-06
+cites: XP, PragProg, APoSD
+---
+
+# 0020 — HTTPS via Caddy with a self-signed `tls internal` cert
+
+Phase 20 puts TLS in front of the Phase-12 EC2: a Caddy container reverse-proxies `:443` to the app's `:8080`, terminating HTTPS. The whole config is a ~15-line `Caddyfile` and one compose service. Browsers reach `https://<ec2-public-ip>` and get the app behind TLS, after dismissing a self-signed-cert warning that is expected and acceptable for this iteration.
+
+## Self-signed, because Let's Encrypt is impossible here — not because it's easier
+
+ACME / Let's Encrypt issues *publicly trusted* certs, but only after proving control of a **domain**. A Whizlabs lab has an **ephemeral public IP and no domain**, so domain validation can't happen. `tls internal` makes Caddy mint a cert from its own local CA — untrusted by browsers (hence the warning), but real TLS on the wire. **PragProg — be honest about the trade:** this is the *correct* choice given the constraint, not a shortcut; the limitation (browser warning, no public trust) is named, not hidden. The cert carries the instance's IP in its SAN (browsers send no SNI for bare-IP URLs), which is why the site address is the public IP, injected at deploy time.
+
+## Caddy over nginx — the cert lifecycle is hidden behind two words
+
+`tls internal` is the entire TLS story: Caddy generates the CA, issues the leaf, renews it, and serves it. The nginx equivalent is a `openssl req` to generate a self-signed cert, a volume to mount it, `ssl_certificate`/`ssl_certificate_key` directives, and a renewal you own. **APoSD — deep module:** Caddy presents a two-word interface over the whole certificate lifecycle; nginx exposes the mechanism and makes the operator the lifecycle manager. **XP YAGNI:** for a reverse proxy that does TLS + one `reverse_proxy` line, the simpler tool wins.
+
+## Same Caddyfile local and cloud — proven before the lab
+
+The site address comes from `{$CADDY_SITE_ADDRESS:localhost}`, so the committed `Caddyfile` is byte-identical in both places: unset locally → `localhost` (so `curl -k https://localhost` works on a laptop), and set to the instance IP on the EC2. **PragProg tracer bullet:** TLS termination was proven end-to-end locally (`https://localhost/actuator/health` returns the app's JSON through Caddy; `/api/v1/products` returns 401 — routing + auth intact) **before** spending a minute of lab time. The lab run is then confirmation, not discovery. A bare `:443` site address was tried first and rejected: with no host/IP, Caddy has no name to issue a cert for and the handshake fails — the site address must be concrete.
+
+## Consequences
+
+HTTPS fronts the app with a self-signed cert; plain `http://<ip>:8080` stays open in this iteration (the SG allows both) and is documented as such — closing it would force HTTPS but is unnecessary for the lab. `mvn verify` is unaffected (infra only). Honest limits:
+
+- **Browser warning is inherent** to self-signed; this is not production-grade trust.
+- **The on-EC2 acceptance** (`https://<ec2-public-ip>` in a browser, QA walkthrough over HTTPS) is a **manual lab step**, batched with Phase 12 in one Whizlabs session — the two phases share the same ephemeral EC2.
+- **On graduation to own AWS, this proxy is replaced by ACM + ALB** — a managed, publicly-trusted cert with auto-renewal, terminating at the load balancer. That's the real-world end state; self-signed Caddy is the lab-appropriate stand-in. Phase 20 is the last numbered phase; nothing depends on it (it's HTTPS polish), so it is skippable in-lab if Phase 12 consumes the time box.

From 80387d711dc37df9184e86962c49eb713c7a2d6d Mon Sep 17 00:00:00 2001
From: Poojithvsc <poojithrokz@gmail.com>
Date: Sat, 6 Jun 2026 17:44:35 +0200
Subject: [PATCH 4/6] build(deploy): harden EC2 module from live lab; add
 use_rds container-Postgres fallback [#58]

Live Whizlabs run surfaced several constraints; fixes:
- use_rds toggle: when false, skip all RDS resources (count=0) and run postgres:16
  in a localdb compose profile on the EC2 (sandbox denied RDS, even rds:Describe).
  The managed-private-RDS path is untouched (use_rds=true, validate-clean).
- t2.micro OOM: 2 GiB swapfile in user-data + KAFKA_HEAP_OPTS=-Xmx384m and app
  JAVA_TOOL_OPTIONS=-Xmx384m (cp-kafka defaults to -Xmx1G, which alone OOM-kills 1 GiB).
- create_instance_profile toggle: lab IAM user is denied iam:CreateRole (empty
  SSM-placeholder role made optional).
- associate_public_ip_address=true: sandbox default subnet has MapPublicIpOnLaunch=false.
- robust public-IP fetch (checkip.amazonaws.com + IMDS fallback) so Caddy's cert SAN
  gets the IP (supports #64).
- ADR-0012: document use_rds as a genuine two-value seam; RDS-private test deferred.

RDS-private negative test stays deferred (no RDS-capable AWS); #58 remains open.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 compose.cloud.yml                             | 27 +++++++++++++++++
 ...c2-deploy-self-hosted-kafka-rds-private.md |  8 +++++
 terraform/ec2/main.tf                         | 25 +++++++++++-----
 terraform/ec2/outputs.tf                      |  6 ++--
 terraform/ec2/user-data.sh.tftpl              | 29 ++++++++++++++-----
 terraform/ec2/variables.tf                    | 12 ++++++++
 6 files changed, 89 insertions(+), 18 deletions(-)

diff --git a/compose.cloud.yml b/compose.cloud.yml
index 5174b35..0bb6e2d 100644
--- a/compose.cloud.yml
+++ b/compose.cloud.yml
@@ -13,6 +13,26 @@
 # Bring up with HTTPS (Phase 20): docker compose -f compose.cloud.yml --profile caddy up -d
 
 services:
+  # Fallback DB for sandboxes that deny RDS (use_rds=false → user-data adds `--profile localdb`).
+  # NOT the real Phase-12 design — the managed-private-RDS posture (ADR-0012) is unaffected: this
+  # service simply never starts when use_rds=true. App reaches it at DB_HOST=postgres (set by user-data).
+  postgres:
+    profiles: ["localdb"]
+    image: postgres:16
+    container_name: shopsphere-postgres
+    environment:
+      POSTGRES_DB: ${DB_NAME:-shopsphere}
+      POSTGRES_USER: ${DB_USER:-shopsphere}
+      POSTGRES_PASSWORD: ${DB_PASSWORD}
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-shopsphere} -d ${DB_NAME:-shopsphere}"]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    restart: unless-stopped
+
   kafka:
     image: confluentinc/cp-kafka:7.6.1
     container_name: shopsphere-kafka
@@ -31,6 +51,9 @@ services:
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
       KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
       CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qg
+      # cp-kafka defaults to -Xmx1G, which alone exhausts a 1 GiB t2.micro and OOM-kills the stack.
+      # Cap it so app + Kafka coexist on the lab box (with swap as the safety net). See ADR-0012.
+      KAFKA_HEAP_OPTS: "-Xmx384m -Xms256m"
     healthcheck:
       test: ["CMD-SHELL", "kafka-broker-api-versions --bootstrap-server localhost:29092 >/dev/null 2>&1 || exit 1"]
       interval: 10s
@@ -54,6 +77,9 @@ services:
       # S3 dormant: blank endpoint → SDK uses real-S3 resolution; the core QA flow never calls it.
       S3_ENDPOINT: ""
       AWS_REGION: ${AWS_REGION:-us-east-1}
+      # Cap the app heap so it shares the 1 GiB t2.micro with Kafka. JAVA_TOOL_OPTIONS is honoured by
+      # the JVM regardless of entrypoint. Lift/remove on a larger box.
+      JAVA_TOOL_OPTIONS: "-Xmx384m"
     ports:
       # Direct HTTP (Phase 12). Caddy (Phase 20) also fronts this on 443.
       - "8080:8080"
@@ -82,3 +108,4 @@ services:
 
 volumes:
   caddy-data:
+  pgdata:
diff --git a/docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md b/docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md
index 26ab8a6..bfdafb8 100644
--- a/docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md
+++ b/docs/adr/0012-ec2-deploy-self-hosted-kafka-rds-private.md
@@ -24,6 +24,14 @@ The image is built locally and pushed to a public Docker Hub repo (`poojithvsc/s
 
 Default VPC (no bespoke network), t3.micro, no backups, no final snapshot, `apply_immediately`. **XP YAGNI:** a 4-hour box does not warrant private subnets + NAT or a backup plan. This is correct *only* because the lab is ephemeral and holds no real data — the same honesty as ADR-0011. The IAM instance profile is minimal (no SSM yet); ADR-0013 already documents what own-AWS would add.
 
+## Fallback: Postgres as a container when RDS is unavailable (`use_rds=false`)
+
+Added live during the 2026-06-06 lab: the Whizlabs **Cloud Sandbox denies RDS entirely** (even `rds:Describe`), while the only RDS-capable lab (the guided "EC2+RDS Terraform" one) is 60-min and attempt-capped. So the module gained a `use_rds` toggle. When `false`, all RDS resources are skipped (`count = 0`) and a `postgres:16` container runs on the EC2 under a `localdb` compose profile; the app reaches it at `DB_HOST=postgres`.
+
+This is **not** the target architecture and it does **not** demonstrate this ADR's headline lesson — the *managed, private DB behind a network boundary*. A co-located container has no separate network to lock down; the "laptop psql must time out" acceptance check is meaningless against it. That check stays **deferred** to RDS-capable AWS (the guided lab after its attempt reset, or own-AWS), exactly as #58 records.
+
+Why keep it in the codebase rather than as a throwaway branch (decided with the books): it is a genuine **two-value seam**, not a dead toggle — `false` ran live, `true` is `terraform validate`-clean and is the own-AWS path. That mirrors the project's existing abstraction-behind-a-seam stance (cf. ADR-0015's payment stub). **XP YAGNI** is noted honestly: once own-AWS is the only target the container path is dead weight and may be removed; until then it earned its place by being the only way the deploy ran at all.
+
 ## Consequences
 
 `terraform apply` brings up the whole deploy; `terraform destroy` removes it. `mvn verify` is unaffected (this is infrastructure, no app code). Three honest limits, recorded so they don't surprise later:
diff --git a/terraform/ec2/main.tf b/terraform/ec2/main.tf
index 1595896..00f5edf 100644
--- a/terraform/ec2/main.tf
+++ b/terraform/ec2/main.tf
@@ -102,6 +102,7 @@ resource "aws_security_group" "ec2" {
 # laptop /32. Proving this is an acceptance criterion: `psql -h <rds-endpoint>` from the laptop must
 # time out. See ADR-0012.
 resource "aws_security_group" "rds" {
+  count       = var.use_rds ? 1 : 0
   name        = "${var.name_prefix}-rds"
   description = "ShopSphere RDS - Postgres 5432 from the EC2 security group only"
   vpc_id      = data.aws_vpc.default.id
@@ -128,11 +129,13 @@ resource "aws_security_group" "rds" {
 # ---------------------------------------------------------------------------
 
 resource "aws_db_subnet_group" "this" {
+  count      = var.use_rds ? 1 : 0
   name       = "${var.name_prefix}-rds"
   subnet_ids = data.aws_subnets.default.ids
 }
 
 resource "aws_db_instance" "this" {
+  count          = var.use_rds ? 1 : 0
   identifier     = "${var.name_prefix}-postgres"
   engine         = "postgres"
   engine_version = var.engine_version
@@ -146,8 +149,8 @@ resource "aws_db_instance" "this" {
   password = var.db_password
   port     = 5432
 
-  db_subnet_group_name   = aws_db_subnet_group.this.name
-  vpc_security_group_ids = [aws_security_group.rds.id]
+  db_subnet_group_name   = aws_db_subnet_group.this[0].name
+  vpc_security_group_ids = [aws_security_group.rds[0].id]
   publicly_accessible    = false # the Phase-12 flip — see ADR-0012
 
   # Throwaway-lab posture — see ADR-0011/0012.
@@ -162,7 +165,8 @@ resource "aws_db_instance" "this" {
 # ---------------------------------------------------------------------------
 
 resource "aws_iam_role" "ec2" {
-  name = "${var.name_prefix}-ec2"
+  count = var.create_instance_profile ? 1 : 0
+  name  = "${var.name_prefix}-ec2"
 
   assume_role_policy = jsonencode({
     Version = "2012-10-17"
@@ -175,8 +179,9 @@ resource "aws_iam_role" "ec2" {
 }
 
 resource "aws_iam_instance_profile" "ec2" {
-  name = "${var.name_prefix}-ec2"
-  role = aws_iam_role.ec2.name
+  count = var.create_instance_profile ? 1 : 0
+  name  = "${var.name_prefix}-ec2"
+  role  = aws_iam_role.ec2[0].name
 }
 
 # ---------------------------------------------------------------------------
@@ -188,14 +193,17 @@ resource "aws_instance" "app" {
   instance_type          = var.instance_class
   subnet_id              = data.aws_subnets.default.ids[0]
   vpc_security_group_ids = [aws_security_group.ec2.id]
-  iam_instance_profile   = aws_iam_instance_profile.ec2.name
-  key_name               = var.key_name != "" ? var.key_name : null
+  # Force a public IP — some sandbox default subnets have MapPublicIpOnLaunch=false, leaving the box
+  # unreachable. Explicit assignment makes the deploy subnet-independent.
+  associate_public_ip_address = true
+  iam_instance_profile        = var.create_instance_profile ? aws_iam_instance_profile.ec2[0].name : null
+  key_name                    = var.key_name != "" ? var.key_name : null
 
   # The compose file and Caddyfile are baked into user-data from the repo copies so the running EC2
   # matches what is committed (no drift between the lab box and version control).
   user_data = templatefile("${path.module}/user-data.sh.tftpl", {
     image         = var.app_image
-    db_host       = aws_db_instance.this.address
+    db_host       = var.use_rds ? aws_db_instance.this[0].address : "postgres"
     db_port       = "5432"
     db_name       = var.db_name
     db_user       = var.db_username
@@ -203,6 +211,7 @@ resource "aws_instance" "app" {
     jwt_secret    = var.jwt_secret
     aws_region    = var.aws_region
     enable_https  = var.enable_https
+    use_rds       = var.use_rds
     compose_cloud = file("${path.module}/../../compose.cloud.yml")
     caddyfile     = file("${path.module}/../../Caddyfile")
   })
diff --git a/terraform/ec2/outputs.tf b/terraform/ec2/outputs.tf
index 8d15c6e..280f68f 100644
--- a/terraform/ec2/outputs.tf
+++ b/terraform/ec2/outputs.tf
@@ -14,11 +14,11 @@ output "app_https_url" {
 }
 
 output "rds_endpoint" {
-  description = "host:port of the PRIVATE RDS. Reachable from the EC2 only — a psql from your laptop should time out (that is the Phase-12 acceptance check)."
-  value       = aws_db_instance.this.endpoint
+  description = "host:port of the PRIVATE RDS. Reachable from the EC2 only — a psql from your laptop should time out (that is the Phase-12 acceptance check). Empty when use_rds=false (Postgres runs as a container on the EC2)."
+  value       = var.use_rds ? aws_db_instance.this[0].endpoint : "(none — Postgres is a container on the EC2; use_rds=false)"
 }
 
 output "rds_host" {
   description = "RDS hostname only — what the EC2's compose.cloud.yml uses as DB_HOST."
-  value       = aws_db_instance.this.address
+  value       = var.use_rds ? aws_db_instance.this[0].address : "postgres (in-EC2 container)"
 }
diff --git a/terraform/ec2/user-data.sh.tftpl b/terraform/ec2/user-data.sh.tftpl
index acf6fd8..afb9511 100644
--- a/terraform/ec2/user-data.sh.tftpl
+++ b/terraform/ec2/user-data.sh.tftpl
@@ -5,6 +5,14 @@
 # fine; templatefile only touches the brace form). Logs to /var/log/cloud-init-output.log.
 set -euxo pipefail
 
+# --- Swap: t2.micro has only 1 GiB RAM and runs two JVMs (app + Kafka). A 2 GiB swapfile absorbs
+# memory spikes on boot so the kernel does not OOM-kill the stack before the app binds :8080.
+# Cheap insurance on an ephemeral box. ---
+fallocate -l 2G /swapfile || dd if=/dev/zero of=/swapfile bs=1M count=2048
+chmod 600 /swapfile
+mkswap /swapfile
+swapon /swapfile
+
 # --- Docker + compose plugin (Amazon Linux 2023) ---
 dnf install -y docker
 systemctl enable --now docker
@@ -31,11 +39,16 @@ JWT_SECRET=${jwt_secret}
 AWS_REGION=${aws_region}
 ENVEOF
 
-# Discover this instance's public IP from IMDSv2 and append it so Caddy mints a cert with the IP in
-# the SAN (browsers don't send SNI for bare-IP URLs, so the cert must carry the IP). Done at runtime
-# rather than templated to avoid the launch-time chicken-and-egg with the public IP.
-TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
-PUBIP=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/public-ipv4)
+# Discover this instance's public IP and append it so Caddy mints a cert with the IP in the SAN
+# (browsers send no SNI for bare-IP URLs, so the cert must carry the IP; an empty site address makes
+# Caddy serve a no-SAN fallback cert that fails the handshake). Done at runtime rather than templated
+# to avoid the launch-time chicken-and-egg with the public IP. Prefer the egress echo service (reliable
+# on a box that has a public IP); fall back to IMDSv2. `|| true` so set -e doesn't abort on a miss.
+PUBIP=$(curl -s --max-time 10 https://checkip.amazonaws.com | tr -d '[:space:]' || true)
+if [ -z "$PUBIP" ]; then
+  TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300" || true)
+  PUBIP=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/public-ipv4 || true)
+fi
 echo "PUBLIC_IP=$PUBIP" >> /opt/shopsphere/.env
 chmod 600 /opt/shopsphere/.env
 
@@ -50,5 +63,7 @@ ${caddyfile}
 CADDYEOF
 
 # --- Bring up the stack ---
-docker compose -f compose.cloud.yml pull
-docker compose -f compose.cloud.yml %{ if enable_https }--profile caddy %{ endif }up -d
+# When use_rds=false, also start the in-EC2 Postgres container (--profile localdb); the app reaches
+# it at DB_HOST=postgres. When true, the app uses the private RDS and no DB container runs.
+docker compose -f compose.cloud.yml %{ if !use_rds }--profile localdb %{ endif }%{ if enable_https }--profile caddy %{ endif }pull
+docker compose -f compose.cloud.yml %{ if !use_rds }--profile localdb %{ endif }%{ if enable_https }--profile caddy %{ endif }up -d
diff --git a/terraform/ec2/variables.tf b/terraform/ec2/variables.tf
index 2ada35c..179e5ce 100644
--- a/terraform/ec2/variables.tf
+++ b/terraform/ec2/variables.tf
@@ -30,6 +30,18 @@ variable "key_name" {
   default     = ""
 }
 
+variable "use_rds" {
+  description = "When true (the real Phase-12 design), provision a PRIVATE managed RDS and point the app at it. When false, skip all RDS resources and run Postgres as a container on the EC2 instead — a fallback for Whizlabs sandboxes that deny RDS (the SAA Cloud Sandbox denies even rds:Describe). The container path does NOT demonstrate the 'private managed DB / network-as-security-control' lesson of ADR-0012; that acceptance criterion is deferred to the guided RDS lab or own-AWS."
+  type        = bool
+  default     = true
+}
+
+variable "create_instance_profile" {
+  description = "Create an IAM role + instance profile for the EC2. Whizlabs IAM users often lack iam:CreateRole; set false to skip it. The lab QA flow needs no AWS API access from the box (public image pull, password DB auth, S3 dormant), so skipping is safe. Re-enable on own-AWS when SSM/Parameter Store land (ADR-0013)."
+  type        = bool
+  default     = true
+}
+
 variable "enable_https" {
   description = "Phase 20 toggle: when true, user-data also starts the Caddy container (HTTPS via tls internal on :443). Leave false for a Phase-12-only deploy."
   type        = bool

From f20b7c73d40c1f551e2d30b34f9a2f20e9991ca3 Mon Sep 17 00:00:00 2001
From: Poojithvsc <poojithrokz@gmail.com>
Date: Sat, 6 Jun 2026 17:44:42 +0200
Subject: [PATCH 5/6] fix(https): set Caddy default_sni so bare-IP HTTPS works
 in browsers [#64]
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Browsers send no SNI for a bare-IP URL, so Caddy couldn't match the IP-keyed site
and served an empty no-SAN fallback cert -> ERR_SSL_PROTOCOL_ERROR. Adding
default_sni {$CADDY_SITE_ADDRESS} makes Caddy present the IP cert (IP in SAN) to
no-SNI clients. Verified live via openssl (no -servername) and a browser.

Note: Windows curl/PowerShell (SChannel) still can't handshake with `tls internal`
certs — a client limitation, not a server fault; verify with openssl or a browser.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 Caddyfile | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Caddyfile b/Caddyfile
index 100c2fa..69fe7a2 100644
--- a/Caddyfile
+++ b/Caddyfile
@@ -12,9 +12,12 @@
 #
 # Expect a browser cert warning — that's the untrusted self-signed CA, acceptable for this iteration.
 # On graduation to own AWS, this proxy is replaced by ACM + ALB (a managed, trusted cert) — ADR-0020.
-
 {
 	auto_https disable_redirects
+	# Browsers send NO SNI for a bare-IP URL, so Caddy can't match the IP-keyed site and would serve
+	# an empty fallback cert (→ ERR_SSL_PROTOCOL_ERROR). default_sni makes Caddy assume the site's
+	# address when SNI is absent, so it presents the IP cert (with the IP in its SAN) to browsers.
+	default_sni {$CADDY_SITE_ADDRESS:localhost}
 }
 
 {$CADDY_SITE_ADDRESS:localhost} {

From ddad455ef85085ee61e5d3fb9ab70bfbb282a1d7 Mon Sep 17 00:00:00 2001
From: Poojithvsc <poojithrokz@gmail.com>
Date: Sat, 6 Jun 2026 17:44:42 +0200
Subject: [PATCH 6/6] docs(runbook): capture live-lab constraints and fixes for
 the EC2+HTTPS lab [#58][#64]

Whizlabs constraints box (t2.micro-only, no IAM roles, no session token), fresh-lab
restart procedure, get-console-output diagnostic (no SSH), and OOM mitigation notes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 docs/lab-runbook-ec2-https.md | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/docs/lab-runbook-ec2-https.md b/docs/lab-runbook-ec2-https.md
index 0f06cbe..36378d3 100644
--- a/docs/lab-runbook-ec2-https.md
+++ b/docs/lab-runbook-ec2-https.md
@@ -16,6 +16,17 @@ docker push poojithvsc/shopsphere:latest
 
 - [ ] `poojithvsc/shopsphere:latest` is on Docker Hub.
 
+## 0b. Restarting in a *fresh* Whizlabs lab (if a prior lab expired mid-session)
+
+Each lab is a new AWS account, so the local Terraform state from a dead lab points at gone resources. Don't try to `destroy` the old one (its creds are expired) — just start clean:
+
+```powershell
+cd D:\shopsphere-project\code\terraform\ec2
+del terraform.tfstate, terraform.tfstate.backup   # ignore "not found"; orphaned resources are auto-reaped by the dead lab
+```
+
+`terraform.tfvars` is reusable as-is (only update `my_ip_cidr` if your public IP changed). Then continue from §1 with the new lab's creds.
+
 ## 1. Start the Whizlabs lab + credentials
 
 1. Launch the Whizlabs AWS sandbox; note the region (assume `us-east-1`).
@@ -34,6 +45,12 @@ copy terraform.tfvars.example terraform.tfvars
 #   db_password, jwt_secret (openssl rand -base64 48). Leave enable_https commented for now.
 ```
 
+> **Whizlabs lab constraints (discovered 2026-06-06 — already baked into `terraform.tfvars`):**
+> - `iam:CreateRole` is **denied** → `create_instance_profile = false` (the instance profile is an empty SSM placeholder; the box needs no AWS API access for the QA flow).
+> - `ec2:RunInstances` is **explicitly denied for any type except `t2.micro`** → `instance_class = "t2.micro"`.
+> - `sts:DecodeAuthorizationMessage` is denied too, so authz-failure messages can't be decoded — diagnose by hypothesis.
+> - The lab IAM user is long-lived (no `AWS_SESSION_TOKEN`); export only the access key + secret.
+
 ## 3. Phase 12 — apply (HTTP only)
 
 ```powershell
@@ -43,7 +60,14 @@ terraform apply        # type yes; ~5-8 min (RDS is the long pole)
 
 Capture outputs: `ec2_public_ip`, `rds_endpoint`, `app_http_url`.
 
-- [ ] **App is up:** open `http://<ec2_public_ip>:8080/swagger-ui.html` in a browser. (If it's not up after ~3 min, SSH in or check: the app waits on Kafka health; user-data logs are in `/var/log/cloud-init-output.log`.)
+- [ ] **App is up:** open `http://<ec2_public_ip>:8080/swagger-ui.html` in a browser. The app boots **slowly on t2.micro** (1 GiB, two JVMs + swap) — allow ~5–8 min after `apply` finishes before worrying.
+- [ ] **If it never answers**, the instance launched without SSH, so read the boot log via the console instead (no SSH/SSM needed):
+  ```powershell
+  aws ec2 get-console-output --instance-id <i-xxxx> --output text | Select-String -Pattern "swapon|docker|compose|Started|ERROR|Killed|OOM" -Context 0,2
+  ```
+  Look for the image pull, `docker compose up`, and any `Killed`/`OOM` (memory) or pull errors. The full boot log is also at `/var/log/cloud-init-output.log` if you do have SSH.
+
+Memory fixes already in place (so the above should not recur): a 2 GiB swapfile in user-data, `KAFKA_HEAP_OPTS=-Xmx384m` and app `JAVA_TOOL_OPTIONS=-Xmx384m` in `compose.cloud.yml`. If it still OOMs, try `instance_class = "t2.small"` (only if the lab policy allows it).
 - [ ] **RDS is private (the negative test):** from your laptop,
   ```powershell
   psql "host=<rds-host> port=5432 dbname=shopsphere user=shopsphere"   # must TIME OUT
@@ -103,8 +127,10 @@ Tell me the results and I'll:
 
 | Symptom | Likely cause | Fix |
 |---|---|---|
-| App never comes up on :8080 | t3.micro OOM (app + Kafka in 1 GiB) | `terraform apply -var instance_class=t3.small` |
+| App never comes up on :8080 | t2.micro OOM (app + Kafka in 1 GiB) | Mitigated: swap + heap caps are baked in. If still OOM, `instance_class = "t2.small"` (if lab allows). Confirm via `aws ec2 get-console-output`. |
 | `terraform apply` AccessDenied | Lab creds expired / wrong scope | Re-export lab creds; `aws sts get-caller-identity` |
+| `iam:CreateRole` denied | Whizlabs IAM user can't make roles | `create_instance_profile = false` (already set) |
+| `ec2:RunInstances` explicit deny | Whizlabs allows only `t2.micro` | `instance_class = "t2.micro"` (already set) |
 | `psql` from laptop *connects* (should time out) | RDS SG still public, or you applied `terraform/rds/` not `terraform/ec2/` | Confirm you're in `terraform/ec2/`; check `aws_security_group.rds` ingress is the EC2 SG |
 | HTTPS handshake fails | Caddy didn't get the public IP | Check `/opt/shopsphere/.env` has `PUBLIC_IP=`; `docker logs shopsphere-caddy` |
 | Image pull fails on EC2 | Docker Hub repo private / typo | Ensure `poojithvsc/shopsphere:latest` is public; check `docker compose logs` |