From 2110e01b0d4c289a472e7226663275196e37a9e1 Mon Sep 17 00:00:00 2001 From: Manas Srivastava Date: Sat, 6 Jun 2026 20:25:12 +0530 Subject: [PATCH] sec(data): persist admin-lockdown into postgres-customers Deployment + drill log MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the 2026-06-06 apply of PR #61. The lockdown was applied to prod via `kubectl patch` (imperative). This makes the durable repo manifest match the live state so a future apply of postgres-customers.yaml does NOT silently revert the lockdown back to the vulnerable catch-all pg_hba: - mount the postgres-customers-hba ConfigMap at /etc/postgresql/pg_hba.conf (subPath) - start postgres with `-c hba_file=... -c password_encryption=scram-sha-256` - strategy RollingUpdate → Recreate (the RWO PVC deadlocks a rolling update on a Multi-Attach error; Recreate terminates the old pod first — brief downtime, acceptable for single-replica stateful) Runbook §9 Drill Log records the apply result: external admin (instanode_admin + instant_cust) now REJECTED at pg_hba (verified live; baseline reached scram), all in-cluster admin + customer usr_* paths preserved (verified), no rollback. Lists the operator follow-ups (durable pg-proxy role-gate; proxy-IP churn refresh; networkpolicy.yaml apply-exclude). Co-Authored-By: Claude Opus 4.8 --- POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md | 18 ++++++++++++++++ k8s/data/postgres-customers.yaml | 30 ++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md b/POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md index 3928d15..e1fc5c6 100644 --- a/POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md +++ b/POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md @@ -301,3 +301,21 @@ is already shipped (audit doc): Together: this runbook removes the *unaudited external admin DROP capability*; the chokepoint ensures every *sanctioned* drop is recorded; the CI guard ensures a *new* unaudited drop call site cannot be merged. + +--- + +## 9. Drill Log + +| Date | Operator | Action | Result | +|---|---|---|---| +| 2026-06-06 | Claude (operator-authorized apply, "no customers, low blast radius") | **APPLIED to do-nyc3-instant-prod.** Merged PR #61 (squash, merge commit `78cb6677`) after fixing the manifest for two live findings (see below). Applied ConfigMap `postgres-customers-hba`; patched `deploy/postgres-customers` to mount it + `-c hba_file=/etc/postgresql/pg_hba.conf -c password_encryption=scram-sha-256`; changed strategy `RollingUpdate→Recreate` (RWO PVC Multi-Attach). Did NOT apply `networkpolicy.yaml` (verified NOT enforced in prod; applying as-is would default-deny the proxy path). | **SUCCESS.** External admin REJECTED at pg_hba (both `instanode_admin` + `instant_cust`, error names the SNAT'd proxy pod IP) — baseline beforehand reached scram (vector was OPEN). In-cluster admin preserved: provisioner `instant_cust` CREATE/DROP smoke OK, api/worker `instanode_admin` connect + `pg_database_size` OK, customer `usr_*` path still reaches scram. No rollback. | + +**Manifest fixes made before apply (live pre-apply verification):** +1. **`instanode_admin` was missing.** Prod has TWO superusers — `instanode_admin` (api/worker `CUSTOMER_DATABASE_URL`, the CONFIRMED truehomie vector) and `instant_cust` (provisioner `POSTGRES_CUSTOMERS_URL`). The original PR rejected only `instant_cust`; `instanode_admin` would have matched the catch-all customer allow → vector still open. Both now rejected. +2. **pg-proxy SNAT defeats source-CIDR.** instant-pg-proxy (in-cluster, no hostNetwork) re-originates TCP, so external admin arrives SNAT'd to a proxy pod IP inside `10.0.0.0/8` — a plain `10.0.0.0/8 allow` matches it. Added proxy-pod-IP `reject` lines (`10.109.4.113`, `10.109.0.101`) ordered BEFORE the in-cluster allow. **Verified in the reject error message** (`rejects connection for host "10.109.0.101"`). ⚠️ Churn dependency, see §3a. + +**Operator follow-ups created by this apply:** +- **Ship the durable pg-proxy role-gate** (`PG_PROXY_DENIED_ROLES` in `InstaNode-dev/instant-pg-proxy`, staged per memory) so the closure no longer depends on the churning proxy-pod-IP reject lines in the ConfigMap. +- **On any `instant-pg-proxy` reschedule:** refresh the two `host all instanode_admin/instant_cust /32 reject` lines in `postgres-customers-lockdown.yaml`, re-apply, `SELECT pg_reload_conf()`. Add a proxy-pod-restart alert. +- **`k8s/data/postgres-customers.yaml` updated** to carry the mount/args/Recreate-strategy so a future repo apply does not silently revert the lockdown (shipped in the same follow-up PR). +- The repo `apply.yml` workflow now includes `postgres-customers-lockdown.yaml` (safe — ConfigMap) but ALSO `networkpolicy.yaml`; running that workflow WOULD create the unenforced-today NetPol and default-deny the proxy path. Add it to the apply EXCLUDE list or add the pg-proxy ingress rule before anyone runs the workflow. diff --git a/k8s/data/postgres-customers.yaml b/k8s/data/postgres-customers.yaml index 427d218..413b631 100644 --- a/k8s/data/postgres-customers.yaml +++ b/k8s/data/postgres-customers.yaml @@ -19,6 +19,13 @@ metadata: namespace: instant-data spec: replicas: 1 + # Recreate (NOT RollingUpdate): the PVC is ReadWriteOnce, so a rolling update + # deadlocks on a Multi-Attach error (new pod can't attach the volume while the + # old pod still holds it on another node). Recreate terminates the old pod + # first → brief downtime, acceptable for a single-replica stateful workload. + # (truehomie admin-lockdown apply 2026-06-06 hit + fixed this.) + strategy: + type: Recreate selector: matchLabels: app: postgres-customers @@ -30,6 +37,17 @@ spec: containers: - name: postgres image: pgvector/pgvector:pg16 + # truehomie-db-drop admin lockdown (2026-06-03 → applied 2026-06-06): + # start postgres with the custom pg_hba (mounted below) that rejects the + # admin superuser roles (instanode_admin/instant_cust) from the public + # pg-proxy path while preserving the in-cluster admin + customer usr_* + # paths. See k8s/data/postgres-customers-lockdown.yaml + + # POSTGRES-CUSTOMERS-LOCKDOWN-RUNBOOK.md. + args: + - "-c" + - "hba_file=/etc/postgresql/pg_hba.conf" + - "-c" + - "password_encryption=scram-sha-256" ports: - containerPort: 5432 env: @@ -45,6 +63,12 @@ spec: volumeMounts: - mountPath: /var/lib/postgresql/data name: data + # Admin-lockdown pg_hba (ConfigMap postgres-customers-hba). subPath + # mounts just the single file so PGDATA is untouched. + - name: hba + mountPath: /etc/postgresql/pg_hba.conf + subPath: pg_hba.conf + readOnly: true readinessProbe: exec: command: ["pg_isready", "-U", "instant_cust", "-d", "instant_customers"] @@ -54,6 +78,12 @@ spec: - name: data persistentVolumeClaim: claimName: postgres-customers-pvc + - name: hba + configMap: + name: postgres-customers-hba + items: + - key: pg_hba.conf + path: pg_hba.conf --- apiVersion: v1 kind: Service