P2: Reference deployment manifests — k8s + prod Compose (#179)

[dkijania]

What & why

Part of the production-readiness epic (#163). Closes #179.

There were no production deployment artifacts — operators had npm/Docker/Compose for dev but no opinionated manifest with probes, resource limits, and a hardened runtime.

Adds deploy/

• kubernetes.yaml — Deployment + Service + HorizontalPodAutoscaler (+ placeholder Secret) with production defaults:
  • liveness on /healthcheck, readiness on /readiness
  • resource requests/limits, 2→6 CPU autoscaler
  • hardened pod securityContext (non-root, readOnlyRootFilesystem, no privilege escalation, all caps dropped, RuntimeDefault seccomp)
  • Prometheus scrape annotations for /metrics
  • terminationGracePeriodSeconds: 30 matching the graceful-shutdown drain
• docker-compose.prod.yml — the published image against an external read-only Postgres, with CPU/memory caps.
• README.md — usage + how it maps to the security deployment contract.

Linked from the root README. References the probe/metrics endpoints delivered by the sibling P1 PRs (#169/#173).

Testing

Docs/manifests only. prettier --debug-check . clean; YAML structure validated. No application code changed.

🤖 Generated with Claude Code

[SanabriaRusso]

Really glad to see opinionated production manifests land — the hardened pod securityContext, HPA, and Secret-based PG_CONN are exactly right, and runAsUser: 1001 correctly matches the Dockerfile's nodeuser. Two things worth tightening before this becomes the copy-paste reference:

1. CORS will silently block the mina-explorer once #184 merges. The Explorer is a cross-origin browser app (mina-explorer/src/config/networks.ts POSTs from its UI origin to https://*-archive-node-api.*.o1test.net), so CORS_ORIGIN is load-bearing for it. Today the server defaults to * when unset (docs/getting-started.md), so the commented-out var happens to work — but the inline note "leave unset for same-origin only" isn't accurate (unset = all origins today), and once #184 flips the default to deny-by-default this reference deploy will reject every browser client, the Explorer included. Since the primary consumer is cross-origin, I'd ship the example as a set value rather than commented-out:

- name: CORS_ORIGIN
  # The mina-explorer (and any browser UI) is cross-origin and CANNOT reach this
  # API unless its origin is allowlisted here. Comma-separate multiple origins.
  # NB: the server currently defaults to '*' when unset; #184 changes that to deny.
  value: 'https://explorer.example.com'

(same note applies to docker-compose.prod.yml.)

2. The readiness probe targets /readiness, which no released image serves yet. /readiness comes from #187 (still open) — on main and the current ghcr.io/o1-labs/archive-node-api image, yoga returns 404 for that path, so the probe never passes, pods never go Ready, and the Service ends up with zero endpoints (full outage) for anyone applying this as-is. The /metrics scrape annotations (#191) and the docs/security.md links (#186) are in the same boat. Could we gate these on the siblings, or add a one-line caveat in the manifest (e.g. "readiness/metrics require v1.x+ — see #187/#191")? The description mentions #169/#173, but those are the issue numbers and read as already-delivered.

Everything else looks solid to me.

[dkijania]

Thanks @SanabriaRusso — both fixed in d3d2afe.

The /readiness outage. This was the one worth catching: :latest resolves to 0.0.6 today, so the probe would 404 forever, no pod would reach Ready, and the Service would sit with zero endpoints — a total outage from the manifest we're offering as the copy-paste reference. Rather than gate on siblings or add an issue-number caveat, both manifests now pin 1.0.0 and state the requirement in a banner at the top. Pinning is what the file already told people to do while doing the opposite, and it makes the version contract enforced rather than advisory. The /metrics annotations had the identical dependency and are covered by the same note.

CORS. Agreed and done — it now ships set rather than commented out, since the primary consumers are cross-origin browsers. I used * for the reference (this is a public read API; docs/security.md in #186 now explains when an allowlist is the better call) and the inline comment spells out both options plus the silent-failure mode. The inaccurate "leave unset for same-origin only" note is gone from both files.

Three more found while in here:

1. deploy/README.md told operators to "point PG_CONN at read replicas for throughput" — same false claim you flagged on P2: Operations runbook — SLOs, capacity, incidents, failover (#180) #197. I verified it against the driver: postgres.js scopes hostIndex per Connection (src/connection.js:89), so every pooled connection starts at host[0] and only advances on failure. Failover, not fan-out. Now points at a real balancer.

2. terminationGracePeriodSeconds: 30 was documented as matching the app's shutdown window (SHUTDOWN_TIMEOUT_MS, 10s). It needs to exceed it, which is the property that actually matters — 30 was already fine, the comment was wrong. SHUTDOWN_TIMEOUT_MS is now set explicitly so the relationship is visible rather than implied.

3. Set TRUST_PROXY: '1' in the k8s manifest — an ingress adds a hop, and after the P0: Add per-IP request rate limiting (#166) #185 fix the default of 0 ignores X-Forwarded-For and buckets every client behind the ingress together.

Verified with kubectl apply --dry-run=client (all four resources) and docker compose config.

One thing to confirm: if #191 adopts the ENABLE_METRICS gating you suggested, this manifest needs ENABLE_METRICS=true for the scrape annotations to do anything. I've left it out rather than bake in a decision that hasn't been made — worth a note on #191 so whoever lands it remembers this file.

[SanabriaRusso]

Verdict: MERGEABLE ✅

New files only, nothing overwritten, so nothing here can break a running deployment. The two points from @SanabriaRusso are genuinely addressed at d3d2afe. Everything below is a sizing/behaviour note on a file operators will copy-paste — none of it blocks.

What I checked

• HARD CONSTRAINT Add a Dockerfile to build and run the server #2 (CORS) is satisfied, and durably. deploy/kubernetes.yaml:59-60 and deploy/docker-compose.prod.yml:19 both ship CORS_ORIGIN: '*' as a set value, not commented out. That means this reference keeps working after P0: Secure CORS default instead of '*' (#167) #184 flips the unset default to deny — which is exactly the failure mode the maintainer flagged. mina-explorer POSTs cross-origin from its own UI origin with no proxy (mina-explorer/src/services/api/client.ts), so this was the one that mattered.
• Probe paths are real. Liveness /healthcheck matches Yoga's built-in healthCheckEndpoint (src/server/server.ts:20) and is not DB-dependent — correct, a DB blip won't crash-loop the fleet. Readiness /readiness matches READINESS_PATH in P1: Add readiness probe distinct from liveness (#169) #187 (src/server/readiness.ts:12). The >= 1.0.0 banner covers the fact that neither /readiness nor /metrics exists on 0.0.x.
• Grace period vs. drain budget. terminationGracePeriodSeconds: 30 > SHUTDOWN_TIMEOUT_MS: 10000, and P1: Graceful shutdown — drain, flush traces, uncaught handlers (#170) #188 reads that env at src/index.ts:9 with the same 10s default. Correct, and now visible rather than implied.
• Secrets. PG_CONN comes from a Secret via secretKeyRef, never a literal env: value or ConfigMap; the committed value is a CHANGE_ME placeholder. Compose uses ${PG_CONN:?...}. No credential committed.
• YAML/apiVersions. Parses clean — 4 docs (Secret, Deployment, Service, HorizontalPodAutoscaler); apps/v1 and autoscaling/v2 are current. Compose parses clean.

Non-blocking nits

1. Readiness probe can black-hole the Service during ordinary DB slowness. A parallel review raised this; I can confirm it with one correction. failureThreshold is unset, so it's k8s' default of 3, not 1 — a single 5s blip does not eject a pod. But: #187's ping() is SELECT 1 on the shared postgres.js pool with no deadline (src/db/archive-node-adapter/archive-node-adapter.ts:98-100), #182 sets max: 10 and statement_timeout: 30_000 (src/db/archive-node-adapter/postgres-options.ts:33-38), and the explorer's analytics query pulls 2000 blocks (mina-explorer/src/services/api/analytics.ts:14). So the ping can queue behind saturated connections for up to the full 30s statement timeout — which is longer than the 30s that periodSeconds: 10 × failureThreshold: 3 gives you. Every replica hits the same Postgres, so they fail together and the Service drops all endpoints. Downstream symptom: the ingress 503s or refuses, and mina-explorer/src/services/api/client.ts has no retry — a transport error surfaces as an error state, not a body with errors[].

          livenessProbe:
            httpGet:
              path: /healthcheck
              port: 8080
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readiness
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 3          # must be < periodSeconds
            failureThreshold: 6        # 60s of DB slowness before losing endpoints
            successThreshold: 1
          startupProbe:                # replaces initialDelaySeconds; no restart while booting
            httpGet:
              path: /healthcheck
              port: 8080
            periodSeconds: 3
            failureThreshold: 20       # 60s to start

60s of tolerance sits comfortably outside #182's 30s statement_timeout, so a query that Postgres will cancel anyway can't take the fleet out first. Worth a note on #187 that ping() should carry its own deadline (sql\SELECT 1`.timeout(2)`) so readiness reports "DB unreachable" rather than "pool busy" — that's the real fix; this is the safety margin on the manifest side.

2. No preStop hook — every rolling deploy drops in-flight requests. k8s removes the pod from Endpoints and sends SIGTERM concurrently, and #188's handler calls server.close() + server.closeIdleConnections() immediately (src/index.ts:22-31). For the ~1-5s until endpoint removal propagates to kube-proxy/ingress, requests still routed to the pod get an RST. Same downstream symptom as above: no errors[] body, so the explorer shows an error rather than degrading.

          lifecycle:
            preStop:
              # Endpoint removal is async and concurrent with SIGTERM. Sleep long
              # enough for kube-proxy/ingress to stop routing here BEFORE the app
              # stops accepting connections. Node has no shell-less sleep, so use
              # the sleep binary; on a distroless base use `httpGet` to a drain path.
              exec:
                command: ['/bin/sh', '-c', 'sleep 15']

With a 15s preStop the budget becomes 15 + 10 (SHUTDOWN_TIMEOUT_MS) = 25s against a 30s grace period — tight. Bump it:

      terminationGracePeriodSeconds: 45   # 15s preStop + 10s drain + margin

3. Applying this file a second time clobbers the operator's real PG_CONN. The Secret and the Deployment share one file, so the documented kubectl apply -f deploy/kubernetes.yaml — the normal loop for changing a replica count or image tag — resets PG_CONN to CHANGE_ME and every pod restarts unable to reach Postgres. Split it: move the Secret to deploy/secret.example.yaml, and change the README snippet to

# once, with your real connection string (never committed):
kubectl create secret generic archive-node-api \
  --from-literal=PG_CONN='postgres://archive_api_ro:...@postgres:5432/archive'
kubectl apply -f deploy/kubernetes.yaml

4. The HPA scales out at 175m CPU. HPA averageUtilization is computed against requests, not limits. With requests.cpu: 250m and target: 70%, the threshold is 175m — a single GraphQL parse crosses that, so this pins to maxReplicas: 6 under any real traffic. Separately, a CPU limit on Node.js means CFS throttling (libuv threadpool + GC on one core) shows up as p99 latency spikes, not as CPU pressure. Suggest:

          resources:
            requests:
              cpu: '500m'
              memory: '512Mi'
            limits:
              # No CPU limit: CFS throttling hurts Node p99 more than the noisy-
              # neighbour risk it prevents. The request is the scheduling guarantee.
              memory: '1Gi'

5. Memory sizing + no NODE_OPTIONS. 512Mi is thin for this workload: with PG_MAX_CONNECTIONS: 10 (#182) the server can be materialising up to 10 concurrent result sets, and the explorer's worst case is 2000 blocks with transactions { userCommands { hash } zkappCommands { hash } } fanned out into JS objects and then a JSON string. More importantly the heap ceiling isn't pinned, so it's whatever Node infers from the cgroup — pin it below the container limit so the failure mode is a recoverable JS heap error (a 500 with a body the client can read) instead of an OOMKill that takes every other in-flight request with it:

            - name: NODE_OPTIONS
              # Keep the V8 heap under limits.memory (1Gi) so we hit a recoverable
              # heap error before the kernel OOM-kills the pod mid-request.
              value: '--max-old-space-size=768'

Mirror in docker-compose.prod.yml: mem_limit: 1g + NODE_OPTIONS: '--max-old-space-size=768'.

6. replicas: 2 + minReplicas: 2 claims HA, but there's no PDB. A node drain can evict both at once.

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: archive-node-api
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: archive-node-api

Also drop replicas: 2 from the Deployment once the HPA owns it (minReplicas: 2 already sets the floor; leaving both means the next kubectl apply scales you back to 2 and the HPA scales back up).

7. RATE_LIMIT_MAX is left to the default; worth setting explicitly with the replica caveat. #185 defaults to 600/60s per client key (src/server/rate-limit.ts:36-40), counted in-process, so the effective ceiling is replicas × 600 — and with the HPA moving 2→6 that ceiling silently varies 1200–3600/min. mina-explorer-api budgets ~4 archive req/s (app/config.py) plus an indexer/backfill worker, and both share one egress IP, so they land in one bucket: 240/min from the API alone, more with the indexer, against a per-replica 600 that a load balancer will not split evenly. The default holds today but it's close enough to be worth stating rather than inheriting:

            - name: RATE_LIMIT_MAX
              # Per client IP per 60s, counted PER REPLICA (in-process counter,
              # #185) — the effective global limit is this × replicas, and the HPA
              # moves replicas 2→6. Server-side consumers (mina-explorer-api and
              # its indexer) share one egress IP and land in one bucket.
              value: '1200'

TRUST_PROXY: '1' is the right call and makes this key on the real client — good catch in d3d2afe.

8. The root README.md:94 still carries the read-replica claim you fixed in deploy/README.md. "point PG_CONN at multiple read replicas — the server fans queries across them" — same false statement, two lines above the ## Deployment section this PR adds, so it's in scope for a one-line fix here rather than another PR.

9. Security context is in good shape; only additions, no gaps that matter: runAsGroup: 1001 + fsGroup: 1001 alongside runAsUser, automountServiceAccountToken: false on the pod spec (this workload never calls the API server), and — because readOnlyRootFilesystem: true — an emptyDir at /tmp as cheap insurance for anything that wants scratch space.

10. Compose relies on the image's HEALTHCHECK (added by #189, Dockerfile HEALTHCHECK ... /healthcheck). That's consistent with the >= 1.0.0 banner, but the comment reads as a statement about today's image — worth the same "requires >= 1.0.0" qualifier the other lines got. Note it's a liveness check: Compose won't route around a container whose DB is down, so /readiness has no analogue there.

On your open question: yes, if #191 lands ENABLE_METRICS gating, this file needs ENABLE_METRICS: 'true' or the prometheus.io/scrape annotations at deploy/kubernetes.yaml:33-36 point at a 404. Leaving it out until that decision lands is the right call.

Automated second-pass review — focus: downstream compatibility with mina-explorer / mina-explorer-api.

[SanabriaRusso]

Approving on the basis of the second-pass review comment above: no mid-to-high severity security, compatibility, or degradation issue found, and the downstream contract with mina-explorer / mina-explorer-api holds — GraphQL validation error text reaches errors[].message verbatim, the browser SPA's cross-origin access is preserved, and the real consumer query shapes (including the 2000-block analytics query and the 500-row page crawl) still pass.

Two things this approval does not mean:

• It does not close the non-blocking items in the review comment. Several are worth fixing before or shortly after merge; they are written up there with patches.
• It does not by itself mean the branch is ready to merge. main requires branches to be up to date, so this needs an update-branch (or a rebase, if the branch is conflicting) first, and a few PRs in this series have cross-PR ordering constraints called out in their review comments.

Automated second-pass review — focus: downstream compatibility with mina-explorer / mina-explorer-api.

[SanabriaRusso]

All four round-1 defects are genuinely fixed. Approving.

Verified fixed

• preStop + grace period — deploy/kubernetes.yaml:90-95 adds preStop: exec ['/bin/sh','-c','sleep 15'], :131 sets terminationGracePeriodSeconds: 45, :69-71 sets SHUTDOWN_TIMEOUT_MS: '10000'. 15 s drain window + 10 s app drain = 25 s inside a 45 s grace, 20 s margin. /bin/sh and sleep are present (#189's base is node:20-alpine, busybox). This closes round 1's cross-cutting gap where every rolling deploy RST'd in-flight requests — that item had no owner, and this PR takes it.
• Secret split out — deploy/secret.example.yaml is its own file, kubernetes.yaml contains only Deployment/Service/HPA/PDB (confirmed: grep -n 'kind:' returns no Secret), and deploy/README.md:38-41 creates the secret imperatively once, then applies -f deploy/kubernetes.yaml — a single file, not a -f deploy/ directory loop. A routine re-apply can no longer clobber a real PG_CONN.
• HPA/resources coherent — request cpu: '500m' with no CPU limit (:86-89, with an explanatory comment) and HPA at averageUtilization: 70 (:165) means scale-out at 350m of the request, with no CFS throttling. That is the right call for a latency-sensitive Node service.
• Probe arithmetic now checks out, given #187's bounded ping (2496d84c):
  • #187's default READINESS_PING_TIMEOUT_MS = 2000, on a dedicated max: 1 client, so the ping cannot head-of-line-block behind a 2000-block analytics query on the shared pool.
  • readinessProbe.timeoutSeconds: 3 (:109) > the 2 s ping bound → the kubelet always gets a real 200/503, never a probe-level timeout.
  • periodSeconds: 10 × failureThreshold: 6 = 60 s of continuous DB unreachability before endpoints are pulled.
  • #182's PG_STATEMENT_TIMEOUT = 15000 caps the worst self-inflicted stall on the shared pool at 15 s — and cannot touch the readiness client at all. 60 s > 15 s by 4×, so a wave of statement-timeout-capped slow queries can no longer correlate all replicas NotReady. That was round 1's total-outage path; it is closed.
  • Liveness /healthcheck 15 s × 3 = 45 s and DB-free, so a DB outage never restarts pods. Startup 3 s × 20 = 60 s boot budget.
• Required sibling env vars are all set — CORS_ORIGIN: '*' (mandatory under #184's secure default, and valid because #184 sets credentials: false; without it this manifest would ship a deployment that blocks mina-explorer's browser client outright), TRUST_PROXY explicitly set in both artifacts (mandatory under #185, which disables the limiter entirely when unset), ENABLE_METRICS: 'true' matching #191's === 'true' gate.

Ordering, not a defect: ghcr.io/o1-labs/archive-node-api:1.0.0 does not exist yet — the current GHCR tag list is 0.0.1-test, latest, 0.0, 0, 0.0.1, 0.0.2, 0.0.4, 0.0.5, 0.0.8, 0.0.6, 0.0.9. Unlike npm (see #208), Docker publishing does work — build.yaml pushes GHCR tags on refs/tags/v* and 0.0.9 is there — so 1.0.0 appears the moment v1.0.0 is tagged. Just make sure v1.0.0 is cut from a main that already contains #187 (/readiness), #191 (/metrics), #188 (SHUTDOWN_TIMEOUT_MS), #184 and #185 — otherwise this manifest's probes 404 and its env vars are no-ops against the image it names.

Non-blocking nits

1. deploy/README.md:46 repeats the incorrect claim (which I have blocked #186 on) that TRUST_PROXY=0 is "the safe default … leaving it unset behind an ingress collapses every client into a single rate-limit bucket." Under #185 as shipped there is no default: useRateLimit() returns a no-op plugin when TRUST_PROXY is unset, so rate limiting is disabled outright, not collapsed. Suggested:

   …but the API disables rate limiting entirely while TRUST_PROXY is unset (there is no safe default), so it must be set explicitly — 0 for a directly exposed server, or the real hop count behind a gateway.

2. deploy/docker-compose.prod.yml has no stop_grace_period. Docker's default is 10 s, but #188's SHUTDOWN_TIMEOUT_MS default is 20000 — so every docker compose up -d redeploy SIGKILLs mid-drain, which is the same defect this PR just fixed on the k8s side. Add, mirroring the k8s reasoning:

       stop_grace_period: 30s

3. The TRUST_PROXY: '1' comment (kubernetes.yaml:73-76) says "set it to the real hop count" but doesn't name the case that bites, and its first clause ("Rate limiting keys on the socket address while this is 0") describes behaviour #185 no longer has. A GKE/GCP external ALB appends two XFF entries, so 1 keys every client on the forwarding-rule IP — which is exactly round 1's blocker, reproduced. #185's warning text, #186 and #197 all say 2. Worth correcting here since this is the artifact people copy:

             # Number of proxy hops in front of the API. A single in-cluster
             # ingress is 1; a GCP external ALB is 2 (client IP, then
             # forwarding-rule IP), plus 1 per extra hop. Too low collapses every
             # client into one bucket; leaving it unset disables rate limiting.

4. PG_MAX_CONNECTIONS is unset, so #182's default of 10 applies: at the HPA's max of 6 replicas that is 60 connections, plus one readiness client each (#187), against a stock max_connections = 100. Worth pinning explicitly next to NODE_OPTIONS so the sizing is visible.

5. The Compose example publishes 8080:8080 with ENABLE_METRICS: 'true', which on a 0.0.0.0-bound host makes /metrics publicly reachable. The k8s path is fine — it uses prometheus.io/* pod annotations, so Prometheus scrapes the pod IP rather than the Service. Worth a line in the Compose comment.

Downstream: none. CORS_ORIGIN: '*' keeps mina-explorer's browser SPA working. mina-explorer-api is server-side and unaffected by CORS; its { __typename } probe and 20 s client timeout are untouched. No schema or error-text surface.

[SanabriaRusso]

Re-approving after the rebase (the previous approval was dismissed by the force-push).

Re-verified the delta against the commit I approved (a6b9328). The deploy/ manifests — the substance of this PR — are unchanged. The only difference is in README.md: the read-replica prose this PR used to correct has since landed on main via another PR, so the branch no longer carries that hunk (correct, not a drop), and the remaining churn is prettier table reformatting plus this PR's own new ## Deployment section. I diffed the documented env-var sets both ways — nothing was dropped. All checks green.

Carrying forward the round-2 verification unchanged:

• preStop sleep 15 + terminationGracePeriodSeconds 45 + SHUTDOWN_TIMEOUT_MS 10000 (25s inside 45s) — this closes round 1's unowned cross-cutting gap.
• The Secret is its own file and the README applies -f kubernetes.yaml (a single file, not a directory loop), so a re-apply cannot clobber a real PG_CONN.
• HPA/resources coherent: 500m request, no CPU limit, scale at 70%.
• Probe arithmetic checks out: 2s bounded ping < 3s probe timeout; 10s × 6 = 60s tolerance vs #182's 15s statement_timeout, a 4x margin.
• Sets CORS_ORIGIN and TRUST_PROXY explicitly — both are mandatory once #184 and #185 land.

Non-blocking:

1. deploy/README.md:46 still calls TRUST_PROXY=0 "the safe default". #185 as shipped gives it no default — unset means the limiter is not installed at all. Prose only; the manifests set the var, so applying them works. This is the same underlying misconception blocking #186 — worth fixing all of them in one pass.
2. docker-compose.prod.yml has no stop_grace_period, so Docker's 10s default SIGKILLs mid-drain against SHUTDOWN_TIMEOUT_MS 10000.
3. deploy/kubernetes.yaml still ships TRUST_PROXY '1'; a GCP external ALB appends two X-Forwarded-For entries and needs 2.
4. The k8s probe failureThreshold is 6 while #187's docs say 3 — pick one.

Release ordering: this pins ghcr.io/...:1.0.0, which does not exist yet. Cut v1.0.0 only from a main that already contains #184, #185 and #188, or these manifests set env vars the image ignores.