From d89f3642e9fa0087579c4e143c8270adf1802461 Mon Sep 17 00:00:00 2001 From: dkijania Date: Mon, 29 Jun 2026 08:29:58 +0200 Subject: [PATCH 1/4] docs: add operations runbook (SLOs, capacity, incidents, failover) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There was one benchmark data point but no runbook, SLOs, or documented failure-mode response. Add docs/runbook.md: - Starting SLOs (availability, p50/p99 latency, error rate). - What to watch (the /metrics RED signals, readiness, structured logs) and suggested alerts. - Scaling & capacity guidance — stateless horizontal scaling, Postgres as the real ceiling, pool-vs-max_connections math. - Multi-host Postgres failover semantics and recovery expectations. - A common-incidents table mapping symptoms to causes and actions. - Deploy/rollback notes tied to graceful shutdown and readiness gating. Linked from the README. An automated replica-failover test is noted as a follow-up (needs a multi-host DB harness). Closes #180. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QSuak9smCHbp4N17xjjLF6 --- README.md | 2 ++ docs/runbook.md | 94 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 docs/runbook.md diff --git a/README.md b/README.md index da37a2a8..0799213d 100644 --- a/README.md +++ b/README.md @@ -94,6 +94,8 @@ CI builds, publishes the npm package with provenance, and pushes Docker tags `1. The bottleneck is the Postgres database, not this server. For production load, point `PG_CONN` at multiple read replicas — the server fans queries across them and recovers automatically as hosts come and go. A recent benchmark on a 12-core / 32 GB box (API + Postgres co-located) sustained ~800 req/s with p99 latency of 39 ms. Use `npm run benchmark` to size your own deployment. +For SLOs, capacity guidance, what to monitor, and incident response, see the [operations runbook](./docs/runbook.md). + ## Contributing - AI coding agents: read [`AGENTS.md`](./AGENTS.md) first. diff --git a/docs/runbook.md b/docs/runbook.md new file mode 100644 index 00000000..da0a1009 --- /dev/null +++ b/docs/runbook.md @@ -0,0 +1,94 @@ +# Operations Runbook + +How to run, observe, and troubleshoot the Archive Node API in production. Pairs +with [`docs/security.md`](./security.md) (deployment contract) and +[`deploy/`](../deploy/) (reference manifests). + +## Service summary + +- **What it is:** a stateless, read-only GraphQL server over an archive-node + Postgres database. It holds no state of its own; every instance is + interchangeable and horizontally scalable. +- **The dependency that matters:** Postgres. Nearly every incident traces back to + the database — availability, latency, or connection capacity. + +## SLOs + +Starting targets — tune to your traffic and DB capacity: + +| Objective | Target | +| ------------------------------------------- | -------- | +| Availability (readiness over 30d) | 99.9% | +| Latency, p99 (typical event/action query) | < 250 ms | +| Latency, p50 | < 50 ms | +| Error rate (5xx + masked unexpected errors) | < 0.1% | + +The reference 12-core / 32 GB single-box benchmark sustained ~800 req/s at p99 +39 ms with API and Postgres co-located (`npm run benchmark`); real numbers depend +on query shape and DB sizing. + +## What to watch + +All from the Prometheus `/metrics` endpoint unless noted: + +| Signal | Metric / source | Watch for | +| ------------ | ----------------------------------------------- | ------------------------------------------------ | +| Request rate | `http_requests_total` | sudden spikes (abuse) or drops (upstream outage) | +| Error rate | `http_requests_total{status=~"5.."}` | sustained > SLO | +| Latency | `http_request_duration_seconds` histogram | p99 regressions | +| Saturation | `http_requests_in_flight` | climbing without draining = backpressure | +| Process | default Node metrics (event loop lag, heap, GC) | event-loop lag, memory growth | +| Readiness | `/readiness` (200/503) | flapping = DB connectivity issues | +| Logs | structured JSON, `requestId` per request | error bursts, slow `durationMs` | + +### Suggested alerts + +- Readiness failing on > 1 replica for > 2 min (DB reachability). +- Error rate > 1% for 5 min. +- p99 latency > 1 s for 10 min. +- `http_requests_in_flight` above a high-water mark for 5 min (saturation). + +## Scaling & capacity + +- **Scale out** by adding replicas — they're stateless (see the HPA in + [`deploy/kubernetes.yaml`](../deploy/kubernetes.yaml)). Rate limiting is + per-instance, so the effective global limit ≈ replicas × `RATE_LIMIT_MAX`. +- **The real ceiling is Postgres.** Add read replicas and point `PG_CONN` at them + before scaling the API further; a larger API fleet against one DB just moves + the bottleneck. +- Tune `PG_MAX_CONNECTIONS` so `replicas × PG_MAX_CONNECTIONS` stays within the + database's `max_connections` (leave headroom for other clients). + +## Multi-host Postgres & failover + +`PG_CONN` accepts multiple hosts (`postgres://host1:5432,host2:5432/archive`). +The `postgres` client connects to an available host and re-establishes +connections as hosts come and go, so a replica dropping out is tolerated without +a restart. Validate the exact behaviour for your topology before relying on it +for HA (an automated failover test is a tracked follow-up). + +Recovery semantics to expect: + +- A dead host: in-flight queries on it fail (surfaced as masked errors); new + connections route to a healthy host. +- `/readiness` returns 503 while no host is reachable, so orchestrators stop + routing traffic until the DB recovers — without killing the (live) pods. + +## Common incidents + +| Symptom | Likely cause | Action | +| ---------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------- | +| `/readiness` 503, `/healthcheck` 200 | Postgres unreachable | check DB health/network; pods recover automatically when it returns | +| p99 latency climbing, `in_flight` rising | slow/expensive queries or DB CPU | check DB load; review slow queries; confirm `PG_STATEMENT_TIMEOUT` is set | +| Many 429s | a client over the rate limit, or limits too low | confirm `X-Forwarded-For` is set by the gateway; adjust `RATE_LIMIT_MAX` | +| Connection-pool exhaustion errors | `PG_MAX_CONNECTIONS` × replicas > DB capacity | lower pool size or raise DB `max_connections` | +| Memory growth / OOM kills | heavy result sets or a leak | lower `BLOCK_RANGE_SIZE`; inspect heap metrics; cap container memory | +| Startup exits immediately | invalid config | read the startup error — config is validated fail-fast (missing `PG_CONN`, bad `PORT`, etc.) | + +## Deploys & rollback + +- Rolling update; `terminationGracePeriodSeconds: 30` lets in-flight requests + drain (the app shuts down gracefully on SIGTERM and flushes traces). +- Readiness gates traffic to new pods until they can reach the DB. +- Roll back by redeploying the previous image tag — the service is stateless and + carries no migrations, so rollback is safe at any time. From c9cf9ec0b355b9b6d089fb4364ce0a128f1fbc3f Mon Sep 17 00:00:00 2001 From: dkijania Date: Fri, 17 Jul 2026 09:45:34 +0200 Subject: [PATCH 2/4] docs(runbook): correct replica scaling, add version scope, TRUST_PROXY row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scaling told operators to "add read replicas and point PG_CONN at them before scaling the API further", which reads as added read capacity. It isn't: postgres.js scopes hostIndex per Connection, so every pooled connection starts at host[0] and only advances on failure. Extra hosts buy redundancy, not throughput — real read scaling needs a balancer in front of Postgres. The failover section now says so plainly rather than leaving "connects to an available host" open to the throughput reading. Adds a version scope note. Nearly everything the runbook says to observe or tune ships in 1.0.0; on 0.0.x, /readiness 404s, the tuning knobs are no-ops, and SIGTERM skips the drain. A runbook that misdirects mid- incident is worse than no runbook, and the published image today is 0.0.6. Scoping by version rather than by in-flight PR numbers keeps the note true after the merge train lands. Splits the 429 incident row: after #185, mass 429s across unrelated clients most likely means TRUST_PROXY is unset behind a gateway, collapsing every client into one bucket — a different fix from a single client exceeding the limit. Addresses review feedback on #197. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/runbook.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/runbook.md b/docs/runbook.md index da0a1009..fec74eb4 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -4,6 +4,14 @@ How to run, observe, and troubleshoot the Archive Node API in production. Pairs with [`docs/security.md`](./security.md) (deployment contract) and [`deploy/`](../deploy/) (reference manifests). +> **Applies to 1.0.0 and later.** Much of what follows — `/readiness`, `/metrics` +> and the `http_*` series, `PG_STATEMENT_TIMEOUT` / `PG_MAX_CONNECTIONS`, +> `RATE_LIMIT_MAX`, and the graceful drain on SIGTERM — does not exist on `0.0.x` +> images: `/readiness` 404s, the tuning knobs are no-ops, and SIGTERM exits +> immediately without draining. **Check your running version before following any +> procedure here mid-incident**, or you'll be diagnosing against endpoints and +> settings your build doesn't have. + ## Service summary - **What it is:** a stateless, read-only GraphQL server over an archive-node @@ -53,17 +61,20 @@ All from the Prometheus `/metrics` endpoint unless noted: - **Scale out** by adding replicas — they're stateless (see the HPA in [`deploy/kubernetes.yaml`](../deploy/kubernetes.yaml)). Rate limiting is per-instance, so the effective global limit ≈ replicas × `RATE_LIMIT_MAX`. -- **The real ceiling is Postgres.** Add read replicas and point `PG_CONN` at them - before scaling the API further; a larger API fleet against one DB just moves - the bottleneck. +- **The real ceiling is Postgres.** A larger API fleet against one DB just moves + the bottleneck. Note that listing replicas in `PG_CONN` does **not** spread + reads across them — that buys failover only (see below). To add read capacity, + put a balancer (PgBouncer, HAProxy, a managed reader endpoint) in front of + Postgres and point `PG_CONN` at it. - Tune `PG_MAX_CONNECTIONS` so `replicas × PG_MAX_CONNECTIONS` stays within the database's `max_connections` (leave headroom for other clients). ## Multi-host Postgres & failover `PG_CONN` accepts multiple hosts (`postgres://host1:5432,host2:5432/archive`). -The `postgres` client connects to an available host and re-establishes -connections as hosts come and go, so a replica dropping out is tolerated without +This is **failover, not load balancing**: every connection starts at the first +host and only moves to the next when its attempt fails, so extra hosts buy +redundancy rather than read throughput. A host dropping out is tolerated without a restart. Validate the exact behaviour for your topology before relying on it for HA (an automated failover test is a tracked follow-up). @@ -80,7 +91,8 @@ Recovery semantics to expect: | ---------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------- | | `/readiness` 503, `/healthcheck` 200 | Postgres unreachable | check DB health/network; pods recover automatically when it returns | | p99 latency climbing, `in_flight` rising | slow/expensive queries or DB CPU | check DB load; review slow queries; confirm `PG_STATEMENT_TIMEOUT` is set | -| Many 429s | a client over the rate limit, or limits too low | confirm `X-Forwarded-For` is set by the gateway; adjust `RATE_LIMIT_MAX` | +| Many 429s, across unrelated clients | `TRUST_PROXY` unset behind a gateway, so every client shares one bucket | set `TRUST_PROXY` to the gateway's hop count (`1` behind a single LB); the app logs a warning on startup when it sees `X-Forwarded-For` with `TRUST_PROXY=0` | +| Many 429s, one client | a client over the rate limit, or limits too low | confirm the gateway sets `X-Forwarded-For`; adjust `RATE_LIMIT_MAX` | | Connection-pool exhaustion errors | `PG_MAX_CONNECTIONS` × replicas > DB capacity | lower pool size or raise DB `max_connections` | | Memory growth / OOM kills | heavy result sets or a leak | lower `BLOCK_RANGE_SIZE`; inspect heap metrics; cap container memory | | Startup exits immediately | invalid config | read the startup error — config is validated fail-fast (missing `PG_CONN`, bad `PORT`, etc.) | From 816a84ce359b9b9fdfbf12d93c4ce106e9609cb6 Mon Sep 17 00:00:00 2001 From: dkijania Date: Wed, 19 Aug 2026 14:33:07 +0200 Subject: [PATCH 3/4] docs(runbook): clarify 1.0 operations caveats --- docs/runbook.md | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/docs/runbook.md b/docs/runbook.md index fec74eb4..08f945b0 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -6,11 +6,11 @@ with [`docs/security.md`](./security.md) (deployment contract) and > **Applies to 1.0.0 and later.** Much of what follows — `/readiness`, `/metrics` > and the `http_*` series, `PG_STATEMENT_TIMEOUT` / `PG_MAX_CONNECTIONS`, -> `RATE_LIMIT_MAX`, and the graceful drain on SIGTERM — does not exist on `0.0.x` -> images: `/readiness` 404s, the tuning knobs are no-ops, and SIGTERM exits -> immediately without draining. **Check your running version before following any -> procedure here mid-incident**, or you'll be diagnosing against endpoints and -> settings your build doesn't have. +> `RATE_LIMIT_MAX`, fail-fast config validation, and the graceful drain on +> SIGTERM — does not exist on `0.0.x` images: `/readiness` 404s, the tuning knobs +> are no-ops, and SIGTERM exits immediately without draining. **Check your +> running version before following any procedure here mid-incident**, or you'll +> be diagnosing against endpoints and settings your build doesn't have. ## Service summary @@ -37,7 +37,8 @@ on query shape and DB sizing. ## What to watch -All from the Prometheus `/metrics` endpoint unless noted: +All from the Prometheus `/metrics` endpoint unless noted. `/metrics` requires +`ENABLE_METRICS=true`; it is off by default. | Signal | Metric / source | Watch for | | ------------ | ----------------------------------------------- | ------------------------------------------------ | @@ -87,20 +88,22 @@ Recovery semantics to expect: ## Common incidents -| Symptom | Likely cause | Action | -| ---------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------- | -| `/readiness` 503, `/healthcheck` 200 | Postgres unreachable | check DB health/network; pods recover automatically when it returns | -| p99 latency climbing, `in_flight` rising | slow/expensive queries or DB CPU | check DB load; review slow queries; confirm `PG_STATEMENT_TIMEOUT` is set | +| Symptom | Likely cause | Action | +| ---------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `/readiness` 503, `/healthcheck` 200 | Postgres unreachable | check DB health/network; pods recover automatically when it returns | +| p99 latency climbing, `in_flight` rising | slow/expensive queries or DB CPU | check DB load; review slow queries; confirm `PG_STATEMENT_TIMEOUT` is set; known consumers give up after 20s, so treat that as the practical latency ceiling | | Many 429s, across unrelated clients | `TRUST_PROXY` unset behind a gateway, so every client shares one bucket | set `TRUST_PROXY` to the gateway's hop count (`1` behind a single LB); the app logs a warning on startup when it sees `X-Forwarded-For` with `TRUST_PROXY=0` | -| Many 429s, one client | a client over the rate limit, or limits too low | confirm the gateway sets `X-Forwarded-For`; adjust `RATE_LIMIT_MAX` | -| Connection-pool exhaustion errors | `PG_MAX_CONNECTIONS` × replicas > DB capacity | lower pool size or raise DB `max_connections` | -| Memory growth / OOM kills | heavy result sets or a leak | lower `BLOCK_RANGE_SIZE`; inspect heap metrics; cap container memory | -| Startup exits immediately | invalid config | read the startup error — config is validated fail-fast (missing `PG_CONN`, bad `PORT`, etc.) | +| Many 429s, one client | a client over the rate limit, or limits too low | confirm the gateway sets `X-Forwarded-For`; adjust `RATE_LIMIT_MAX` | +| Connection-pool exhaustion errors | `PG_MAX_CONNECTIONS` × replicas > DB capacity | lower pool size or raise DB `max_connections` | +| Memory growth / OOM kills | heavy result sets or a leak | lower `BLOCK_RANGE_SIZE`; inspect heap metrics; cap container memory | +| Startup exits immediately | invalid config | read the startup error — config is validated fail-fast (missing `PG_CONN`, bad `PORT`, etc.) | ## Deploys & rollback - Rolling update; `terminationGracePeriodSeconds: 30` lets in-flight requests drain (the app shuts down gracefully on SIGTERM and flushes traces). - Readiness gates traffic to new pods until they can reach the DB. -- Roll back by redeploying the previous image tag — the service is stateless and - carries no migrations, so rollback is safe at any time. +- Roll back within the 1.0.x line — the service is stateless and carries no + migrations. Do **not** roll back to a 0.0.x image while readiness targets + `/readiness`: that path 404s, no pod goes Ready, and the Service loses all + endpoints. From 4298f83650e2335ea572c9567e19ad0002bcd873 Mon Sep 17 00:00:00 2001 From: dkijania Date: Mon, 24 Aug 2026 19:23:08 +0200 Subject: [PATCH 4/4] docs(runbook): clarify trusted proxy hop count --- docs/runbook.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/runbook.md b/docs/runbook.md index 08f945b0..7a0f6ef1 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -88,15 +88,15 @@ Recovery semantics to expect: ## Common incidents -| Symptom | Likely cause | Action | -| ---------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `/readiness` 503, `/healthcheck` 200 | Postgres unreachable | check DB health/network; pods recover automatically when it returns | -| p99 latency climbing, `in_flight` rising | slow/expensive queries or DB CPU | check DB load; review slow queries; confirm `PG_STATEMENT_TIMEOUT` is set; known consumers give up after 20s, so treat that as the practical latency ceiling | -| Many 429s, across unrelated clients | `TRUST_PROXY` unset behind a gateway, so every client shares one bucket | set `TRUST_PROXY` to the gateway's hop count (`1` behind a single LB); the app logs a warning on startup when it sees `X-Forwarded-For` with `TRUST_PROXY=0` | -| Many 429s, one client | a client over the rate limit, or limits too low | confirm the gateway sets `X-Forwarded-For`; adjust `RATE_LIMIT_MAX` | -| Connection-pool exhaustion errors | `PG_MAX_CONNECTIONS` × replicas > DB capacity | lower pool size or raise DB `max_connections` | -| Memory growth / OOM kills | heavy result sets or a leak | lower `BLOCK_RANGE_SIZE`; inspect heap metrics; cap container memory | -| Startup exits immediately | invalid config | read the startup error — config is validated fail-fast (missing `PG_CONN`, bad `PORT`, etc.) | +| Symptom | Likely cause | Action | +| ---------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/readiness` 503, `/healthcheck` 200 | Postgres unreachable | check DB health/network; pods recover automatically when it returns | +| p99 latency climbing, `in_flight` rising | slow/expensive queries or DB CPU | check DB load; review slow queries; confirm `PG_STATEMENT_TIMEOUT` is set; known consumers give up after 20s, so treat that as the practical latency ceiling | +| Many 429s, across unrelated clients | `TRUST_PROXY` unset or too low behind a gateway, so clients share buckets | set `TRUST_PROXY` to the exact trusted-proxy hop count for the topology (GCP external ALB commonly needs `2`); the app logs a warning on startup when it sees `X-Forwarded-For` with `TRUST_PROXY=0` | +| Many 429s, one client | a client over the rate limit, or limits too low | confirm the gateway sets `X-Forwarded-For`; adjust `RATE_LIMIT_MAX` | +| Connection-pool exhaustion errors | `PG_MAX_CONNECTIONS` × replicas > DB capacity | lower pool size or raise DB `max_connections` | +| Memory growth / OOM kills | heavy result sets or a leak | lower `BLOCK_RANGE_SIZE`; inspect heap metrics; cap container memory | +| Startup exits immediately | invalid config | read the startup error — config is validated fail-fast (missing `PG_CONN`, bad `PORT`, etc.) | ## Deploys & rollback