feat(deploy): health-checked rolling restarts — container-local /readyz, aligned shutdown budget, pgbus-health probe

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ### Added
 
+- **Health-checked rolling restarts for the job container (issue #386).** ⚠️ **Behavior change on the standalone `/readyz`.** The supervisor's `health_port` server previously answered `/readyz` with the cluster-wide HealthAnalyzer verdict — so during a rolling deploy a freshly-booted container could pass an orchestrator's health gate on the strength of the *old* container's still-heartbeating workers, and the old container (with all its capacity) was stopped before the new one had forked a single child. The standalone `/readyz` is now **container-local**: 200 only when *this* supervisor verified its connection, bootstrapped queues, forked every configured child, and all of them are currently alive — with 503 bodies `BOOTING` (pre-boot), `DEGRADED` (a child died and is waiting out crash-restart backoff — precisely the state a deploy gate must fail on, keeping the old container running), and `DRAINING` (stop signal received). No database access on the probe path; the supervisor publishes an immutable snapshot per monitor pass and the accept thread reads it. The Rails-mounted `Pgbus::Web::HealthApp` keeps the cluster-wide verdict unchanged. Alongside it: **`pgbus-health`**, a shipped executable probe for docker `HEALTHCHECK` blocks (plain Ruby + stdlib sockets, loads neither Bundler nor the gem — cheap at 1–5s intervals, works in curl-less images; exit 0/1/2 = healthy/unhealthy/usage), and a README "Rolling restarts (Kamal, docker)" guide covering the healthcheck block, stop-timeout alignment, overlap-window duplicate-supervisor safety, and the `read_ct`-vs-deploy-kill DLQ caveat. Refs #386.
+
+### Changed
+
+- **Shutdown budgets are now alignable end-to-end (issue #386).** New `config.shutdown_timeout` bounds how long the supervisor waits for children after forwarding TERM before escalating to SIGKILL — previously a hardcoded 30s, which silently SIGKILLed workers mid-drain the moment `drain_timeout` was raised past it. Default derives `drain_timeout + 5` so the deadline tracks the drain window automatically; an explicit value below `drain_timeout` logs a boot warning. `Consumer#shutdown`'s pool wait (its only drain bound) now follows `config.drain_timeout` instead of a hardcoded 30s, and `Worker#shutdown`'s post-drain residual wait drops from a second full 30s window to 5s — the drain loop already waited `drain_timeout`, and a job still running has proven it won't finish. Rule of thumb: orchestrator stop grace period > `shutdown_timeout` > `drain_timeout`. Refs #386.
+
 - **Streams: one LISTEN connection per web host — `streams_listen_scope` (issue #382).** ⚠️ **Default behavior change.** Previously every Puma worker lazily opened its own dedicated streams LISTEN connection on first SSE use, so a web host pinned one direct connection per worker. Under the new default (`streams_listen_scope = :master`) the `pgbus_streams` Puma plugin runs a **MasterHub** in the preforking master: ONE `Web::Streamer::Listener` on the refcounted union of every worker's stream channels, fanning wakes — **including ephemeral payloads** — out to workers over a Unix domain socket with length-prefixed frames (`Streamer::HubProtocol`). Workers connect lazily (nothing is inherited across fork) and the synchronous `ensure_listening` ack contract is preserved cross-process: a sub is registered before LISTEN executes and acked only after, so the no-lost-broadcast guarantee holds. Backpressure follows the streams rules: durable wakes are droppable at a per-worker cap (they self-heal via `read_after`), **ephemeral wakes are never dropped** — a worker that stops draining is evicted, which triggers its own fallback. **Fallback is per-worker listeners, not loss**: whenever the hub is absent or dies (no `preload_app!`, single-mode Puma, crash, eviction) each worker's `FailoverListener` swaps in a real per-worker `Listener` and re-LISTENs its recorded subscriptions — connection footprint balloons back to pre-#382 levels (census-visible) but no broadcast semantics change; the worker stays local until it recycles. Measured (local PG, n=50): the master→worker hop is noise-level free — single-broadcast SSE roundtrip p50 16.00ms via the hub vs 16.93ms per-worker. **`:master` effectively requires `preload_app!`** (the hub waits for the app's pgbus initializer; without it the deadline expires quietly and workers stay per-worker). **Rollback:** `config.streams_listen_scope = :process`. Refs #382, builds on the #381 patterns.
 
 - **Host-level shared LISTEN: `worker_notify_scope` — the supervisor now owns ONE direct LISTEN connection for the whole host (issue #381).** ⚠️ **Default behavior change.** Previously every worker fork and every consumer fork opened its own dedicated LISTEN connection (`NotifyListener`), so a host's direct-connection footprint scaled with fork count — on transaction-pool PgBouncer platforms those connections come out of the scarcest slice of `max_connections`, and a 5-capsule + 2-consumer host pinned 7. Under the new default (`config.worker_notify_scope = :supervisor`) the supervisor runs a single `NotifyHub`: one `NotifyListener` on the union of every capsule's and consumer's queue channels (wildcards via the shared resolver, consumer sets via the registry), fanning wakes out to forks over per-fork pipes (`W` wake / `H` healthy / `P` degraded bytes; a fork whose pipe reports degraded or reaches EOF falls back to fast polling exactly like a failed local listener). Footprint drops to **1 direct LISTEN connection per job host**, verified by integration test: routing is per-fork (an insert wakes only the forks reading that queue, wildcard capsules unconditionally), and `pg_terminate_backend` on the shared connection is survived — reconnect, re-LISTEN, wakes flow again. **Rollback:** `config.worker_notify_scope = :fork` restores the previous per-fork listeners byte-for-byte. Dedicated LISTEN connections are now census-tagged `application_name=pgbus-listen` so `pg_stat_activity` can count them. Refs #381.

README.md

@@ -1117,14 +1117,18 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 
 ### Health endpoints (liveness / readiness)
 
-For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz` (are queues draining, or is a worker silently wedged?). `/readyz` runs the same `OK` / `DEGRADED` / `STALLED` verdict as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz`. Readiness means different things in the two places the probes are served:
+
+- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+- **Standalone from the supervisor** (`health_port`): `/readyz` is **container-local** — did *this* supervisor finish booting, and are all the children *it* forked alive? That is the signal a rolling deploy's health gate needs; the cluster verdict would let a brand-new container pass on the strength of the *old* container's workers.
 
 | Path | Method | 200 | 503 | Touches DB |
 |---|---|---|---|---|
 | `/livez` | GET | always (`ok`) | never | no |
-| `/readyz` | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (mounted) | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (supervisor) | GET | `OK` — booted, all children live | `BOOTING`, `DEGRADED` (child down), `DRAINING` (stopping) | no |
 
-Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is the verdict JSON, so a probe failure is self-describing in the pod's event log.
+Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is JSON, so a probe failure is self-describing in the pod's event log.
 
 #### Mount in your Rails app
 
@@ -1165,6 +1169,61 @@ readinessProbe:
   httpGet: { path: /readyz, port: 9394 }
 ```
 
+The supervisor's `/readyz` answers from its own state, never the database:
+
+```json
+{ "status": "OK", "expected": 3, "live": 3 }
+```
+
+- `BOOTING` (503) until the connection is verified, queues are bootstrapped, and every configured child has been forked. `expected` is stamped at that instant.
+- `OK` (200) while all expected children are in the fork table. A clean worker recycle never dips the count — the snapshot refreshes after reap-and-restart each monitor pass.
+- `DEGRADED` (503) when a child died and is waiting out crash-restart backoff. During a rolling deploy this is the desired failure mode: a crash-looping replacement never goes ready, so the old container keeps running.
+- `DRAINING` (503) the moment a stop signal arrives.
+
+#### `pgbus-health`: container HEALTHCHECK probe
+
+`pgbus-health` ships with the gem: a dependency-free probe (plain Ruby + stdlib sockets — no Bundler, no Rails, nothing else loaded) that GETs `127.0.0.1:<port>/readyz` and exits `0` on 200, `1` on anything else, `2` on usage errors. Cheap enough for a 1–5s `HEALTHCHECK` interval, and it works in images without curl:
+
+```bash
+pgbus-health --port 9394              # or PGBUS_HEALTH_PORT=9394 pgbus-health
+pgbus-health --port 9394 --path /livez --timeout 2
+```
+
+### Rolling restarts (Kamal, docker)
+
+Kamal distributions with per-role health checks (for example the [`dash` branch](https://github.com/mhenrixon/kamal)) can rolling-restart a non-proxied job role: start the new container, poll its docker `HEALTHCHECK` until healthy, and only then `docker stop` the old one. Wire the pgbus container into that gate:
+
+```yaml
+# config/deploy.yml
+servers:
+  job:
+    hosts: [...]
+    cmd: bin/pgbus start
+    healthcheck:
+      cmd: bin/pgbus-health --port 9394
+      interval: 5s
+      start_period: 30s     # cover Rails boot + queue bootstrap
+    stop_timeout: 45        # must exceed pgbus shutdown_timeout (see below)
+env:
+  clear:
+    PGBUS_HEALTH_PORT: 9394
+```
+
+(`bundle binstubs pgbus` generates `bin/pgbus-health`; adjust the path if your image invokes gem executables differently.)
+
+**The shutdown timeline.** On `docker stop`, SIGTERM reaches the supervisor and readiness flips to `DRAINING`; children stop claiming work and drain in-flight jobs for up to `drain_timeout` (default 30s); the supervisor waits `shutdown_timeout` (default `drain_timeout + 5`) before SIGKILLing stragglers. Align the three knobs outside-in:
+
+```text
+orchestrator stop_timeout  >  pgbus shutdown_timeout  >  pgbus drain_timeout
+        45s                        35s (derived)                30s
+```
+
+If the orchestrator's stop grace period is *shorter* than `shutdown_timeout`, docker SIGKILLs the whole tree mid-drain and the graceful path never gets to finish. Raising `drain_timeout` raises the derived `shutdown_timeout` automatically; raise `stop_timeout` to match.
+
+**The overlap window is safe by construction.** Between "new container healthy" and "old container stopped", two supervisors run against the same database. Nothing double-fires: queue claims use `FOR UPDATE SKIP LOCKED`, `single_active_consumer` queues arbitrate via session-level advisory locks (released the instant a killed process's connection dies), two live recurring schedulers dedup on the `(task_key, run_at)` unique record, and dispatcher maintenance is idempotent. "One scheduler per deployment" is a steady-state rule; a deploy window may briefly violate it without consequence.
+
+**What a hard kill still costs.** Jobs killed past the drain window are redelivered after their visibility timeout (at-least-once holds) — but PGMQ's `read_ct` increments exactly like a logical failure, so a long-running job that straddles *repeated* deploy kills can be pushed to the DLQ without its code ever raising. `zombie_detection` logs exactly this pattern (`read_ct > 1` with no recorded failure). Keep jobs shorter than `drain_timeout`, or raise it (and `stop_timeout`) for queues that can't be. For `idempotent!` event handlers there is a separate crash-window caveat tracked in [#385](https://github.com/mhenrixon/pgbus/issues/385).
+
 ### Boot diagnostics banner
 
 `Supervisor#run` logs a one-block banner right after the heartbeat starts and before queues bootstrap, so a misconfigured deployment states its actual settings instead of forcing an operator to attach a console. Every line is `"[Pgbus] boot:"`-prefixed and renders cleanly under both the `:text` and `:json` log formatters:
@@ -2111,6 +2170,7 @@ Curated headline options for the README. The full operator reference (with types
 | `zombie_detection` | `true` | Detect and reclaim work from crashed workers |
 | `read_timeout` | `30` | Seconds before a single PGMQ read is bounded (libpq `statement_timeout` + `tcp_user_timeout` on a dedicated connection; nil disables) |
 | `drain_timeout` | `30` | Seconds to wait for in-flight jobs during graceful shutdown before abandoning them |
+| `shutdown_timeout` | `drain_timeout + 5` | Seconds the supervisor waits for children after TERM before SIGKILL; an orchestrator's stop grace period must exceed it |
 | `stall_threshold` | `300` | Seconds without progress before a worker is considered stalled |
 | `priority_levels` | `nil` | Number of priority sub-queues (nil = disabled, 2-10) |
 | `default_priority` | `1` | Default priority for jobs without explicit priority |

docs/app/models/config_reference.rb

@@ -106,7 +106,10 @@ module ConfigReference
       { name: "health_bind", type: "String", default: '"127.0.0.1"', desc: "Bind address for the health server." },
       { name: "stall_threshold", type: "Numeric", default: "300", desc: "Seconds without progress before a worker is stalled." },
       { name: "read_timeout", type: "Numeric", default: "30", desc: "Read timeout for worker fetches." },
-      { name: "drain_timeout", type: "Numeric", default: "30", desc: "Seconds to wait for in-flight jobs to finish during graceful shutdown before abandoning them." }
+      { name: "drain_timeout", type: "Numeric", default: "30", desc: "Seconds to wait for in-flight jobs to finish during graceful shutdown before abandoning them." },
+      { name: "shutdown_timeout", type: "Numeric, nil", default: "drain_timeout + 5",
+        desc: "Seconds the supervisor waits for children after TERM before SIGKILL. nil derives drain_timeout + 5; " \
+              "an orchestrator's stop grace period (Kamal stop_timeout, K8s terminationGracePeriodSeconds) must exceed it." }
     ],
     "Streams (SSE)" => [
       { name: "streams_enabled", type: "Boolean", default: "true", desc: "Enable the SSE streams transport." },

docs/app/models/doc.rb

@@ -40,6 +40,7 @@ class Doc
   page "Observability",         group: "Operations"
   page "Performance & tuning",  group: "Operations", slug: "performance-tuning", view: "PerformanceTuning"
   page "Separate database",     group: "Operations", slug: "separate-database", view: "SeparateDatabase"
+  page "Rolling restarts",      group: "Operations"
 
   # Testing
   page "Testing", group: "Testing"

docs/app/views/docs/pages/observability.rb

@@ -199,17 +199,23 @@ def health
     DocsUI::Section("Health endpoints", description: "Liveness and readiness for Kubernetes.") do
       md <<~'MD'
         Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and
-        `/readyz` (are queues draining, or is a worker silently wedged?). `/readyz`
-        runs the same `OK` / `DEGRADED` / `STALLED` verdict as the MCP
-        `pgbus_health` tool — `DEGRADED` deliberately stays ready; only the
-        silent-wedge `STALLED` signal fails readiness.
+        `/readyz`. Readiness means different things in the two places it is served.
+        Mounted in Rails, `/readyz` runs the same cluster-wide `OK` / `DEGRADED` /
+        `STALLED` verdict as the MCP `pgbus_health` tool — `DEGRADED` deliberately
+        stays ready; only the silent-wedge `STALLED` signal fails readiness. Served
+        standalone from the supervisor (`health_port`), `/readyz` is
+        **container-local**: did *this* supervisor finish booting, and are all the
+        children it forked alive? That is the signal a rolling deploy's health gate
+        needs — see [Rolling restarts](/docs/rolling-restarts).
       MD
       DocsUI::Table(
         [ "Path", "Method", "200", "503", "Touches DB" ],
         [
           [ [ :code, "/livez" ], "GET", [ :md, "always (`ok`)" ], "never", "no" ],
-          [ [ :code, "/readyz" ], "GET", [ :md, "verdict `OK` or `DEGRADED`" ],
-            [ :md, "verdict `STALLED`, or DB unreachable (`{\"status\":\"ERROR\"}`)" ], "yes" ]
+          [ [ :md, "`/readyz` (mounted)" ], "GET", [ :md, "verdict `OK` or `DEGRADED`" ],
+            [ :md, "verdict `STALLED`, or DB unreachable (`{\"status\":\"ERROR\"}`)" ], "yes" ],
+          [ [ :md, "`/readyz` (supervisor)" ], "GET", [ :md, "`OK` — booted, all children live" ],
+            [ :md, "`BOOTING`, `DEGRADED` (child down), `DRAINING` (stopping)" ], "no" ]
         ]
       )
       md <<~'MD'