fix(metrics): vt-aware oldest_claimable_age_sec — one delayed job no longer reads as a degraded queue

CHANGELOG.md

@@ -1,5 +1,9 @@
 ## [Unreleased]
 
+### Fixed
+
+- **Queue-age metrics no longer count vt-parked (scheduled/retrying) messages — one delayed job stops reading as a degraded queue (issue #389).** ⚠️ **Behavior change on the AppSignal `pgbus_queue_latency` gauge.** pgmq's `oldest_msg_age_sec` is computed from `enqueued_at` and ignores `vt`, but a job enqueued with `wait:` or parked on a long retry backoff lives in the queue table with a future `vt` — that *is* the delayed-delivery mechanism. So a single parked message made the age metric grow at wall-clock rate for hours on an otherwise drained queue, and any latency alert thresholding on it fired continuously ("oldest message is 17045s old" on a healthy queue with depth 1, `read_ct` 0). Every metrics surface now also exposes **`oldest_claimable_age_sec`** — `now() - min(vt)` over rows with `vt <= now()`, i.e. the age of the oldest message actually *eligible for pickup*: an immediately-enqueued message contributes from enqueue time (matching the old number on a plain backlog), a scheduled/backoff-parked message contributes nothing until due, an in-flight message (vt pushed forward) is excluded, and nil means "no claimable backlog" even when the table is non-empty. Surfaces: `Web::DataSource` (dashboard, JSON API, MCP `pgbus_queues` tool), a new Prometheus gauge `pgbus_queue_oldest_claimable_age_seconds`, a new AppSignal gauge `pgbus_queue_oldest_claimable_age_seconds`, `Pgbus::Client#oldest_claimable_ages` (raw-SQL reader, since pgmq's `metrics_result` type is frozen upstream), and a CLAIMABLE column in `pgbus queues`. The AppSignal **`pgbus_queue_latency` gauge now derives from the claimable age** and always emits — `(claimable_age || 0) * 1000`, 0 = no claimable backlog — so existing latency alerts stop false-firing with no dashboard changes; the raw `pgbus_queue_oldest_message_age_seconds` gauge keeps its enqueue-time semantics everywhere. The dashboard queue tables additionally split depth into **Parked** (`depth − visible`) and show the claimable age in place of the raw age, so a queue holding only backoff retries reads visibly healthy. Refs #389.
+
 ### Added
 
 - **Health-checked rolling restarts for the job container (issue #386).** ⚠️ **Behavior change on the standalone `/readyz`.** The supervisor's `health_port` server previously answered `/readyz` with the cluster-wide HealthAnalyzer verdict — so during a rolling deploy a freshly-booted container could pass an orchestrator's health gate on the strength of the *old* container's still-heartbeating workers, and the old container (with all its capacity) was stopped before the new one had forked a single child. The standalone `/readyz` is now **container-local**: 200 only when *this* supervisor verified its connection, bootstrapped queues, forked every configured child, and all of them are currently alive — with 503 bodies `BOOTING` (pre-boot), `DEGRADED` (a child died and is waiting out crash-restart backoff — precisely the state a deploy gate must fail on, keeping the old container running), and `DRAINING` (stop signal received). No database access on the probe path; the supervisor publishes an immutable snapshot per monitor pass and the accept thread reads it. The Rails-mounted `Pgbus::Web::HealthApp` keeps the cluster-wide verdict unchanged. Alongside it: **`pgbus-health`**, a shipped executable probe for docker `HEALTHCHECK` blocks (plain Ruby + stdlib sockets, loads neither Bundler nor the gem — cheap at 1–5s intervals, works in curl-less images; exit 0/1/2 = healthy/unhealthy/usage), and a README "Rolling restarts (Kamal, docker)" guide covering the healthcheck block, stop-timeout alignment, overlap-window duplicate-supervisor safety, and the `read_ct`-vs-deploy-kill DLQ caveat. Refs #386.

README.md

@@ -991,6 +991,7 @@ When `config.metrics_enabled = true` (default), the dashboard exposes Prometheus
 
 | Metric | Description |
 |--------|-------------|
+| `pgbus_queue_oldest_claimable_age_seconds` | Age of the oldest message eligible for pickup (visibility timeout elapsed) — safe to alert on: scheduled/backoff-parked messages don't count until due; the series is omitted entirely when no claimable backlog exists (while the raw `pgbus_queue_oldest_message_age_seconds` gauge may still report a parked message's age) |
 | `pgbus_table_dead_tuples` | Dead tuple count per PGMQ table |
 | `pgbus_table_live_tuples` | Live tuple count per PGMQ table |
 | `pgbus_table_bloat_ratio` | Dead / (dead + live) per table |

app/views/pgbus/dashboard/_queues_table.html.erb

@@ -12,7 +12,8 @@
             <th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.queue") %></th>
             <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.depth") %></th>
             <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.visible") %></th>
-            <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.oldest") %></th>
+            <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.parked") %></th>
+            <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.oldest_claimable") %></th>
             <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.total") %></th>
           </tr>
         </thead>
@@ -25,12 +26,13 @@
               </td>
               <td data-label="Depth" class="px-4 py-3 text-sm text-right text-gray-700"><%= pgbus_number(q[:queue_length]) %></td>
               <td data-label="Visible" class="px-4 py-3 text-sm text-right text-gray-700"><%= pgbus_number(q[:queue_visible_length]) %></td>
-              <td data-label="Oldest" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_msg_age_sec] || "—" %></td>
+              <td data-label="Parked" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:parked_length]) %></td>
+              <td data-label="Oldest claimable" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_claimable_age_sec] || "—" %></td>
               <td data-label="Total" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:total_messages]) %></td>
             </tr>
           <% end %>
           <% if @queues.empty? %>
-            <tr><td colspan="5" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.dashboard.queues_table.empty") %></td></tr>
+            <tr><td colspan="6" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.dashboard.queues_table.empty") %></td></tr>
           <% end %>
         </tbody>
       </table>

app/views/pgbus/queues/_queues_list.html.erb

@@ -6,7 +6,8 @@
           <th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.queue") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.depth") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.visible") %></th>
-          <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.oldest") %></th>
+          <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.parked") %></th>
+          <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.oldest_claimable") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.newest") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.total_ever") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.actions") %></th>
@@ -24,7 +25,8 @@
             </td>
             <td data-label="Depth" class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_length]) %></td>
             <td data-label="Visible" class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_visible_length]) %></td>
-            <td data-label="Oldest" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_msg_age_sec] || "—" %></td>
+            <td data-label="Parked" class="px-4 py-3 text-sm text-right font-mono text-gray-500"><%= pgbus_number(q[:parked_length]) %></td>
+            <td data-label="Oldest claimable" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_claimable_age_sec] || "—" %></td>
             <td data-label="Newest" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:newest_msg_age_sec] || "—" %></td>
             <td data-label="Total" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:total_messages]) %></td>
             <td data-label="Actions" class="px-4 py-3 text-sm text-right space-x-2">
@@ -51,7 +53,7 @@
           </tr>
         <% end %>
         <% if @queues.empty? %>
-          <tr><td colspan="7" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.queues.queues_list.empty") %></td></tr>
+          <tr><td colspan="8" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.queues.queues_list.empty") %></td></tr>
         <% end %>
       </tbody>
     </table>

app/views/pgbus/queues/show.html.erb

@@ -13,6 +13,9 @@
       <p class="text-sm text-gray-500 mt-1">
         <%= t("pgbus.queues.show.depth") %> <span class="font-mono"><%= @queue[:queue_length] %></span> |
         <%= t("pgbus.queues.show.visible") %> <span class="font-mono"><%= @queue[:queue_visible_length] %></span> |
+        <%= t("pgbus.queues.show.parked") %> <span class="font-mono"><%= @queue[:parked_length] %></span> |
+        <%= t("pgbus.queues.show.oldest") %> <span class="font-mono"><%= @queue[:oldest_msg_age_sec] || "—" %></span> |
+        <%= t("pgbus.queues.show.oldest_claimable") %> <span class="font-mono"><%= @queue[:oldest_claimable_age_sec] || "—" %></span> |
         <%= t("pgbus.queues.show.total") %> <span class="font-mono"><%= pgbus_number(@queue[:total_messages]) %></span>
       </p>
     <% end %>

config/locales/da.yml

@@ -65,7 +65,8 @@ da:
         empty: Ingen køer fundet
         headers:
           depth: Dybde
-          oldest: Ældste (s)
+          oldest_claimable: Ældste tilgængelig (s)
+          parked: Parkeret
           queue: Kø
           total: Total
           visible: Synlig
@@ -475,7 +476,8 @@ da:
           actions: Handlinger
           depth: Dybde
           newest: Nyeste (s)
-          oldest: Ældste (s)
+          oldest_claimable: Ældste tilgængelig (s)
+          parked: Parkeret
           queue: Kø
           total_ever: Total nogensinde
           visible: Synlig
@@ -516,6 +518,9 @@ da:
           scheduled: 'Planlagt:'
           timezone: 'Tidszone:'
           visible_at: 'Synlig fra:'
+        oldest: 'Ældste:'
+        oldest_claimable: 'Ældste tilgængelig:'
+        parked: 'Parkeret:'
         pause: Pause
         pause_confirm: Pause behandling?
         purge_confirm: Rens alle beskeder?

config/locales/de.yml

@@ -65,7 +65,8 @@ de:
         empty: Keine Warteschlangen gefunden
         headers:
           depth: Tiefe
-          oldest: Älteste (s)
+          oldest_claimable: Älteste verfügbar (s)
+          parked: Geparkt
           queue: Warteschlange
           total: Gesamt
           visible: Sichtbar
@@ -475,7 +476,8 @@ de:
           actions: Aktionen
           depth: Tiefe
           newest: Neueste (s)
-          oldest: Älteste (s)
+          oldest_claimable: Älteste verfügbar (s)
+          parked: Geparkt
           queue: Warteschlange
           total_ever: Insgesamt jemals
           visible: Sichtbar
@@ -516,6 +518,9 @@ de:
           scheduled: 'Geplant:'
           timezone: 'Zeitzone:'
           visible_at: 'Sichtbar ab:'
+        oldest: 'Älteste:'
+        oldest_claimable: 'Älteste verfügbar:'
+        parked: 'Geparkt:'
         pause: Pause
         pause_confirm: Verarbeitung pausieren?
         purge_confirm: Alle Nachrichten löschen?

config/locales/en.yml

@@ -65,7 +65,8 @@ en:
         empty: No queues found
         headers:
           depth: Depth
-          oldest: Oldest (s)
+          oldest_claimable: Oldest claimable (s)
+          parked: Parked
           queue: Queue
           total: Total
           visible: Visible
@@ -475,7 +476,8 @@ en:
           actions: Actions
           depth: Depth
           newest: Newest (s)
-          oldest: Oldest (s)
+          oldest_claimable: Oldest claimable (s)
+          parked: Parked
           queue: Queue
           total_ever: Total Ever
           visible: Visible
@@ -516,6 +518,9 @@ en:
           scheduled: 'Scheduled at:'
           timezone: 'Timezone:'
           visible_at: 'Visible at:'
+        oldest: 'Oldest:'
+        oldest_claimable: 'Oldest claimable:'
+        parked: 'Parked:'
         pause: Pause
         pause_confirm: Pause processing?
         purge_confirm: Purge all messages?

config/locales/es.yml

@@ -65,7 +65,8 @@ es:
         empty: No se encontraron colas
         headers:
           depth: Profundidad
-          oldest: El más antiguo (s)
+          oldest_claimable: Más antiguo disponible (s)
+          parked: Aparcados
           queue: Cola
           total: Total
           visible: Visible
@@ -475,7 +476,8 @@ es:
           actions: Acciones
           depth: Profundidad
           newest: Más nuevo (s)
-          oldest: Más antiguo (s)
+          oldest_claimable: Más antiguo disponible (s)
+          parked: Aparcados
           queue: Cola
           total_ever: Total acumulado
           visible: Visible
@@ -516,6 +518,9 @@ es:
           scheduled: 'Programado:'
           timezone: 'Zona horaria:'
           visible_at: 'Visible en:'
+        oldest: 'Más antiguo:'
+        oldest_claimable: 'Más antiguo disponible:'
+        parked: 'Aparcados:'
         pause: Pausar
         pause_confirm: "¿Pausar el procesamiento?"
         purge_confirm: "¿Purgar todos los mensajes?"

config/locales/fi.yml

@@ -65,7 +65,8 @@ fi:
         empty: Jonot eivät löytyneet
         headers:
           depth: Syvyys
-          oldest: Vanhin (s)
+          oldest_claimable: Vanhin saatavilla (s)
+          parked: Pysäköidyt
           queue: Jono
           total: Yhteensä
           visible: Näkyvissä
@@ -475,7 +476,8 @@ fi:
           actions: Toiminnot
           depth: Syvyys
           newest: Uusimmat (s)
-          oldest: Vanhimmat (s)
+          oldest_claimable: Vanhin saatavilla (s)
+          parked: Pysäköidyt
           queue: Jono
           total_ever: Yhteensä koskaan
           visible: Näkyvissä
@@ -516,6 +518,9 @@ fi:
           scheduled: 'Aikataulutettu:'
           timezone: 'Aikavyöhyke:'
           visible_at: 'Näkyvissä:'
+        oldest: 'Vanhin:'
+        oldest_claimable: 'Vanhin saatavilla:'
+        parked: 'Pysäköidyt:'
         pause: Tauko
         pause_confirm: Keskeytetäänkö käsittely?
         purge_confirm: Tyhjennetäänkö kaikki viestit?

config/locales/fr.yml

@@ -65,7 +65,8 @@ fr:
         empty: Aucune file d'attente trouvée
         headers:
           depth: Profondeur
-          oldest: Le plus ancien (s)
+          oldest_claimable: Plus ancien disponible (s)
+          parked: Différés
           queue: File d'attente
           total: Total
           visible: Visible
@@ -475,7 +476,8 @@ fr:
           actions: Actions
           depth: Profondeur
           newest: Le plus récent (s)
-          oldest: Le plus ancien (s)
+          oldest_claimable: Plus ancien disponible (s)
+          parked: Différés
           queue: File d'attente
           total_ever: Total jamais
           visible: Visible
@@ -516,6 +518,9 @@ fr:
           scheduled: 'Planifié :'
           timezone: 'Fuseau horaire :'
           visible_at: 'Visible à :'
+        oldest: 'Plus ancien :'
+        oldest_claimable: 'Plus ancien disponible :'
+        parked: 'Différés :'
         pause: Pause
         pause_confirm: Mettre en pause le traitement ?
         purge_confirm: Purger tous les messages ?

config/locales/it.yml

@@ -65,7 +65,8 @@ it:
         empty: Nessuna coda trovata
         headers:
           depth: Profondità
-          oldest: Più vecchio (s)
+          oldest_claimable: Più vecchio disponibile (s)
+          parked: Posticipati
           queue: Coda
           total: Totale
           visible: Visibile
@@ -475,7 +476,8 @@ it:
           actions: Azioni
           depth: Profondità
           newest: Più recente (s)
-          oldest: Più vecchio (s)
+          oldest_claimable: Più vecchio disponibile (s)
+          parked: Posticipati
           queue: Coda
           total_ever: Totale mai
           visible: Visibile
@@ -516,6 +518,9 @@ it:
           scheduled: 'Programmato:'
           timezone: 'Fuso orario:'
           visible_at: 'Visibile alle:'
+        oldest: 'Più vecchio:'
+        oldest_claimable: 'Più vecchio disponibile:'
+        parked: 'Posticipati:'
         pause: Pausa
         pause_confirm: Mettere in pausa l'elaborazione?
         purge_confirm: Eliminare tutti i messaggi?

config/locales/ja.yml

@@ -65,7 +65,8 @@ ja:
         empty: キューが見つかりません
         headers:
           depth: 深さ
-          oldest: 最古 (秒)
+          oldest_claimable: 取得可能な最古 (秒)
+          parked: 待機中
           queue: キュー
           total: 合計
           visible: 表示中
@@ -475,7 +476,8 @@ ja:
           actions: アクション
           depth: 深さ
           newest: 最新 (秒)
-          oldest: 最古 (秒)
+          oldest_claimable: 取得可能な最古 (秒)
+          parked: 待機中
           queue: キュー
           total_ever: 合計数
           visible: 表示中
@@ -516,6 +518,9 @@ ja:
           scheduled: スケジュール済み：
           timezone: タイムゾーン：
           visible_at: 表示可能日時：
+        oldest: '最古:'
+        oldest_claimable: '取得可能な最古:'
+        parked: '待機中:'
         pause: 一時停止
         pause_confirm: 処理を一時停止しますか？
         purge_confirm: すべてのメッセージを削除しますか？

config/locales/nb.yml

@@ -65,7 +65,8 @@ nb:
         empty: Ingen køer funnet
         headers:
           depth: Dybde
-          oldest: Eldste (s)
+          oldest_claimable: Eldste tilgjengelig (s)
+          parked: Parkert
           queue: Kø
           total: Totalt
           visible: Synlig
@@ -475,7 +476,8 @@ nb:
           actions: Handlinger
           depth: Dybde
           newest: Nyeste (s)
-          oldest: Eldste (s)
+          oldest_claimable: Eldste tilgjengelig (s)
+          parked: Parkert
           queue: Kø
           total_ever: Totalt noensinne
           visible: Synlig
@@ -516,6 +518,9 @@ nb:
           scheduled: 'Planlagt:'
           timezone: 'Tidssone:'
           visible_at: 'Synlig fra:'
+        oldest: 'Eldste:'
+        oldest_claimable: 'Eldste tilgjengelig:'
+        parked: 'Parkert:'
         pause: Pause
         pause_confirm: Pause behandling?
         purge_confirm: Rens alle meldinger?