From d16ef00eae2534160f7ee8a5a68fae19192b77cc Mon Sep 17 00:00:00 2001 From: Mike Date: Fri, 7 Aug 2026 17:23:01 +0300 Subject: [PATCH] fix(chart): label the Services so the VMServiceScrape can select them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both planes have been UNOBSERVED in production since roughly 13:45 UTC on 2026-08-07, while both pods stayed Healthy and both served traffic normally. WHAT BROKE. A `VMServiceScrape` selects SERVICES by their own `metadata.labels`. The merged chart carries `app: ` only under the Services' `spec.selector`, which selects PODS and is a different field. With no labels on the Service objects themselves, neither scrape matched anything, so no target existed to scrape. The chart this replaced did label the Services, which is why this worked before the merge and stopped after it. Nothing else changed: the ports are named `metrics` on both Services and both Deployments, and the pods emit on 9464 as before. EVIDENCE, measured rather than inferred: - `up{namespace="agent-rpc-mcp"}` returns no series at all; - `sum(up{job=~"agent-rpc-mcp|agent-rpc-mgmt-mcp"})` held a steady 2 until 13:45 UTC and has nothing after it; - `mcp_ankr_build_info` is absent, so "which build is running" is no longer answerable from a dashboard; - the service itself is fine: `initialize` still answers `0.2.0+e9a0b572…`, `/healthz` and `/readyz` are 200, `POST /mcp` is 401. This is an observability outage, not a service outage, which is precisely why nothing paged. THE FIX is `metadata.labels` on both Services, rendered from the same helper the selector already uses, so the two cannot drift apart again. VERIFIED BY RENDERING BOTH WAYS, not by reading: rendering the templates and comparing each `VMServiceScrape.spec.selector.matchLabels` against the matching `Service.metadata.labels` reports False for both planes before this change and True for both after it. Chart version bumped to 0.1.1. `helmChartVersion` in the `agent-rpc-mcp` AppSet in `infrastructure-k8s` needs re-pinning to 0.1.1 for this to reach the cluster. NOTE FOR THE FOLLOW-UP. SHARK-3608 contains an alert for exactly this, `McpNoScrapeTarget` = `absent(up{namespace="agent-rpc-mcp"})` for 10m, whose annotation names the VMServiceScrape and the `metrics` port as the things to check. It is sitting unmerged in infrastructure-observability PR #310. Had it been merged, this would have paged instead of being found by hand. Co-Authored-By: Claude Opus 5 (1M context) --- charts/agent-rpc-mcp/Chart.yaml | 2 +- charts/agent-rpc-mcp/templates/service-data.yaml | 6 ++++++ charts/agent-rpc-mcp/templates/service-mgmt.yaml | 6 ++++++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/charts/agent-rpc-mcp/Chart.yaml b/charts/agent-rpc-mcp/Chart.yaml index a8783f4..4b04b52 100644 --- a/charts/agent-rpc-mcp/Chart.yaml +++ b/charts/agent-rpc-mcp/Chart.yaml @@ -2,4 +2,4 @@ apiVersion: v2 name: agent-rpc-mcp description: Deploys the Agent RPC MCP data plane and management plane as separate pods. type: application -version: 0.1.0 +version: 0.1.1 diff --git a/charts/agent-rpc-mcp/templates/service-data.yaml b/charts/agent-rpc-mcp/templates/service-data.yaml index 2a6e793..95f0dbf 100644 --- a/charts/agent-rpc-mcp/templates/service-data.yaml +++ b/charts/agent-rpc-mcp/templates/service-data.yaml @@ -3,6 +3,12 @@ kind: Service metadata: name: agent-rpc-mcp namespace: {{ .Values.namespace }} + # The VMServiceScrape selects SERVICES by their own metadata labels. + # Without this block the scrape matches nothing and the plane goes dark + # while the pod stays Healthy, which is exactly what happened on + # 2026-08-07: `up{namespace="agent-rpc-mcp"}` returned no series at all. + labels: + {{- include "agent-rpc-mcp.data.labels" . | nindent 4 }} spec: selector: {{- include "agent-rpc-mcp.data.labels" . | nindent 4 }} diff --git a/charts/agent-rpc-mcp/templates/service-mgmt.yaml b/charts/agent-rpc-mcp/templates/service-mgmt.yaml index 2d24163..e23d140 100644 --- a/charts/agent-rpc-mcp/templates/service-mgmt.yaml +++ b/charts/agent-rpc-mcp/templates/service-mgmt.yaml @@ -3,6 +3,12 @@ kind: Service metadata: name: agent-rpc-mgmt-mcp namespace: {{ .Values.namespace }} + # The VMServiceScrape selects SERVICES by their own metadata labels. + # Without this block the scrape matches nothing and the plane goes dark + # while the pod stays Healthy, which is exactly what happened on + # 2026-08-07: `up{namespace="agent-rpc-mcp"}` returned no series at all. + labels: + {{- include "agent-rpc-mcp.mgmt.labels" . | nindent 4 }} spec: selector: {{- include "agent-rpc-mcp.mgmt.labels" . | nindent 4 }}