feat: otel-collector pushing curated metrics to Grafana Cloud

[jonathandieu]

Replaces Prometheus's direct remoteWrite to Grafana Cloud with an in-cluster OTel
Collector: it scrapes Prometheus's /federate endpoint for the same curated allowlist
of series (outage canary: up, pod status, node memory, apiserver requests) and exports
via OTLP to Grafana Cloud, reusing the existing grafana-cloud-credentials secret.

Deliberately avoids exposing Prometheus itself for now — that's deferred to Tailscale
(dieubernetes#15). No HTTPRoute, no basic-auth, no network policy in this PR.

Deployed via the infrastructure appset (all clusters), same tier as kube-prometheus-stack.

Note: config.exporters.otlphttp/grafana_cloud.endpoint is an empty placeholder — needs
the real Grafana Cloud OTLP gateway URL filled in before this can actually export data.

[github-actions]

Helm template diff

diff -u --recursive --label base --label head base head
--- base
+++ head
@@ -1239,7 +1239,7 @@
   listenLocal: false
   enableOTLPReceiver: false
   enableAdminAPI: false
-  retention: 10d
+  retention: 30d
   tsdb:
     outOfOrderTimeWindow: 0s
   walCompression: true
@@ -1260,15 +1260,6 @@
       release: release
 
   probeNamespaceSelector: {}
-  remoteWrite:
-    - basicAuth:
-        password:
-          key: password
-          name: grafana-cloud-credentials
-        username:
-          key: username
-          name: grafana-cloud-credentials
-      url: https://prometheus-prod-66-prod-us-east-3.grafana.net/api/prom/push
   securityContext:
     fsGroup: 2000
     runAsGroup: 2000
@@ -1286,6 +1277,15 @@
       release: release
 
   scrapeConfigNamespaceSelector: {}
+  storage:
+    volumeClaimTemplate:
+      spec:
+        accessModes:
+        - ReadWriteOnce
+        resources:
+          requests:
+            storage: 20Gi
+        storageClassName: do-block-storage
   affinity:
     podAntiAffinity:
       preferredDuringSchedulingIgnoredDuringExecution:
Only in /tmp/head: otel-collector.yaml

[jonathandieu]

Splitting into separate platform vs. workloads PRs for clearer review — closing in favor of those.