Prometheus and OpenCost are both infra-tier charts, deployed independently to every
cluster (platform, prod, stage) with no shared view across them. Self-hosting Grafana
runs into the same problem: enabling it naively gives three separate Grafanas, each
seeing only its own cluster.
overengineered.dieu.dev (roadmap.md) is meant to show cost, availability, and traffic
across the whole setup, not just the cluster it runs on. Any unified dashboard or cost
aggregation needs some form of cross-cluster reachability, so this is worth deciding
once as shared infra rather than per-feature.
Options considered:
- Tailscale operator per cluster, subnet-router mode (advertise each cluster's CIDR
onto a shared tailnet). Least chart churn going forward, no public exposure.
- Tailscale operator, per-Service exposure instead of subnet router. More granular,
more config per service.
- Public HTTPRoute + basic-auth + network policy per cluster. Cheapest to build, but
internet-facing.
- DigitalOcean VPC peering between the three clusters' VPCs. Native, but unverified
whether DOKS's VPC-native CNI actually routes pod IPs across peered VPCs.
- Thanos-style federation through a shared DO Spaces bucket. Sidesteps networking
entirely for Prometheus, but doesn't obviously help OpenCost's REST API, and adds
more moving parts (sidecar, compactor, querier).
Leaning toward Tailscale subnet router: already used elsewhere in this project, no
public exposure, one mechanism reusable by Grafana, cost aggregation, and anything
else that needs to reach another cluster later.
If Tailscale, note that tailnet ACLs live in the Tailscale admin console, not in this
repo, and should get documented (ADR) as an out-of-band dependency.
Not urgent, not started.
Prometheus and OpenCost are both infra-tier charts, deployed independently to every
cluster (platform, prod, stage) with no shared view across them. Self-hosting Grafana
runs into the same problem: enabling it naively gives three separate Grafanas, each
seeing only its own cluster.
overengineered.dieu.dev (roadmap.md) is meant to show cost, availability, and traffic
across the whole setup, not just the cluster it runs on. Any unified dashboard or cost
aggregation needs some form of cross-cluster reachability, so this is worth deciding
once as shared infra rather than per-feature.
Options considered:
onto a shared tailnet). Least chart churn going forward, no public exposure.
more config per service.
internet-facing.
whether DOKS's VPC-native CNI actually routes pod IPs across peered VPCs.
entirely for Prometheus, but doesn't obviously help OpenCost's REST API, and adds
more moving parts (sidecar, compactor, querier).
Leaning toward Tailscale subnet router: already used elsewhere in this project, no
public exposure, one mechanism reusable by Grafana, cost aggregation, and anything
else that needs to reach another cluster later.
If Tailscale, note that tailnet ACLs live in the Tailscale admin console, not in this
repo, and should get documented (ADR) as an out-of-band dependency.
Not urgent, not started.