Summary
Substrate hardcodes its OTLP endpoint to the GKE Managed OpenTelemetry collector (opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317) in at least nine places across manifests, a CLI flag, and a best-practices doc. That collector is a centrally-deployed Deployment (single gateway tier), the feature is currently in Preview, and it is GKE-only.
We have no documented answer to any of these:
- Is a single centralized collector Deployment going to scale for Substrate's telemetry shape?
- Should we instead recommend operators run their own collector, and should it be a DaemonSet?
- How does an operator point Substrate at their own collector today?
- What is the recommended collector topology and config?
Today the honest answer to (3) is "patch six manifests and accept that TLS is impossible." That's not a supportable story for anyone running Substrate outside our GKE dev setup.
This issue asks for a written Collector Setup Guide plus the small amount of code/manifest work needed to make that guide actually followable.
Two potential blockers for a bring-your-own collector
- TLS is impossible.
serverboot.go:109,133 hardcode otlptracegrpc.WithInsecure() /otlpmetricgrpc.WithInsecure(), with the comment "GKE managed traces doesn't support validating the TLS certs of the collector." WithInsecure() overrides scheme inference, so setting OTEL_EXPORTER_OTLP_ENDPOINT=https://… will not produce a TLS connection — it will silently stay plaintext. Any operator pointing at an authenticated or off-cluster collector is blocked outright.
- Protocol is fixed to gRPC. The
otlptracegrpc / otlpmetricgrpc exporters are compiled in, so OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf is ignored.
Summary
Substrate hardcodes its OTLP endpoint to the GKE Managed OpenTelemetry collector (
opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317) in at least nine places across manifests, a CLI flag, and a best-practices doc. That collector is a centrally-deployed Deployment (single gateway tier), the feature is currently in Preview, and it is GKE-only.We have no documented answer to any of these:
Today the honest answer to (3) is "patch six manifests and accept that TLS is impossible." That's not a supportable story for anyone running Substrate outside our GKE dev setup.
This issue asks for a written Collector Setup Guide plus the small amount of code/manifest work needed to make that guide actually followable.
Two potential blockers for a bring-your-own collector
serverboot.go:109,133hardcodeotlptracegrpc.WithInsecure()/otlpmetricgrpc.WithInsecure(), with the comment "GKE managed traces doesn't support validating the TLS certs of the collector."WithInsecure()overrides scheme inference, so settingOTEL_EXPORTER_OTLP_ENDPOINT=https://…will not produce a TLS connection — it will silently stay plaintext. Any operator pointing at an authenticated or off-cluster collector is blocked outright.otlptracegrpc/otlpmetricgrpcexporters are compiled in, soOTEL_EXPORTER_OTLP_PROTOCOL=http/protobufis ignored.