feat(plane-enterprise): native OpenTelemetry APM support (v2.5.1)#241
feat(plane-enterprise): native OpenTelemetry APM support (v2.5.1)#241pratapalakshmi wants to merge 3 commits into
Conversation
|
Warning Review limit reached
Next review available in: 52 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughThis PR adds OpenTelemetry support to the Plane Enterprise Helm chart. It adds OTEL values, helper-based environment variable injection, app config for tracing and headers, an optional bundled collector, and OTEL-aware workload env rendering. The chart version is bumped to 2.8.0. ChangesOpenTelemetry Observability Integration
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@charts/plane-enterprise/templates/config-secrets/app-env.yaml`:
- Line 122: The OTEL_EXPORTER_OTLP_ENDPOINT value is hardcoded to
"cluster.local" which breaks clusters with custom domains; update the
OTEL_EXPORTER_OTLP_ENDPOINT entry to construct the service FQDN using the same
pattern used elsewhere by interpolating {{ .Release.Name }}, {{
.Release.Namespace }} and the parameterized cluster domain via {{
.Values.env.default_cluster_domain | default "cluster.local" }} so the endpoint
resolves correctly for custom cluster domains while preserving the collector
host and port.
- Around line 121-126: The OTEL_EXPORTER_OTLP_ENDPOINT currently hardcodes port
4317 (gRPC) regardless of .Values.observability.otel.protocol, causing
protocol/port mismatch; update the template that sets
OTEL_EXPORTER_OTLP_ENDPOINT to choose port based on the protocol
(OTEL_EXPORTER_OTLP_PROTOCOL) — use 4317 for "grpc" and 4318 for "http/protobuf"
(or map other protocol values accordingly) when
.Values.observability.otel.collector.enabled is true, keeping the same host
formation ({{ .Release.Name }}-otel-collector.{{ .Release.Namespace
}}.svc.cluster.local) so the exporter speaks the correct port for the configured
protocol.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: dc26bcbe-f3cf-4ff4-8e2b-d3198a45510e
📒 Files selected for processing (4)
charts/plane-enterprise/Chart.yamlcharts/plane-enterprise/templates/config-secrets/app-env.yamlcharts/plane-enterprise/templates/observability/otel-collector.yamlcharts/plane-enterprise/values.yaml
Add first-class OTEL configuration to the backend instead of relying on ad-hoc extraEnv. Bumps chart version to 2.5.1. - values.yaml: new `observability.otel` block (off by default) with a nested `collector` sub-block for an optional bundled OTLP collector. - config-secrets/app-env.yaml: when `observability.otel.enabled`, inject OTEL_* into the backend `-app-vars` ConfigMap (scoped to the six workloads that envFrom it: api, worker, beat-worker, automation-consumer, outbox-poller, migrator). Auth headers go into `-app-secrets`. When the endpoint is blank and the bundled collector is enabled, the backend auto-targets the in-cluster collector Service. - templates/observability/otel-collector.yaml: bundled collector (ConfigMap/Service/Deployment), gated on `observability.otel.collector.enabled`, with an overridable config that defaults to an OTLP-in -> debug-out pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ds + browser tracing The initial OTel commit only instrumented the Django backend (via the app-vars ConfigMap). Bring the chart to parity with the commercial-deployments kustomize otel-observability component: - New plane.otelServiceEnv helper renders the shared OTEL_* keys inline so the workloads that do NOT consume app-vars get full coverage: the Node services (silo, live), the SSR frontend (space) and Plane Intelligence (pi-api, pi-worker, pi-beat). Each reports its own OTEL_SERVICE_NAME. - Django workers (worker, beat-worker, automation-consumer, outbox-poller) get a per-service OTEL_SERVICE_NAME override so their traces are distinguishable from the api (which keeps the configurable serviceName from app-vars). - app-vars gains OTEL_DEBUG_CONSOLE and the browser/client tracing keys (FRONTEND_OTEL_ENABLED / FRONTEND_OTLP_ENDPOINT / FRONTEND_OTLP_HEADERS), which are served to web/admin/space browsers by the API and so belong on the API alone. - values.yaml: add observability.otel.debugConsole and observability.otel.frontend.*. - Fix the bundled otel-collector's plane.labelsAndAnnotations call to the dict signature introduced by the recommended-labels change on master (rebase fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pratapalakshmi
force-pushed
the
chore/add/support/otel
branch
from
e286fd4 to
630c57c
Compare
…col and honor custom cluster domain Address CodeRabbit review on the bundled-collector endpoint (app-vars ConfigMap and the plane.otelServiceEnv helper): - Port was hardcoded to 4317 (gRPC) regardless of OTEL_EXPORTER_OTLP_PROTOCOL, causing a protocol/port mismatch for http/protobuf. Now selects 4317 for grpc and 4318 otherwise; the collector Service already exposes both. - Endpoint host hardcoded "cluster.local", breaking clusters with a custom domain. Now uses .Values.env.default_cluster_domain (default cluster.local), matching every other in-cluster URL in the chart. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
charts/plane-enterprise/templates/observability/otel-collector.yaml (2)
96-115: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winNo liveness/readiness probes on the collector container.
Without probes, a hung or crash-looping-but-not-exiting collector process won't be detected/restarted by Kubernetes, silently dropping telemetry. The
otel/opentelemetry-collector-contribimage exposes a health check extension (typically port 13133) that could back a readiness/liveness probe.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@charts/plane-enterprise/templates/observability/otel-collector.yaml` around lines 96 - 115, The otel collector container is missing Kubernetes health checks, so add readiness and liveness probes to the {{ .Release.Name }}-otel-collector container in the otel-collector template. Use the OpenTelemetry Collector health extension endpoint (typically on port 13133) as the probe target, and wire the probes alongside the existing container fields like args, ports, and volumeMounts so Kubernetes can detect and restart a hung collector.
13-17: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueInconsistent labeling across ConfigMap/Service vs Deployment.
The Deployment uses the
plane.labelsAndAnnotationshelper (line 81), while the ConfigMap and Service only set a rawapp.namelabel. This means ConfigMap/Service miss any standard chart labels (e.g.helm.sh/chart,app.kubernetes.io/*) that the helper likely adds, weakening consistency with the rest of the chart's resources.Also applies to: 55-59, 81-81
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@charts/plane-enterprise/templates/observability/otel-collector.yaml` around lines 13 - 17, The ConfigMap and Service in the otel-collector template are using only a raw app.name label, while the Deployment uses the plane.labelsAndAnnotations helper and therefore gets the chart’s standard labels. Update the ConfigMap and Service metadata to use the same plane.labelsAndAnnotations helper (or equivalent shared labels block) as the Deployment so all three resources carry consistent helm.sh/chart and app.kubernetes.io/* labels, and keep the unique app.name label only if still needed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@charts/plane-enterprise/templates/_helpers.tpl`:
- Around line 272-283: The bundled collector fallback in the helpers template
always uses the gRPC port even when OTEL_EXPORTER_OTLP_PROTOCOL is configured
differently. Update the OTEL_EXPORTER_OTLP_ENDPOINT logic in the _helpers.tpl
block that sets $otel.endpoint / $otel.collector.enabled to choose the collector
port from $otel.protocol, so the generated endpoint matches grpc versus
http/protobuf. Keep the change localized to the endpoint construction near the
OTEL_EXPORTER_OTLP_PROTOCOL setting.
- Around line 296-299: The OTEL exporter headers in the full workload path are
being rendered inline as a plain container env value, which exposes sensitive
auth data in the Deployment manifest. Update the helpers in _helpers.tpl around
the OTEL_EXPORTER_OTLP_HEADERS rendering so the full path sources headers from a
Secret via valueFrom.secretKeyRef instead of value, reusing or creating an
otel-headers Secret as needed. Keep the existing Django backend secret-based
pattern consistent across full, live, space, and silo/pi-* callers by routing
through the same secret-backed helper/branch.
In `@charts/plane-enterprise/templates/observability/otel-collector.yaml`:
- Around line 87-95: The `template.metadata.annotations` block in the
`otel-collector` Deployment uses an unconditional `now` timestamp, which forces
a rollout on every render. Replace `timestamp: {{ now | quote }}` with a stable
checksum annotation derived from the OTel collector ConfigMap data, and wire it
to the collector template so the pod restarts only when the config changes. Use
the existing `template.metadata.annotations` section in `otel-collector.yaml` to
add the checksum key and remove the time-based value.
In `@charts/plane-enterprise/templates/workloads/pi-api.deployment.yaml`:
- Around line 91-98: The OTLP headers for pi-api are being inlined as a plain
env value through plane.otelServiceEnv, which exposes sensitive auth headers in
the Deployment manifest. Update the pi-api workload to source
OTEL_EXPORTER_OTLP_HEADERS from a Secret via valueFrom.secretKeyRef, matching
the backend secret-backed pattern instead of relying on the default literal
output from plane.otelServiceEnv. Use the existing plane.otelServiceEnv helper
and the pi-api deployment env block to locate the change.
---
Nitpick comments:
In `@charts/plane-enterprise/templates/observability/otel-collector.yaml`:
- Around line 96-115: The otel collector container is missing Kubernetes health
checks, so add readiness and liveness probes to the {{ .Release.Name
}}-otel-collector container in the otel-collector template. Use the
OpenTelemetry Collector health extension endpoint (typically on port 13133) as
the probe target, and wire the probes alongside the existing container fields
like args, ports, and volumeMounts so Kubernetes can detect and restart a hung
collector.
- Around line 13-17: The ConfigMap and Service in the otel-collector template
are using only a raw app.name label, while the Deployment uses the
plane.labelsAndAnnotations helper and therefore gets the chart’s standard
labels. Update the ConfigMap and Service metadata to use the same
plane.labelsAndAnnotations helper (or equivalent shared labels block) as the
Deployment so all three resources carry consistent helm.sh/chart and
app.kubernetes.io/* labels, and keep the unique app.name label only if still
needed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0e4dd953-caa3-4ceb-8945-2ec0d7891340
📒 Files selected for processing (15)
charts/plane-enterprise/Chart.yamlcharts/plane-enterprise/templates/_helpers.tplcharts/plane-enterprise/templates/config-secrets/app-env.yamlcharts/plane-enterprise/templates/observability/otel-collector.yamlcharts/plane-enterprise/templates/workloads/automation-consumer.deployment.yamlcharts/plane-enterprise/templates/workloads/beat-worker.deployment.yamlcharts/plane-enterprise/templates/workloads/live.deployment.yamlcharts/plane-enterprise/templates/workloads/outbox-poller.deployment.yamlcharts/plane-enterprise/templates/workloads/pi-api.deployment.yamlcharts/plane-enterprise/templates/workloads/pi-beat.deployment.yamlcharts/plane-enterprise/templates/workloads/pi-worker.deployment.yamlcharts/plane-enterprise/templates/workloads/silo.deployment.yamlcharts/plane-enterprise/templates/workloads/space.deployment.yamlcharts/plane-enterprise/templates/workloads/worker.deployment.yamlcharts/plane-enterprise/values.yaml
✅ Files skipped from review due to trivial changes (1)
- charts/plane-enterprise/Chart.yaml
🚧 Files skipped from review as they are similar to previous changes (2)
- charts/plane-enterprise/templates/config-secrets/app-env.yaml
- charts/plane-enterprise/values.yaml
| {{- if $otel.headers }} | ||
| - name: OTEL_EXPORTER_OTLP_HEADERS | ||
| value: {{ $otel.headers | quote }} | ||
| {{- end }} |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
OTLP auth headers embedded in plaintext in Deployment manifests for non-Django workloads.
For callers using full (the default, e.g. live, space, and presumably silo/pi-*), OTEL_EXPORTER_OTLP_HEADERS is rendered as an inline literal value: on the container spec rather than via valueFrom.secretKeyRef. This puts auth tokens/API keys directly into the Deployment object (visible via kubectl get deployment -o yaml to anyone with Deployment read access, and persisted in plaintext in etcd absent encryption-at-rest), which contradicts the design intent stated for the Django backend path — where headers are deliberately routed into the -app-secrets Secret rather than the -app-vars ConfigMap specifically to avoid this exposure.
Consider sourcing this value from a Secret (create/reuse an otel-headers Secret and reference it via secretKeyRef) for the full path as well, to match the security posture already established for the Django backend workloads.
💡 Sketch of an alternative using a Secret reference
{{- if $otel.headers }}
- name: OTEL_EXPORTER_OTLP_HEADERS
- value: {{ $otel.headers | quote }}
+ valueFrom:
+ secretKeyRef:
+ name: {{ $ctx.Release.Name }}-otel-secrets
+ key: OTEL_EXPORTER_OTLP_HEADERS
{{- end }}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| {{- if $otel.headers }} | |
| - name: OTEL_EXPORTER_OTLP_HEADERS | |
| value: {{ $otel.headers | quote }} | |
| {{- end }} | |
| {{- if $otel.headers }} | |
| - name: OTEL_EXPORTER_OTLP_HEADERS | |
| valueFrom: | |
| secretKeyRef: | |
| name: {{ $ctx.Release.Name }}-otel-secrets | |
| key: OTEL_EXPORTER_OTLP_HEADERS | |
| {{- end }} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@charts/plane-enterprise/templates/_helpers.tpl` around lines 296 - 299, The
OTEL exporter headers in the full workload path are being rendered inline as a
plain container env value, which exposes sensitive auth data in the Deployment
manifest. Update the helpers in _helpers.tpl around the
OTEL_EXPORTER_OTLP_HEADERS rendering so the full path sources headers from a
Secret via valueFrom.secretKeyRef instead of value, reusing or creating an
otel-headers Secret as needed. Keep the existing Django backend secret-based
pattern consistent across full, live, space, and silo/pi-* callers by routing
through the same secret-backed helper/branch.
| template: | ||
| metadata: | ||
| namespace: {{ .Release.Namespace }} | ||
| labels: | ||
| app.name: {{ .Release.Namespace }}-{{ .Release.Name }}-otel-collector | ||
| annotations: | ||
| timestamp: {{ now | quote }} | ||
| spec: | ||
| {{- include "plane.podScheduling" .Values.observability.otel.collector }} |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- target file (relevant ranges) ---'
nl -ba charts/plane-enterprise/templates/observability/otel-collector.yaml | sed -n '1,220p'
echo
echo '--- checksum patterns in repo ---'
rg -n 'checksum/|now \| quote|sha256sum' charts/plane-enterprise -SRepository: makeplane/helm-charts
Length of output: 237
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- target file (relevant ranges) ---'
sed -n '1,220p' charts/plane-enterprise/templates/observability/otel-collector.yaml | cat -n
echo
echo '--- checksum patterns in repo ---'
rg -n 'checksum/|now \| quote|sha256sum' charts/plane-enterprise -SRepository: makeplane/helm-charts
Length of output: 7360
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- checksum / configmap annotation patterns in repo ---'
rg -n 'checksum/.*config|configmap.*sha256sum|sha256sum.*config|include .*sha256sum|checksum/config' charts -S
echo
echo '--- config-related pod annotations in repo ---'
rg -n 'annotations:|timestamp: \{\{ now \| quote \}\}' charts/plane-enterprise/templates -SRepository: makeplane/helm-charts
Length of output: 216
Avoid the unconditional now annotation.
timestamp: {{ now | quote }} changes on every render, so this Deployment rolls on every upgrade even when the collector config is unchanged. Use a checksum of the ConfigMap data instead so pods restart only when the config changes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@charts/plane-enterprise/templates/observability/otel-collector.yaml` around
lines 87 - 95, The `template.metadata.annotations` block in the `otel-collector`
Deployment uses an unconditional `now` timestamp, which forces a rollout on
every render. Replace `timestamp: {{ now | quote }}` with a stable checksum
annotation derived from the OTel collector ConfigMap data, and wire it to the
collector template so the pod restarts only when the config changes. Use the
existing `template.metadata.annotations` section in `otel-collector.yaml` to add
the checksum key and remove the time-based value.
| {{- if or .Values.extraEnv (include "plane.s3CAEnabled" .) .Values.observability.otel.enabled }} | ||
| env: | ||
| {{- with (include "plane.s3CAEnvVars" .) }} | ||
| {{ . | indent 10 }} | ||
| {{- end }} | ||
| {{- with (include "plane.otelServiceEnv" (dict "context" $ "service" "pi-api")) }} | ||
| {{ . | indent 10 }} | ||
| {{- end }} |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
OTLP auth headers rendered as plaintext env value, unlike the Secret-backed path used for backend workloads.
plane.otelServiceEnv (per _helpers.tpl) emits OTEL_EXPORTER_OTLP_HEADERS as a literal quoted value:
- name: OTEL_EXPORTER_OTLP_HEADERS
value: {{ $otel.headers | quote }}
This is called here without "full" false, so pi-api (and pi-beat, pi-worker, silo) get this literal value baked directly into the Deployment spec. Per the PR objectives, auth headers are otherwise treated as sensitive and routed into -app-secrets for the Django backend path — this inline path breaks that contract, exposing potential OTLP auth tokens (e.g. Bearer/API-key headers) in the Deployment manifest, which is visible to a broader RBAC surface (and GitOps/CI diff logs) than a Secret would be.
Consider sourcing OTEL_EXPORTER_OTLP_HEADERS via valueFrom.secretKeyRef against a dedicated Secret (mirroring the -app-secrets treatment), instead of inlining the raw value in _helpers.tpl.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@charts/plane-enterprise/templates/workloads/pi-api.deployment.yaml` around
lines 91 - 98, The OTLP headers for pi-api are being inlined as a plain env
value through plane.otelServiceEnv, which exposes sensitive auth headers in the
Deployment manifest. Update the pi-api workload to source
OTEL_EXPORTER_OTLP_HEADERS from a Secret via valueFrom.secretKeyRef, matching
the backend secret-backed pattern instead of relying on the default literal
output from plane.otelServiceEnv. Use the existing plane.otelServiceEnv helper
and the pi-api deployment env block to locate the change.
1 participant
Summary
Adds first-class OpenTelemetry APM support to the
plane-enterprisechart so self-hosters can enable backend tracing/metrics/log-correlation via values instead of hand-rolledextraEnv. Pairs with thefeat/otel-api-observabilitywork inplane-ee(which adds theconfigure_otel()bootstrap to the Django backend). Chart version bumped 2.5.0 → 2.5.1.What changed
values.yaml— newobservability.otelblock (off by default) with a nestedcollectorsub-block for an optional bundled OTLP collector.templates/config-secrets/app-env.yaml— whenobservability.otel.enabled, injectsOTEL_*into the backend-app-varsConfigMap. Scoped to the six workloads thatenvFromit (api, worker, beat-worker, automation-consumer, outbox-poller, migrator) — no frontend pods touched. Auth headers (if any) go into-app-secrets. Whenendpointis blank and the bundled collector is enabled, the backend auto-targets the in-cluster collector Service.templates/observability/otel-collector.yaml— bundled collector (ConfigMap/Service/Deployment), gated onobservability.otel.collector.enabled, with an overridableconfigdefaulting to anOTLP-in → debug-outpipeline.How to enable
Point at an external collector instead by setting
observability.otel.endpointand leavingcollector.enabled: false.Testing
Deployed to a live EKS cluster (namespace
gpotel, isolated DB, backend image built fromfeat/otel-api-observability) and drove real traffic.Render gating —
helm template+helm lint:helm lint1 chart(s) linted, 0 chart(s) failedOTEL_*envOTEL_EXPORTER_OTLP_ENDPOINT: ""(external-collector mode)http://<release>-otel-collector.<ns>.svc.cluster.local:4317Live spans received by the bundled collector (
kubectl logs deploy/plane-gpotel-otel-collector):Log ↔ trace correlation: worker/Celery logs carry populated
trace_id(32-hex), confirming theTraceContextFilterpath. (Follow-up, app-side not chart-side: the API request access log currently emits emptytrace_id/span_idbecause it logs outside the active span context — tracked against theplane-eebranch, not this chart.)Notes
debugexporter (verification only) — setobservability.otel.collector.configto export to a real backend (Tempo/Datadog/Honeycomb), or disable it and useobservability.otel.endpoint.otel/opentelemetry-collector-contrib:0.115.1.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Chores