obs(spireentry): add Prometheus metrics and OTel tracing to entry reconciler [experimental]

[antoinedao-cohere]

Experimental / RFC — this is a draft I'm sharing with colleagues and maintainers for early feedback. It is not intended as a merge-ready PR today, but I think the approach could be genuinely useful and I'd love to hear whether maintainers see value in it.

Motivation

The default controller-runtime / kubebuilder metric tooling only instruments the Kubernetes reconcile loops (queue depth, processing latency, errors). It tells you nothing about what is happening inside a reconcile pass — specifically the SPIRE entry reconciler, which has its own internal loop that is not a standard controller-runtime reconciler.

We are seeing pod attestation delays of up to ~15 minutes in production. A pod is admitted to the cluster but its SPIRE registration entry is not created for a very long time, blocking workload identity issuance. We suspect the controller is the source of the delay but we have no visibility into which phase is slow. This change adds the instrumentation we need to confirm or rule that out.

What this adds

Prometheus metrics (pkg/metrics)

Metric	Type	Labels	What it tells you
spire_controller_manager_reconcile_duration_seconds	Histogram	kind, trigger, result	End-to-end wall-clock time per reconcile pass
spire_controller_manager_reconcile_phase_duration_seconds	Histogram	phase	Per-phase breakdown: list_entries, list_static_entries, list_cluster_spiffeids, build_node_map, render_state, diff, delete, create, update, status_update
spire_controller_manager_spire_api_request_duration_seconds	Histogram	operation	Latency of each SPIRE server RPC (list_entries, batch_create, batch_update, batch_delete)
spire_controller_manager_spire_entry_write_total	Counter	operation, code	Per-RPC result counts (ok, already_exists, not_found, error)
spire_controller_manager_reconcile_trigger_total	Counter	result	Triggers enqueued vs dropped — a high drop rate means new pod events are being coalesced away
spire_controller_manager_entry_render_cache_total	Counter	result	Render cache hit vs miss ratio
spire_controller_manager_pod_attestation_delay_seconds	Histogram	—	Primary user-facing SLI: time from pod.CreationTimestamp to successful SPIRE entry creation
spire_controller_manager_reconcile_batch_entries	Gauge	operation	Batch size at the point each write RPC is issued

Key PromQL queries

# P99 end-to-end reconcile pass duration
histogram_quantile(0.99,
  rate(spire_controller_manager_reconcile_duration_seconds_bucket[5m]))

# Which phase is the bottleneck?
histogram_quantile(0.99,
  rate(spire_controller_manager_reconcile_phase_duration_seconds_bucket[5m]))
  by (phase)

# SPIRE server API latency by operation
histogram_quantile(0.99,
  rate(spire_controller_manager_spire_api_request_duration_seconds_bucket[5m]))
  by (operation)

# Pod attestation SLI — P95 delay
histogram_quantile(0.95,
  rate(spire_controller_manager_pod_attestation_delay_seconds_bucket[15m]))

# Fraction of triggers that are dropped (high = back-pressure)
sum(rate(spire_controller_manager_reconcile_trigger_total{result="dropped"}[5m]))
  /
sum(rate(spire_controller_manager_reconcile_trigger_total[5m]))

Distributed tracing (pkg/tracing)

When spec.tracing.enabled: true is set in the controller config, an OTLP/gRPC trace exporter is initialised and every reconcile pass produces a tree of spans:

entry.reconcile  (trigger=triggered|periodic)
├── entry.list_entries
│   └── grpc.BatchListEntries  ← from otelgrpc
├── entry.list_static_entries
├── entry.list_cluster_spiffeids
├── entry.build_node_map
├── entry.render_state
├── entry.diff
├── entry.delete  (count=N)
│   └── grpc.BatchDeleteEntry
├── entry.create  (count=N)
│   └── grpc.BatchCreateEntry
├── entry.update  (count=N)
│   └── grpc.BatchUpdateEntry
└── entry.status_update

When enabled: false (the default), there is zero runtime overhead — a no-op TracerProvider is installed so all instrumentation call sites are compiled in but cost nothing.

# Example controller config snippet
spec:
  tracing:
    enabled: true
    otlpEndpoint: "otel-collector.monitoring:4317"
    sampleRatio: 0.1   # 10% sampling in high-throughput clusters

Design notes

• No-op by default: tracing is opt-in; metrics are always emitted (same model as the existing cluster_static_entry_failures counter).
• otelgrpc on the SPIRE client: a single grpc.WithStatsHandler(otelgrpc.NewClientHandler()) call instruments all gRPC calls automatically. When tracing is off, the handler is present but the no-op provider means no spans are created.
• pod_attestation_delay_seconds excludes AlreadyExists races: if a second controller replica creates the same entry (because leader election is off), the AlreadyExists response is not recorded as an attestation event — only genuine first-creates contribute to the SLI.
• already_exists / not_found counters: spire_entry_write_total{code="already_exists"} on creates and code="not_found" on deletes are the fingerprint of two replicas running without leader election. Enabling leader election should drop these to zero.

Checklist

• Metrics compile and are registered at startup
• Unit tests for EntryWriteTotal counters (create/update/delete, OK + error codes)
• Unit test for SPIREAPIRequestDuration observation on ListEntries
• Integration / e2e test with a real SPIRE server
• Grafana dashboard (out of scope for this draft)
• CRD / config documentation update

Made with Cursor