Best practice for integrating OpenTelemetry-based observability (e.g. Pydantic Logfire) with Semantic Kernel?

[Stefano Sambruna (stefanosambruna)]

Hi everyone, I’m looking into integrating third-party observability tools built on OpenTelemetry (for example Pydantic Logfire) into a project using Semantic Kernel.

To clarify upfront:
I’m not interested in Application Insights. I’m specifically looking for guidance on plugging external OpenTelemetry-compatible tools, mainly because of different requirements around UX, trace quality, and developer experience.

The goal is to achieve end-to-end observability (traces, logs, metrics) across multi-step SK workflows and agent interactions, without adding invasive instrumentation.

Any best practices, patterns, or pointers would be greatly appreciated. Thanks! 🙌

[Dar (darfaz)]

Great question — we went through this exact exercise integrating observability into our SK pipelines.

What worked for us:

SK already emits OpenTelemetry traces for kernel invocations and connector calls. The key is configuring the OTLP exporter to point at your collector (Logfire, Jaeger, whatever):

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="your-collector:4317")))
trace.set_tracer_provider(provider)

SK's Python SDK picks up the global tracer provider automatically for LLM calls. For custom spans around your own logic, just use the standard tracer.start_as_current_span() API.

One thing observability alone misses: security telemetry.

Traces tell you what happened (latency, token counts, errors), but not what went wrong from a security perspective — like whether a user prompt contained injection attempts, or whether an LLM response leaked PII before it reached the user.

We added a lightweight pre/post validation layer alongside our traces. Tools like ClawMoat can scan for prompt injection, PII leakage, and secret exposure with zero dependencies — easy to wire into SK's filter pipeline so you get security events as part of your trace context.

TL;DR: OTLP exporter + global tracer provider gets you 80% there. Add security scanning at the filter layer for the other 20%.

[Aditya Pratap Singh (akkupratap323)]

For Python, the cleanest pattern is to let the observability backend configure the global OpenTelemetry providers before any Semantic Kernel objects are created. Semantic Kernel emits to those providers, so you do not need an SK-specific Logfire exporter.

For Logfire, a minimal setup is:

import logging
import logfire

# Configure OTel providers/export to Logfire first.
logfire.configure()

# Optional: SK uses stdlib logging, which needs an explicit handler if you
# also want those records in Logfire.
logging.basicConfig(
    level=logging.INFO,
    handlers=[logfire.LogfireLoggingHandler()],
)

# Import/build the Kernel and connectors after telemetry setup.
from semantic_kernel import Kernel

Enable SK's connector spans with the non-sensitive switch in the deployment environment:

SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS=true
OTEL_SERVICE_NAME=my-sk-service

Use SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true only if you deliberately want prompts and completions exported; those can contain user data, tool arguments, and secrets. The non-sensitive switch still gives model/operation/token metadata.

For one end-to-end trace, put a parent span around the request/workflow boundary:

with logfire.span("agent workflow", conversation_id=conversation_id):
    result = await kernel.invoke(...)

SK's connector/function spans then become children of that span through normal OTel context propagation. Instrument the surrounding web/HTTP/database libraries as well so the inbound request, SK calls, tools, and outbound calls share the same trace.

The main trap is configuring two SDKs. Do not call logfire.configure() and then replace the global TracerProvider with another one (or do the reverse); configure one provider/export path per process, otherwise spans can be lost or duplicated. If you want a vendor-neutral deployment, send OTLP to an OpenTelemetry Collector and fan out there instead.

References:

• SK telemetry setup and the two diagnostics switches: https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/telemetry-with-console
• Logfire lists Semantic Kernel as using native OpenTelemetry: https://pydantic.dev/docs/logfire/integrations/
• Capturing Python standard-library logs in Logfire: https://pydantic.dev/docs/logfire/integrations/logging/logging/

[Ícaro Galvão do Nascimento (Icaro0310)]

SK already emits OpenTelemetry natively, so there is nothing to plug in beyond an exporter. Set SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS=true (and ..._SENSITIVE=true if you want prompt/completion content in spans) and SK writes GenAI spans into the ambient tracer provider.

For Python with Logfire, I just call logfire.configure() early and the SK spans land in the same provider. On the .NET side I add builder.AddSource("Microsoft.SemanticKernel*") and point the OTLP exporter at Logfire's ingest endpoint with the write token in the Authorization header.

The sensitive flag being off by default trips people up. Spans show up but look empty, and it seems like the exporter is broken when it isn't.

—
More of my work lives here: github.com/Icaro0310 · Website: icaro0310.github.io