tracing: OpenTelemetry spans for the agent loop

CHANGELOG.md

@@ -17,6 +17,21 @@ and [`agents/peggy/CHANGELOG.md`](agents/peggy/CHANGELOG.md).
 
 ## Unreleased
 
+- **OpenTelemetry tracing for the agent loop (`tracing`).** Every run
+  now emits an OTLP trace — `glue.agent.run` → `glue.agent.turn` →
+  `glue.llm.generate` / `glue.tool.<name>` — with GenAI-convention
+  attributes (model, provider, finish reason, token usage) and tool
+  error flags. A `Recorder` translates the existing loop `Event` stream
+  into spans, so the `loop` and core `glue` packages stay free of
+  OpenTelemetry imports. Configured from the standard `OTEL_*`
+  environment and off by default (zero overhead until an exporter is
+  set); `OTEL_TRACES_EXPORTER=stdout` gives a zero-backend local view.
+  Wired into `glue run`, the TUI, and `glue goal` (which traces its
+  planner/maker/checker sessions via the new `GoalSpec.OnSession` seam).
+  Borrowed from Vercel's Eve framework; see
+  [ADR-0018](docs/adr/0018-otel-tracing.md) and
+  [docs/tracing.md](docs/tracing.md). (#358)
+
 - **TUI: `/` picker shows every command; `@` picker gains scroll
   indicators (`cmd/glue/tui`).** The slash-command popup previously
   reused the file picker's 8-row scroll window with no indicator, so a

cmd/glue/goalcmd.go

@@ -12,6 +12,7 @@ import (
 	"github.com/erain/glue"
 	"github.com/erain/glue/cmd/glue/worktree"
 	filestore "github.com/erain/glue/stores/file"
+	"github.com/erain/glue/tracing"
 )
 
 // Exit codes for `glue goal`, so cron/CI schedulers can branch on the
@@ -174,6 +175,12 @@ func goalCommand(ctx context.Context, args []string, stdin io.Reader, stdout, st
 		}
 	}
 
+	// Trace every planner/maker/checker session so unattended goal runs
+	// are observable end-to-end. A no-op unless OTEL_* is configured.
+	spec.OnSession = func(s *glue.Session) {
+		s.Subscribe(tracing.NewSessionRecorder(s.ID()).Handle)
+	}
+
 	spec.Emit = func(ev glue.GoalEvent) {
 		switch ev.Type {
 		case glue.GoalEventPlan:

cmd/glue/main.go

@@ -28,6 +28,7 @@ import (
 	"github.com/erain/glue/providers"
 	filestore "github.com/erain/glue/stores/file"
 	toolscoding "github.com/erain/glue/tools/coding"
+	"github.com/erain/glue/tracing"
 
 	// Register the shipped providers so they resolve through the
 	// providers registry by name (--provider). Importing for side
@@ -108,6 +109,18 @@ func runCLIWithDeps(ctx context.Context, args []string, stdin io.Reader, stdout
 		return 0
 	}
 
+	// Configure OpenTelemetry tracing from the OTEL_* environment. A no-op
+	// unless an exporter is configured, so it costs nothing by default.
+	if shutdown, err := tracing.Setup(ctx); err != nil {
+		fmt.Fprintf(stderr, "glue: tracing disabled: %v\n", err)
+	} else if shutdown != nil {
+		defer func() {
+			flushCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+			defer cancel()
+			_ = shutdown(flushCtx)
+		}()
+	}
+
 	switch args[0] {
 	case "run":
 		if err := runCommand(ctx, args[1:], stdin, stdout, stderr, newProvider); err != nil {

cmd/glue/run.go

@@ -20,6 +20,7 @@ import (
 	"github.com/erain/glue/cmd/glue/tui"
 	filestore "github.com/erain/glue/stores/file"
 	toolscoding "github.com/erain/glue/tools/coding"
+	"github.com/erain/glue/tracing"
 	// Register the shipped providers so they resolve through the
 	// providers registry by name (--provider). Importing for side
 )
@@ -242,6 +243,7 @@ func runCommand(ctx context.Context, args []string, stdin io.Reader, stdout io.W
 	if err != nil {
 		return err
 	}
+	defer session.Subscribe(tracing.NewSessionRecorder(session.ID()).Handle)()
 
 	if jsonMode {
 		return runOneShotJSON(ctx, session, effectivePrompt, *id, providerName, effectiveModel, stdout, *showUsage, usagePricing, stderr)

cmd/glue/tui/tui.go

@@ -21,6 +21,7 @@ import (
 
 	"github.com/erain/glue"
 	"github.com/erain/glue/cmd/glue/atmentions"
+	"github.com/erain/glue/tracing"
 )
 
 // Config wires the TUI to a glue agent. Provider/Model/WorkDir are
@@ -1239,6 +1240,7 @@ func (m *Model) startTurn(prompt string) tea.Cmd {
 
 	go func() {
 		defer cancel()
+		defer session.Subscribe(tracing.NewSessionRecorder(session.ID()).Handle)()
 		unsubscribe := session.Subscribe(func(e glue.Event) {
 			switch e.Type {
 			case glue.EventTextDelta:

docs/adr/0018-otel-tracing.md

@@ -0,0 +1,92 @@
+# ADR-0018: OpenTelemetry Tracing for the Agent Loop
+
+## Status
+
+Accepted (2026-06-18).
+
+## Context
+
+Glue had no observability: a run was a black box once it left the
+terminal. Diagnosing a slow turn, an unexpected tool failure, or a
+runaway goal loop meant re-reading stdout or the JSON event stream by
+hand. There was no way to answer "where did the latency go?", "how many
+tokens did this objective burn, per iteration?", or "which tool errored
+in last night's unattended run?".
+
+Research into Vercel's **Eve** agent framework (June 2026) flagged
+first-class tracing as the cleanest idea to borrow: Eve emits a span per
+model call and per tool invocation over standard OpenTelemetry, so runs
+drop into Jaeger / Honeycomb / Datadog with no bespoke tooling. Glue
+already has the ideal substrate — `loop.Run` emits a strictly-ordered
+`Event` stream (loop/turn/message/tool start-end, error), and
+`Session.Subscribe(func(Event))` is a public, additive, fan-out sink.
+Nothing in the hot path needed to change to observe it.
+
+## Decision
+
+Add a dependency-isolated `tracing/` package that translates the loop
+event stream into an OpenTelemetry span tree, one trace per prompt:
+
+```
+glue.agent.run                 (one trace per loop.Run / prompt)
+├── glue.agent.turn            (one per assistant turn)
+│   ├── glue.llm.generate      (the provider call; model, tokens, finish reason)
+│   └── glue.tool.<name>       (each tool call; is_error)
+└── ...
+```
+
+Key choices:
+
+- **Event-stream translation, not loop instrumentation.** A `Recorder`
+  consumes the existing `Event` stream and maintains the span tree
+  (root/turn/message + a map keyed by tool-call id so parallel tool
+  calls close against the right span). The `loop` and root `glue`
+  packages stay free of OpenTelemetry imports; all otel deps live in
+  `tracing/` and `cmd/glue`. This mirrors how `WithStreamWriter` /
+  `WithToolLogger` already attach to the event stream.
+
+- **GenAI semantic conventions.** Spans carry `gen_ai.*` attributes
+  (request/response model, `gen_ai.system` = provider, finish reason,
+  input/output/total tokens from `Message.Usage`) plus `glue.*` detail
+  (session id, tool name/call-id/is_error, turn index), so existing LLM
+  dashboards light up without remapping.
+
+- **Standard env contract, off by default.** `NewTracerProvider` /
+  `Setup` read the standard `OTEL_*` environment
+  (`OTEL_TRACES_EXPORTER` = `otlp` | `stdout` | `none`,
+  `OTEL_EXPORTER_OTLP_*`, `OTEL_SERVICE_NAME`). With nothing configured
+  the whole path is a no-op (`Enabled()` is false, `NewSessionRecorder`
+  returns a nil-tracer recorder), so there is zero overhead and zero
+  config until a backend is wired up.
+
+- **Resilient span lifecycle.** The recorder closes any open spans on
+  `EventLoopEnd`, on a fresh `EventLoopStart` (a prior crashed run can't
+  leak spans into the next prompt), and on `EventError` (which also
+  flags the root span so failed traces are filterable at the top). This
+  matters because the loop returns early on cancellation, provider
+  error, and max-turns without emitting the matching end events.
+
+- **Wired at the host edge.** `cmd/glue` calls `tracing.Setup` once in
+  `main` (with a flush-on-exit shutdown) and attaches a recorder via
+  `Session.Subscribe` in the `run`, TUI, and `goal` paths. The goal loop
+  gains a `GoalSpec.OnSession` seam so the host can observe its internal
+  planner/maker/checker sessions without the core loop depending on any
+  tracing backend.
+
+## Consequences
+
+- glue runs are now observable end-to-end against any OTLP backend with
+  a single env var; `OTEL_TRACES_EXPORTER=stdout` gives a zero-backend
+  local view.
+- The module now depends on the OpenTelemetry SDK and OTLP/stdout
+  exporters. They are only imported by `tracing/` and `cmd/glue`; the
+  embeddable `glue` / `loop` libraries remain otel-free at the type
+  level (the `glue` package references tracing only through the optional
+  `GoalSpec.OnSession` hook, which is an ordinary `func(*Session)`).
+- Provider HTTP-level spans are **not** yet nested under
+  `glue.llm.generate` — context is not threaded into `Provider.Stream`.
+  The event-stream design records accurate wall-clock per call; deeper
+  propagation (and instrumenting the provider HTTP clients) is a
+  follow-up.
+- The daemon (`serve`) path is not yet traced; it can subscribe a
+  recorder per session the same way when prioritized.

docs/tracing.md

@@ -0,0 +1,110 @@
+# Tracing glue runs with OpenTelemetry
+
+Glue emits OpenTelemetry traces for every agent run, so you can see where
+latency, tokens, and tool failures actually go — in Jaeger, Honeycomb,
+Datadog, Grafana Tempo, or any OTLP-compatible backend.
+
+Tracing is **off until you configure an exporter**, so there is zero
+overhead by default. See [ADR-0018](adr/0018-otel-tracing.md) for the
+design.
+
+## The span tree
+
+Each prompt (one `loop.Run`) is one trace:
+
+```
+glue.agent.run                 root span — one per prompt
+├── glue.agent.turn            one per assistant turn
+│   ├── glue.llm.generate      the provider call
+│   └── glue.tool.<name>       each tool invocation
+└── glue.agent.turn
+    └── ...
+```
+
+Attributes follow the OpenTelemetry GenAI semantic conventions where
+they apply, so existing LLM dashboards work unchanged:
+
+| Span | Attributes |
+| --- | --- |
+| `glue.agent.run` | `glue.session.id` |
+| `glue.agent.turn` | `glue.turn.index` |
+| `glue.llm.generate` | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.system` (provider), `gen_ai.response.finish_reason`, `gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens`, `glue.usage.cache_read_tokens` / `cache_write_tokens` |
+| `glue.tool.<name>` | `glue.tool.name`, `glue.tool.call_id`, `glue.tool.is_error` |
+
+A failed run flags both the deepest open span and the root span with an
+`Error` status, so failed traces are easy to filter.
+
+## Enabling it
+
+Tracing reads the standard `OTEL_*` environment — no glue-specific
+config.
+
+**Send to an OTLP backend** (Jaeger, Honeycomb, Datadog, Tempo, …):
+
+```sh
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
+export OTEL_SERVICE_NAME=glue            # optional, defaults to "glue"
+glue run --coding "fix the failing test"
+```
+
+Setting an endpoint is enough to enable the OTLP/HTTP exporter. All the
+standard `OTEL_EXPORTER_OTLP_*` variables (headers, protocol, per-signal
+endpoints) are honored.
+
+**Print spans locally** with no backend (handy for debugging):
+
+```sh
+OTEL_TRACES_EXPORTER=stdout glue run --coding "..."
+```
+
+**Force it off** even if an endpoint is set:
+
+```sh
+OTEL_TRACES_EXPORTER=none glue run ...
+```
+
+`glue goal` traces its planner, every maker, and every checker session,
+which makes unattended goal loops observable end to end.
+
+## Embedding glue
+
+If you embed the `glue` library, wire tracing yourself:
+
+```go
+import (
+    "github.com/erain/glue"
+    "github.com/erain/glue/tracing"
+)
+
+// Once at startup: configure the global tracer provider from OTEL_*.
+shutdown, err := tracing.Setup(ctx)
+if err == nil && shutdown != nil {
+    defer shutdown(context.Background()) // flush on exit
+}
+
+// Per session: attach a recorder. No-op when tracing is disabled.
+sess, _ := agent.Session(ctx, "my-session")
+defer sess.Subscribe(tracing.NewSessionRecorder(sess.ID()).Handle)()
+```
+
+For the goal loop, set `GoalSpec.OnSession` to subscribe a recorder to
+each internal session:
+
+```go
+spec.OnSession = func(s *glue.Session) {
+    s.Subscribe(tracing.NewSessionRecorder(s.ID()).Handle)
+}
+```
+
+The `loop` and core `glue` packages carry no OpenTelemetry imports; all
+of it lives in `tracing/`.
+
+## Not yet traced
+
+- **Provider HTTP spans** are not nested under `glue.llm.generate` —
+  span timing is accurate, but the underlying HTTP request is not a
+  child span yet.
+- **The `serve` daemon** does not attach recorders to its sessions yet.
+
+Both are straightforward follow-ups on the same `Session.Subscribe`
+seam.

go.mod

@@ -7,19 +7,26 @@ require (
 	github.com/charmbracelet/bubbletea v1.3.10
 	github.com/charmbracelet/glamour v1.0.0
 	github.com/charmbracelet/lipgloss v1.1.1-0.20250404203927-76690c660834
-	golang.org/x/term v0.36.0
+	go.opentelemetry.io/otel v1.40.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0
+	go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.40.0
+	go.opentelemetry.io/otel/sdk v1.40.0
+	go.opentelemetry.io/otel/trace v1.40.0
+	golang.org/x/term v0.39.0
 	google.golang.org/genai v1.55.0
 	modernc.org/sqlite v1.50.1
 )
 
 require (
 	cloud.google.com/go v0.116.0 // indirect
 	cloud.google.com/go/auth v0.9.3 // indirect
-	cloud.google.com/go/compute/metadata v0.5.0 // indirect
+	cloud.google.com/go/compute/metadata v0.9.0 // indirect
 	github.com/alecthomas/chroma/v2 v2.20.0 // indirect
 	github.com/atotto/clipboard v0.1.4 // indirect
 	github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
 	github.com/aymerick/douceur v0.2.0 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/charmbracelet/colorprofile v0.4.1 // indirect
 	github.com/charmbracelet/x/ansi v0.11.6 // indirect
 	github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
@@ -31,13 +38,16 @@ require (
 	github.com/dlclark/regexp2 v1.11.5 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
-	github.com/google/go-cmp v0.6.0 // indirect
+	github.com/google/go-cmp v0.7.0 // indirect
 	github.com/google/s2a-go v0.1.8 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
 	github.com/gorilla/css v1.0.1 // indirect
 	github.com/gorilla/websocket v1.5.3 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.7 // indirect
 	github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/mattn/go-localereader v0.0.1 // indirect
@@ -54,13 +64,18 @@ require (
 	github.com/yuin/goldmark v1.7.13 // indirect
 	github.com/yuin/goldmark-emoji v1.0.6 // indirect
 	go.opencensus.io v0.24.0 // indirect
-	golang.org/x/crypto v0.36.0 // indirect
-	golang.org/x/net v0.38.0 // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 // indirect
+	go.opentelemetry.io/otel/metric v1.40.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.9.0 // indirect
+	golang.org/x/crypto v0.47.0 // indirect
+	golang.org/x/net v0.49.0 // indirect
 	golang.org/x/sys v0.42.0 // indirect
-	golang.org/x/text v0.30.0 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
-	google.golang.org/grpc v1.66.2 // indirect
-	google.golang.org/protobuf v1.34.2 // indirect
+	golang.org/x/text v0.33.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 // indirect
+	google.golang.org/grpc v1.78.0 // indirect
+	google.golang.org/protobuf v1.36.11 // indirect
 	modernc.org/libc v1.72.3 // indirect
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.11.0 // indirect