feat(eve): keep local tracing on with authored instrumentation

.changeset/wise-otters-tap.md

@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Local dev tracing now stays on when an agent authors `agent/instrumentation.ts`. Instead of standing down, `eve dev` observes the tracer provider your `setup` registered — including tracers your module took before it registered that provider — so your exporter keeps receiving everything it did before and additionally sees eve's `agent.*` spans, while `.eve/traces/v1`, `eve trace ls`, and `eve trace <id>` keep working. `setup` may now be `async`: eve awaits it before installing its writer. Neither sink receives the Workflow SDK's own spans any more — a turn runs inside a durable step, so a single turn used to send dozens of `step.execute`-style spans to your exporter, and eve now declines to create them while the dev server is running, reparenting anything nested under one onto the nearest span above it. If eve cannot observe spans in a given dev worker it logs a warning and leaves your instrumentation alone rather than failing the dev server. Production is unchanged; `EVE_TRACES=off` still opts out of the local writer.

docs/guides/instrumentation.md

@@ -9,13 +9,15 @@ If you intend to export telemetry, review the exporter destination, data categor
 
 ## Zero-config local traces
 
-When no authored `instrumentation.ts` exists, `eve dev` records agent, AI SDK, and user-created OpenTelemetry spans under `.eve/traces/v1`. Each session has one trace, rooted independently from Workflow telemetry, with turns, model steps, and tool actions represented explicitly. Spans created by application code while a model or tool is executing inherit that active agent context.
+`eve dev` records agent, AI SDK, and user-created OpenTelemetry spans under `.eve/traces/v1`. Each session has one trace, rooted independently from Workflow telemetry, with turns, model steps, and tool actions represented explicitly. Spans created by application code while a model or tool is executing inherit that active agent context. A span your code creates outside a session — in a channel route, say — belongs to no session trace, so it is not written to `.eve/traces/v1`; an exporter you configured yourself still receives it.
 
 The directory is an immutable OTLP/JSON spool and remains available after `eve dev` exits, subject to the [retention policy](#local-trace-retention) below. Inspection tools may build a query index from these segments, but the index is derived and can be rebuilt without changing the captured trace data.
 
 Use `eve trace ls` to list captured traces and `eve trace <trace>` to inspect a session's span tree.
 
-The local writer is an internal development default, not a second provider layered over authored instrumentation. When `instrumentation.ts` exists, its setup retains control and the zero-config writer is not installed.
+The writer is always installed in `eve dev`, including when you author an `instrumentation.ts`. It does not compete with your provider: eve wraps the provider your `setup` registered rather than replacing it, so your exporter keeps receiving everything it received before, and in dev it additionally sees eve's `agent.*` spans. Because it observes your provider rather than running beside it, the local writer sees the spans your provider records: a sampler that drops a span drops it for `.eve/traces/v1` too. Production is unaffected — outside `eve dev`, authored instrumentation is the only provider and nothing is written to disk. Set `EVE_TRACES=off` to opt out of the local writer.
+
+eve is moving toward agent-centric traces — a trace that records what the agent did, not how eve ran it — and in `eve dev` that shapes what your exporter receives. The Workflow SDK's own spans are the first thing it applies to: a turn runs inside a durable step, so spans like `step.execute` arrive in the middle of the agent's trace and describe eve's execution machinery, dozens of them per turn. While the dev server is running, eve declines to create them at all, for your exporter as well as for `.eve/traces/v1`. Nesting is unaffected: a span started underneath one that was declined attaches to the nearest span above it, so your spans still sit under `agent.step`. In production nothing intercepts your provider, so it receives Workflow spans exactly as it does today.
 
 ### Local trace retention
 
@@ -80,6 +82,8 @@ Export the result of `defineInstrumentation` as the default export.
 
 Use the `setup` callback to register your OTel provider (for example `registerOTel` from `@vercel/otel`). The framework invokes it at server startup with the resolved agent name. `context.agentName` is resolved at compile time from your project (the package's `name`, falling back to the app directory name), so you never hard-code a service name.
 
+`setup` may be `async`. eve awaits it before anything else looks for a tracer provider, so a provider you register after an `await` is still in place for the first turn.
+
 Any OTel-compatible backend works (Braintrust, Raindrop, Arize, Honeycomb, Datadog, Jaeger). Install the exporter package you need and configure it in the callback.
 
 Three more fields control what the AI SDK records inside those spans (see the AI SDK's [telemetry reference](https://ai-sdk.dev/docs/ai-sdk-core/telemetry)):
@@ -136,7 +140,7 @@ Channel metadata is channel-owned. Built-in channels expose only the fields they
 
 ## Authored trace hierarchy
 
-The existing authored `instrumentation.ts` path remains separate from zero-config local traces. When authored telemetry is enabled, each turn currently produces a trace like:
+Your exporter receives the hierarchy the AI SDK produces. When authored telemetry is enabled, each turn currently produces a trace like:
 
 ```text
 ai.eve.turn  {eve.session.id}

packages/eve/scripts/vendor-compiled/declarations/@opentelemetry/api.d.ts

@@ -8,7 +8,8 @@ export interface SpanContext {
 
 export interface Span {
   addEvent(name: string, attributes?: Attributes): this;
-  end(): void;
+  end(endTime?: number): void;
+  isRecording?(): boolean;
   recordException(
     exception: Error | string | { message?: string; name?: string; stack?: string },
   ): void;
@@ -17,12 +18,25 @@ export interface Span {
   spanContext(): SpanContext;
 }
 
+export interface SpanOptions {
+  attributes?: Attributes | undefined;
+  root?: boolean | undefined;
+}
+
 export interface Tracer {
-  startSpan(
-    name: string,
-    options?: { attributes?: Attributes | undefined; root?: boolean | undefined },
-    context?: Context,
-  ): Span;
+  startSpan(name: string, options?: SpanOptions, context?: Context): Span;
+}
+
+export interface TracerProvider {
+  getTracer(name: string, version?: string, options?: unknown): Tracer;
+}
+
+export declare class ProxyTracerProvider implements TracerProvider {
+  getDelegate(): TracerProvider;
+  /** `undefined` until a delegate is set — how eve tells an unclaimed proxy from a claimed one. */
+  getDelegateTracer(name: string, version?: string, options?: unknown): Tracer | undefined;
+  getTracer(name: string, version?: string, options?: unknown): Tracer;
+  setDelegate(delegate: TracerProvider): void;
 }
 
 export interface Context {}
@@ -42,11 +56,16 @@ export declare const context: {
 
 export declare const trace: {
   getActiveSpan(): Span | undefined;
+  getSpanContext(context: Context): SpanContext | undefined;
   getTracer(name: string, version?: string): Tracer;
+  getTracerProvider(): TracerProvider;
   setSpan(context: Context, span: Span): Context;
   wrapSpanContext(spanContext: SpanContext): Span;
 };
 
+/** All-zero ids: what the API hands out when there is no span to refer to. */
+export declare const INVALID_SPAN_CONTEXT: SpanContext;
+
 export declare enum SpanKind {
   INTERNAL = 0,
   SERVER = 1,

packages/eve/src/harness/agent-trace-span-processor.test.ts

@@ -65,6 +65,32 @@ describe("AgentTraceSpanProcessor", () => {
     expect(child.onStart).toHaveBeenCalledTimes(1);
     expect(child.onEnd).not.toHaveBeenCalled();
   });
+
+  // An authored `instrumentation.ts` chooses its own SDK version, and
+  // `@opentelemetry/sdk-trace-base` 1.x names this field `instrumentationLibrary`.
+  // Reading only the 2.x name would let a run's spans into the local store for
+  // anyone still on `@vercel/otel@1`.
+  it("excludes Workflow instrumentation reported under the pre-2.x field name", () => {
+    const child = {
+      forceFlush: vi.fn(async () => {}),
+      onEnd: vi.fn(),
+      onStart: vi.fn(),
+      shutdown: vi.fn(async () => {}),
+    };
+    const processor = new AgentTraceSpanProcessor([child]);
+    processor.onStart(span("trace-1", { "agent.session.id": "session-1" }), {});
+
+    const workflow = {
+      attributes: {},
+      instrumentationLibrary: { name: "workflow" },
+      spanContext: () => ({ traceId: "trace-1" }),
+    };
+    processor.onStart(workflow, {});
+    processor.onEnd(workflow);
+
+    expect(child.onStart).toHaveBeenCalledTimes(1);
+    expect(child.onEnd).not.toHaveBeenCalled();
+  });
 });
 
 function span(

packages/eve/src/harness/agent-trace-span-processor.ts

@@ -1,17 +1,32 @@
 import type { SpanProcessor } from "#compiled/@vercel/otel/index.js";
 
+import { WORKFLOW_INSTRUMENTATION_SCOPE } from "#harness/workflow-instrumentation-scope.js";
+
 interface SpanLike {
   readonly attributes: Readonly<Record<string, unknown>>;
+  readonly instrumentationLibrary?: { readonly name?: string };
   readonly instrumentationScope?: { readonly name?: string };
   readonly spanContext: () => { readonly traceId: string };
 }
 
+/**
+ * The name of the instrumentation that created `span`.
+ *
+ * `@opentelemetry/sdk-trace-base` 2.x renamed `instrumentationLibrary` to
+ * `instrumentationScope`. eve reads spans from providers it did not build — an
+ * authored `instrumentation.ts` picks its own SDK version — so both shapes
+ * arrive here, and a span whose scope cannot be read is one this processor
+ * cannot decide about.
+ */
+function instrumentationName(span: SpanLike): string | undefined {
+  return span.instrumentationScope?.name ?? span.instrumentationLibrary?.name;
+}
+
 /** Routes spans from agent-owned traces to provider-neutral child processors. */
 export class AgentTraceSpanProcessor implements SpanProcessor {
   readonly #children: readonly SpanProcessor[];
   readonly #ownedTraceIds = new Set<string>();
   readonly #sessionTraceIds = new Map<string, string>();
-  #attached = false;
 
   constructor(children: readonly SpanProcessor[]) {
     this.#children = children;
@@ -21,12 +36,7 @@ export class AgentTraceSpanProcessor implements SpanProcessor {
     await Promise.all(this.#children.map((child) => child.forceFlush()));
   }
 
-  isAttached(): boolean {
-    return this.#attached;
-  }
-
   onStart(span: unknown, parentContext: unknown): void {
-    this.#attached = true;
     if (!isSpanLike(span)) return;
     const sessionId = span.attributes["agent.session.id"];
     if (typeof sessionId === "string") {
@@ -61,9 +71,17 @@ export class AgentTraceSpanProcessor implements SpanProcessor {
     await Promise.all(this.#children.map((child) => child.shutdown()));
   }
 
+  /**
+   * A run's internal spans are declined even inside a trace eve owns, so
+   * `eve trace` shows the session's own work.
+   *
+   * Still needed with `suppressWorkflowInstrumentation` in place: that only
+   * covers a provider eve intercepts, and on the zero-config path eve registers
+   * the provider itself, so the Workflow SDK's spans are real and arrive here.
+   */
   #accepts(span: SpanLike): boolean {
     return (
-      span.instrumentationScope?.name !== "workflow" &&
+      instrumentationName(span) !== WORKFLOW_INSTRUMENTATION_SCOPE &&
       this.#ownedTraceIds.has(span.spanContext().traceId)
     );
   }

packages/eve/src/harness/delegating-tracer-provider.test.ts

@@ -0,0 +1,83 @@
+import { describe, expect, it } from "vitest";
+
+import {
+  ProxyTracerProvider,
+  type Span,
+  type Tracer,
+  type TracerProvider,
+} from "#compiled/@opentelemetry/api/index.js";
+import {
+  hasTracerProviderDelegate,
+  isDelegatingTracerProvider,
+  type DelegatingTracerProvider,
+} from "#harness/delegating-tracer-provider.js";
+
+describe("isDelegatingTracerProvider", () => {
+  it("recognizes the real proxy provider", () => {
+    expect(isDelegatingTracerProvider(new ProxyTracerProvider())).toBe(true);
+  });
+
+  // The case `instanceof` cannot answer: an authored `instrumentation.ts`
+  // resolves its own copy of `@opentelemetry/api`, so the proxy eve meets at the
+  // global slot is an instance of a class eve has never seen.
+  it("recognizes a proxy from a foreign copy of the API", () => {
+    expect(isDelegatingTracerProvider(createForeignProxy())).toBe(true);
+  });
+
+  it("rejects a plain provider", () => {
+    expect(isDelegatingTracerProvider({ getTracer: () => createTracer() })).toBe(false);
+  });
+});
+
+describe("hasTracerProviderDelegate", () => {
+  it("is false for a proxy nobody has claimed", () => {
+    expect(hasTracerProviderDelegate(new ProxyTracerProvider())).toBe(false);
+    expect(hasTracerProviderDelegate(createForeignProxy())).toBe(false);
+  });
+
+  it("is true once a delegate is set", () => {
+    const proxy = new ProxyTracerProvider();
+    proxy.setDelegate({ getTracer: () => createTracer() });
+
+    expect(hasTracerProviderDelegate(proxy)).toBe(true);
+  });
+
+  // The probe must not leave a tracer behind in the delegate's cache under a
+  // name the author would later use, so it asks for one of eve's own.
+  it("probes under an eve-namespaced tracer name", () => {
+    const asked: string[] = [];
+    const proxy = createForeignProxy();
+    proxy.setDelegate({
+      getTracer: (name: string) => {
+        asked.push(name);
+        return createTracer();
+      },
+    });
+
+    expect(hasTracerProviderDelegate(proxy)).toBe(true);
+    expect(asked.every((name) => name.startsWith("eve."))).toBe(true);
+  });
+});
+
+/**
+ * `ProxyTracerProvider` as another copy of the API would define it: same shape,
+ * unrelated class, and its own no-op provider that eve has no reference to.
+ */
+function createForeignProxy(): DelegatingTracerProvider {
+  let delegate: TracerProvider | undefined;
+  const noop: TracerProvider = { getTracer: () => createTracer() };
+  return {
+    getDelegate: () => delegate ?? noop,
+    getDelegateTracer: (name: string, version?: string, options?: unknown) =>
+      delegate?.getTracer(name, version, options),
+    getTracer: (name: string, version?: string, options?: unknown) =>
+      delegate?.getTracer(name, version, options) ?? noop.getTracer(name),
+    setDelegate: (next: TracerProvider) => {
+      delegate = next;
+    },
+  };
+}
+
+function createTracer(): Tracer {
+  return { startSpan: () => ({}) as Span };
+}

packages/eve/src/harness/delegating-tracer-provider.ts

@@ -0,0 +1,47 @@
+import type { TracerProvider } from "#compiled/@opentelemetry/api/index.js";
+
+/**
+ * The `ProxyTracerProvider` surface, duck-typed.
+ *
+ * `setGlobalTracerProvider` registers this proxy rather than the provider
+ * handed to it, so it is the shape eve meets at the global slot. It cannot be
+ * recognized with `instanceof`: an authored `instrumentation.ts` resolves
+ * `@opentelemetry/api` from the agent's own dependencies, so its class is a
+ * different object than the one in eve's vendored copy.
+ */
+export interface DelegatingTracerProvider extends TracerProvider {
+  getDelegate: () => TracerProvider;
+  getDelegateTracer: (name: string, version?: string, options?: unknown) => unknown;
+  setDelegate: (delegate: TracerProvider) => void;
+}
+
+/**
+ * Whether `provider` resolves its tracers through a delegate that can be
+ * replaced.
+ *
+ * @internal — not part of the public API.
+ */
+export function isDelegatingTracerProvider(
+  provider: TracerProvider,
+): provider is DelegatingTracerProvider {
+  const candidate = provider as Partial<DelegatingTracerProvider>;
+  return (
+    typeof candidate.getDelegate === "function" &&
+    typeof candidate.getDelegateTracer === "function" &&
+    typeof candidate.setDelegate === "function"
+  );
+}
+
+/**
+ * Whether a delegate has been set on `provider`.
+ *
+ * `getDelegate()` cannot answer this: with nothing set it returns the API's
+ * no-op provider, and that singleton belongs to whichever copy of
+ * `@opentelemetry/api` created the proxy, so eve has no object to compare it
+ * against. `getDelegateTracer` returns `undefined` in exactly that case.
+ *
+ * @internal — not part of the public API.
+ */
+export function hasTracerProviderDelegate(provider: DelegatingTracerProvider): boolean {
+  return provider.getDelegateTracer("eve.delegate-probe") !== undefined;
+}

packages/eve/src/harness/instrumentation-config.test.ts

@@ -25,7 +25,7 @@ describe("instrumentation-config chunk-isolation regression", () => {
     vi.resetModules();
     const moduleA = await import("#harness/instrumentation-config.js");
     const config = { functionId: "test.instrumentation.cross-module.alice" };
-    moduleA.registerInstrumentationConfig(config, { agentName: "test-agent" });
+    await moduleA.registerInstrumentationConfig(config, { agentName: "test-agent" });
 
     vi.resetModules();
     const moduleB = await import("#harness/instrumentation-config.js");
@@ -40,7 +40,7 @@ describe("instrumentation-config chunk-isolation regression", () => {
     const { registerInstrumentationConfig } = await import("#harness/instrumentation-config.js");
 
     const canary = { functionId: "test.instrumentation.global-mount.canary" };
-    registerInstrumentationConfig(canary, { agentName: "test-agent" });
+    await registerInstrumentationConfig(canary, { agentName: "test-agent" });
 
     expect((globalThis as Record<symbol, unknown>)[globalKey]).toBe(canary);
   });
@@ -51,7 +51,7 @@ describe("instrumentation-config chunk-isolation regression", () => {
     vi.resetModules();
     const moduleA = await import("#harness/instrumentation-config.js");
     const config = { functionId: "test.instrumentation.reimport.canary" };
-    moduleA.registerInstrumentationConfig(config, { agentName: "test-agent" });
+    await moduleA.registerInstrumentationConfig(config, { agentName: "test-agent" });
     const firstRef = (globalThis as Record<symbol, unknown>)[globalKey];
 
     vi.resetModules();
@@ -66,8 +66,33 @@ describe("instrumentation-config chunk-isolation regression", () => {
     const { registerInstrumentationConfig } = await import("#harness/instrumentation-config.js");
 
     const setup = vi.fn();
-    registerInstrumentationConfig({ setup }, { agentName: "weather-agent" });
+    await registerInstrumentationConfig({ setup }, { agentName: "weather-agent" });
 
     expect(setup).toHaveBeenCalledExactlyOnceWith({ agentName: "weather-agent" });
   });
+
+  // An `async setup` that registers a tracer provider after an `await` must be
+  // finished before the next Nitro plugin looks for one. Dropping the promise
+  // let eve's own telemetry setup win the race and silently starve the
+  // authored exporter.
+  it("resolves only after an async setup has finished", async () => {
+    vi.resetModules();
+    const { getInstrumentationConfig, registerInstrumentationConfig } =
+      await import("#harness/instrumentation-config.js");
+
+    const registered: string[] = [];
+    const config = {
+      setup: async () => {
+        await Promise.resolve();
+        registered.push("provider");
+      },
+    };
+
+    const pending = registerInstrumentationConfig(config, { agentName: "async-agent" });
+    expect(registered).toEqual([]);
+    await pending;
+
+    expect(registered).toEqual(["provider"]);
+    expect(getInstrumentationConfig()).toBe(config);
+  });
 });