All customs spans broken in Puter after upgrading opentelemetry

[KernelDeimos]

Hello, I maintain the backend for Puter. For years I've had opentelemetry and jaeger working together to diagnose performance issues but after upgrading it appears we've lost this. I've spent the past 2 days trying to get it back to no avail. This week we're planning on going full-focus on performance and this is really messing up our plans.

It all started after we upgraded opentelemetry packages. I wasn't able to run Puter with the performance module enabled anymore until these two commits fixed it:

• the fix, and this fix for an error in the fix

This fix itself seems pretty harmless - we pass spanProcessors in the constructor of the provider now instead of calling addSpanProcessor. While I feel introducing this regression was unnecessary it wasn't a big deal once we found out what we needed to do. (however if the method was replaced with a method that throws an exception explaining the regression we would have found this much quicker)

If I try to run everything the same as before I don't see any custom spans. In jaeger, I only ever see the built-in spans for things like HTTP requests and accessing files on the system. I've tried a lot of different things to fix it - re-ordering things, using getTracer from getTraceProvider() instead, calling getTracer where specific things are instrumented as the documentation recommends, etc; nothing is working. I'm writing this now because I don't know what to try next - I have run out of ideas.

For now we have downgraded and everything is working again.

[LarryHu0217]

Based on the linked commits, I would first treat this as two setup problems rather than a regression in custom spans.

First, the upgrade mixes incompatible OpenTelemetry release families:

• @opentelemetry/auto-instrumentations-node 0.49.x
• @opentelemetry/sdk-metrics 1.30.x
• @opentelemetry/sdk-node and the OTLP exporter 0.208.x

The official compatibility matrix pairs stable 1.30.x packages with experimental 0.57.x packages. More directly, sdk-node 0.208.0 depends on sdk-metrics, sdk-trace-base, and sdk-trace-node 2.2.0. The Puter lockfile therefore resolves multiple SDK generations.

I would first select one compatible release family, regenerate the lockfile in a branch, and check:

npm ls @opentelemetry/api \
  @opentelemetry/sdk-node \
  @opentelemetry/sdk-trace-node \
  @opentelemetry/sdk-trace-base \
  @opentelemetry/sdk-metrics \
  @opentelemetry/instrumentation

Second, TelemetryService currently registers two tracer providers: it calls provider.register() and then starts a NodeSDK, which constructs and attempts to register another provider. NodeSDK.start() performs that global registration. Only one provider should own both the custom spans and automatic instrumentation.

A simpler shape is:

const exporter = this.#getConfiguredExporter();

const sdk = new NodeSDK({
  resource,
  spanProcessors: [new BatchSpanProcessor(exporter)],
  metricReader: new PeriodicExportingMetricReader({
    exporter: new ConsoleMetricExporter(),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

this.sdk = sdk;
this.exporter = exporter;
this.tracer_ = opentelemetry.trace.getTracer('puter-tracer');

Remove the separate NodeTracerProvider and provider.register() path, and make sure the metric reader and span processor imports come from the same compatible SDK family.

For a focused check, create and end one custom-smoke span, then call sdk.shutdown() before process exit so the batch processor flushes. Enabling DiagConsoleLogger at DEBUG level should also reveal duplicate global-provider registration or instrumentation-version problems.

I cannot tell which mismatch is the sole cause without running the Puter setup, but both are concrete problems in the linked upgrade and should be eliminated before investigating a deeper SDK regression.