Skip to content

feat(sdk-node, configuration)!: refactor of *env*-based code path for startNodeSDK - #6999

Open
trentm wants to merge 43 commits into
open-telemetry:mainfrom
trentm:trentm-startNodeSDK-refactor
Open

feat(sdk-node, configuration)!: refactor of *env*-based code path for startNodeSDK#6999
trentm wants to merge 43 commits into
open-telemetry:mainfrom
trentm:trentm-startNodeSDK-refactor

Conversation

@trentm

@trentm trentm commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Closes: #5945
Closes: #6107
Closes: #6488
Obsoletes: #6988

overview

This is a significant refactor (especially a breaking change to what the configuration package is about).

  • The main change is that the startNodeSdk() code path handling env-based config -- i.e. the thing that is intended to replace const sdk = new NodeSDK(); sdk.start() -- is fully implemented. startNodeSdk() now supports a full set of options (e.g. opts.views, opts.resourceDetectors, et al). Those options are not supported when using a config file, the point of declarative config is that the full config is defined by the YAML file. The options (including detailed diffs from new NodeSDK() options) are here: https://github.com/open-telemetry/opentelemetry-js/pull/6999/changes#diff-cf9f37462550254b82df2157b0ddaf0ba4c0a12782d46b803a076117740aac46R55-R206
  • The purview of the configuration package has been reduced to just having the parsing, validation, and TypeScript types for the ConfigurationModel. It no longer handles envvars at all. Those have moved to sdk-node.
  • A number of changes have been made around resource detectors, for both new NodeSDK() and startNodeSdk(). See section below for this.

Resource detector changes

We've been discussing how the SDK should create the Resource lately. (

The main question is how to handle the user configuring resource detectors given that:

  1. we currently support OTEL_NODE_RESOURCE_DETECTORS with names host, os, process, serviceinstance, env,
  2. but the spec (https://opentelemetry.io/docs/specs/otel/resource/sdk/#resource-detector-name) and declarative config have now defined host and service names that conflict.

This PR implements the following changes:

  • new NodeSDK(): The os detector has been added to the default set. Some history: osDetector was created in 2022 in feat: implement OSDetector #2927, hostDetector was added to the default detectors in Mar 2024 in feat(sdk-node): add HostDetector as default resource detector #4566. I see no reason not to add osDetector to the default set.

  • new NodeSDK(): The serviceinstance detector has been added to the default set, because service.instance.id is now stable in semconv.

  • Other that those two things, new NodeSDK() behaviour is unchanged.

  • When a user switches to using startNodeSdk() (and is not using a config file), then OTEL_NODE_RESOURCE_DETECTORS names will change to the spec names.

    • Removed 'env': OTEL_RESOURCE_ATTRIBUTES is now always read, OTEL_SERVICE_NAME is handled by the 'service' detector
    • Removed 'serviceinstance': Its functionality is now available with the 'service' name.
    • Added 'service': It handles OTEL_SERVICE_NAME and service.instance.id.
    • Removed 'os': os.* attributes are now provided with the 'host' detector name.
    • Updated 'host': It now provides both host.* and os.* attributes
  • startNodeSdk() will diag.warn appropriately when the old detector names are used, e.g.:

    > const { startNodeSdk } = require('@opentelemetry/sdk-node');
    > process.env.OTEL_NODE_RESOURCE_DETECTORS = 'serviceinstance,os,process';
    > const sdk = startNodeSdk();
    "serviceinstance" resource detector name is no longer supported, use "service" which populates 'service.instance.id' and reads OTEL_SERVICE_NAME (see https://opentelemetry.io/docs/specs/otel/resource/sdk/#resource-detector-name)
    "os" resource detector name is no longer supported, use "host" which populates 'host.*' and 'os.*' resource attributes (see https://opentelemetry.io/docs/specs/otel/resource/sdk/#resource-detector-name)
    
  • By default, i.e. when OTEL_NODE_RESOURCE_DETECTORS is not specified, the detector behaviour between new NodeSDK() and startNodeSdk() will be the same.

  • No breaking changes to the resources package were needed. Instead new resourceAttributesEnvDetector and serviceNameEnvDetector detectors were added and envDetector deprecated: 8b5b494#diff-4b2e7f2d8a9a7c6779860d70c98b1c3b0d11151296d044f1f261ae69b53ec98c

breaking changes

  • The configuration package API has changed (mostly to just a parseConfigFile() method and types).

Why startNodeSDK() -> startNodeSdk() name change?

I changed to startNodeSdk(). Why?

  • In my changes, I named the type for the thing that startNodeSdk() returns (the object with a shutdown() method). I named it NodeSdk, which seems natural. It is handy that this doesn't collide with the existing NodeSDK class -- although fair that they are close so there is some potential for confusion there. I'm thinking forward to when we deprecate NodeSDK soonish.

  • The fairly recent Browser SDK package has a const sdk: WebSdk = startBrowserSdk(). See https://github.com/open-telemetry/opentelemetry-browser/blob/main/packages/sdk/src/index.ts. This matches that change, for what its worth.

  • There aren't a lot of current examples of exported function names in opentelemetry-js.git where all-caps-or-not-for-acronyms is a consideration. Some of Marc's recentish work on exporters bias towards not all-caps, e.g.:

    76:export function mergeOtlpGrpcConfigurationWithDefaults(
    114:export function getOtlpGrpcDefaultConfiguration(): UnresolvedOtlpGrpcConfiguration {
    45:export function convertLegacyHttpOptions(
    220:export function getOtlpGrpcConfigurationFromEnv(
    46:export function mergeOtlpSharedConfigurationWithDefaults(
    

trentm added 25 commits July 27, 2026 17:05
This refactors the MeterProvider creation from declarative config
in the `startNodeSDK()` code path:
- fail-fast (and fallback to a no-op SDK) if the config has unknown
  values or values that aren't yet supported by the SDK
- fixes a number of missed cases
- adds a number of TODO for meter_provider config cases to follow-up on
  (Some of these are to keep this PR smaller, some are because
  sdk-metrics doesn't yet support everything the declarative config
  schema supports configuring.)

Refs: open-telemetry#6785
Refs: open-telemetry#6107 (comment)
Also limit reading of TLS files to *absolute paths* as required
by the declarative config.

Note: that doesn't work properly for the 'build-config-from-env-vars'
code path.

Update to have real TLS files with the usual maint:... npm script.
Real TLS files are required by createSslCredentials(), which was
being masked before by the just-warn semantics.
…fig file

**Note: This depends on the following PRs being merged first:
open-telemetry#6954,
open-telemetry#6962,
open-telemetry#6987.
**

This refactors and fixes creation of the Resource for the "startNodeSDK()" code path.
`setupResource(...)` -> `createResourceFromConfig(...)`

fix: The `defaultResource()` is now always included.
It isn't super clear from the declarative config spec that this is intended, but if it helps this is the OTel Java behaviour:
https://github.com/open-telemetry/opentelemetry-java/blob/d948e130ebf31fd41136ee2d72e28729047f8b9d/sdk-extensions/declarative-config/src/main/java/io/opentelemetry/sdk/autoconfigure/declarativeconfig/ResourceFactory.java#L36
This means that a declarative config like this will still get the default `service.name` and `telemetry.sdk.*` attributes.

```yaml
resource:
  attributes:
    - name: foo
      value: bar
```

fix: Attributes specified in `resource.attributes` properly override values from resource detectors.

fix: Fail-fast (i.e. throw, resulting in a no-op SDK) when there are unknown resource detectors specified.

fix: Change the behaviour of the "well-known" resource detectors to match that described in the spec (and the OTel Java impl):
https://opentelemetry.io/docs/specs/otel/resource/sdk/#resource-detector-name

This is a **breaking change** for `startNodeSDK()` usage, and there is some confusion on resource detector names to clarify.
- The spec'd behaviour for the "host" and "service" detector names are **different** to the behaviour of the "host" and "service" detector names used in `OTEL_NODE_RESOURCE_DETECTORS`.
- The spec says "host": "Populates `host.*` and `os.*` attributes."
  This is equivalent to "host" **and "os"** in `OTEL_NODE_RESOURCE_DETECTORS`.
- The spec says "service": "Populates `service.name` based on `OTEL_SERVICE_NAME` environment variable; populates `service.instance.id` [...]".
  This is close, **but not quite**, to "service" and "env" in `OTEL_NODE_RESOURCE_DETECTORS`.
- "Not quite" the same, because if we used the existing `envDetector` for handling a file-based config, we would **incorrectly** read resource attributes from `OTEL_RESOURCE_ATTRIBUTES`.
- I struggled with this for a while, figuring Java would have the same confusion.
  OTel Java *does* have an option similar to `OTEL_NODE_RESOURCE_DETECTORS`.
  It is `OTEL_JAVA_ENABLED_RESOURCE_PROVIDERS` (or the `otel.java.enabled.resource.providers` system property).
  However, the strings passed to this setting are **not** convenient short strings, they are instead fully qualified class names, e.g.:
  ```bash
  export OTEL_JAVA_ENABLED_RESOURCE_PROVIDERS=io.opentelemetry.sdk.autoconfigure.EnvironmentResourceProvider,io.opentelemetry.instrumentation.resources.HostResourceProvider
  ```
  There is no confusion in Java-land between those settings and "host" in a declarative config file.
- Options:
  1. We treat the names used in `OTEL_NODE_RESOURCE_DETECTORS` and those in `detectors: ...` in a declarative config file as **not in the same namespace**. They are unrelated names that cannot be mapped to each other.
  2. We *break* the meaning of the current "host", "service" and "env" detectors used in `OTEL_NODE_RESOURCE_DETECTORS` to match what the spec now uses.
  In this PR I have opted for Option 1.
  (Option 2 means breaking current de facto stable behaviour.)
  There are implications to this discussed below.

Implications of `OTEL_NODE_RESOURCE_DETECTORS names !== declarative config detectors names`.
The main implication is that `FileConfigFactory` in the "configuration" package **cannot express `OTEL_NODE_RESOURCE_DETECTORS` in a `ConfigurationModel`.**
This was somewhat already true, because there is no way to express `OTEL_NODE_RESOURCE_DETECTORS=all` in `ConfigurationModel`.
I have ideas for changes in the sdk-node and configuration packages around this, but I propose to handle them in a separate issue/PR.
(I'll link back to this one.)

This PR also tweaks the dev tool "configuration/scripts/parse-config.mjs" to read config from the environment if an argument is not provided.  This is useful for inspecting what the "config from environment" code path is generating as a ConfigurationModel.

Refs: open-telemetry#6107 (comment)
… full effort is done on startNodeSDK() usage with env-based config
… path

I believe this is basically working, but I haven't yet added tests.
trentm added 8 commits August 14, 2026 14:05
…EnvDetector`, mark `serviceInstanceIdDetector` stable

Deprecate `envDetector` in favor of separate `resourceAttributesEnvDetector` and `serviceNameEnvDetector`.
Also mark `serviceInstanceIdDetector` as stable (the `service.instance.id` semconv attribute is now stable).
…IBUTES, when 'service' detector is being used
…default set; note serviceInstanceIdDetector as stable
…ase'; consistent with opts.resourceDetectors
…rks that way. No need for out-of-band null to signal this. Also consistent with opts.propagators and opts.resourceDetectors
@trentm
trentm marked this pull request as ready for review August 15, 2026 00:04
@trentm
trentm requested a review from a team as a code owner August 15, 2026 00:04
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Aug 15, 2026

Copy link
Copy Markdown

Pull request dashboard status

Waiting on reviewers · refreshed 2026-09-04 11:22 UTC

Review the latest changes.

Also blocked by: Merge conflicts.

Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

Comment thread experimental/packages/configuration/scripts/parse-config.mjs Outdated
@trentm
trentm requested a review from JacksonWeber August 17, 2026 18:22
@trentm

trentm commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

The CodeQL check failure is because of a current GH incident.

@maryliag

Copy link
Copy Markdown
Member

big change, thank you for working on it, but I'm not sure I agree 😅

I created the configuration package so that one would be responsible for the setup of the config that any other package would need, could be from a file or could be env.

Other repos have created something similar to what you have here, with the config package looking at only file, but then when using, you need to also have an env config path, but this makes hard to find all around the code the places where something should be updated. If you have a single package that handles config (file and env), it's all contained and it's easier to find all existing config, tests and so on.
Because I saw other repos doing like you're changing here I actually asked them why it was done this way and explained my idea, and they agree that my idea was more organized and they regretted not having done this way, because the maintanance is harder the one you're doing now.

With the way I created whoever is calling the config object, doesn't have to care about the setup itself, because it was already done, it just calls the factory and that's it. But now you had to create startNodeSdkFromConfig and startNodeSdkFromEnv and you will have to keep repeating this pattern to every other package than wants to use it, instead of a more straight forward const configFactory: ConfigFactory = createConfigFactory();.

So you're making a big change that would be the opposite of my initial intention, so I would like to understand more what is the advantage of those changes.

@maryliag

Copy link
Copy Markdown
Member

I hear the impedance mismatch case for OTEL_NODE_RESOURCE_DETECTORS=all etc., but I'd rather solve those with an explicit env-overrides mechanism inside configuration than by moving all env parsing into sdk-node.

Some options in between the current version and the new could be:

Keep unified for the env → model layer, but let SDKs augment. configuration handles the declarative-config-expressible env vars (which is most of them), and each SDK adds its own layer for Node-specific env vars only.

Keep the split you have has now, but make the common shape reusable. Instead of full duplication between create-from-config and create-from-env, each helper function takes a common "resolved config" shape and both entry points build it their own way. Cuts the "two places to update per new feature" cost.

@trentm

trentm commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

(Sorry, this got long.)

It seems appealing to have a design where all sources of config can be read and validated into one common model (ConfigurationModel). Then usage of that config model (to create SDK components) can have one single code path; theoretically making its maintenance simpler. If we had many sources of config and many users of the config model, then the benefit would compound.

However, we don't have that many of either. There are two sources of config: env and YAML config file. There is currently one user of the config model: the sdk-node package.

  • The browser SDK is very unlikely to use file-based or env-based config. If they were to consider YAML-string-based config, they'd want a tree-shakeable implementation that kept env and YAML-based create() code separate.
  • Yes, we could eventually have sdk-bun and/or sdk-deno. Neither seems likely in the near term. To support them, we could move most of the create-from-{env,config}.ts support to a shared lib that all three could use.

Creating a ConfigurationModel from envvars, to then be used to create SDK components, is a leaky abstraction.

One leaky case is OTEL_NODE_RESOURCE_DETECTORS=all.
OTEL_NODE_EXPERIMENTAL_SDK_METRICS=true might be considered another one.

I'd rather solve those with an explicit env-overrides mechanism inside configuration than by moving all env parsing into sdk-node.

Sure, we could try that. Perhaps something like this customizeConfigForEnv:

function startNodeSdk() {
  const factory = createConfigFactory({
    customizeConfigForEnv: (config) => {
      // Read OTEL_NODE_RESOURCE_DETECTORS and add `config['detection/development'].detectors`...
    }
  });
  const config = factory.getConfig();
  // ...
}

Another leak is error handling. The spec says that (a) creating the SDK from env should warn and gracefully ignore invalid env vars (https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#type-specific-guidance) and (b) creating the SDK from config should fail-fast (https://opentelemetry.io/docs/specs/otel/configuration/sdk/#create).

If we have one common create(config) code path for both env-based and file-based config, then we need to pass through some state (an errorMode or what the config source was) and most/every create<Thing>FromConfig() function needs to add if-guards on the error mode.

As well, good error messages are harder to make when the configuration source is abstracted away. In the env-specific createPropagatorFromEnv(), if an unknown propagator name is given, the current warning to the user is: diag.warn(`unknown propagator from "OTEL_PROPAGATORS": "${name}"`);. Being able to name OTEL_PROPAGATORS is very helpful to the user. To do the equivalent in a generic createPropagatorFromConfig(config) requires more code (making the impl more complex), a leak of the abstraction (duplicating the OTEL_PROPAGATOR envvar used).


The following spec discussion suggests a direction where env-based configuration should not evolve with config-based. open-telemetry/opentelemetry-specification#3967 (comment)

Not sure what you mean by freeze. Perhaps prohibit the expansion of the allowed values? I.e. don't allow expansion of OTEL_METRICS_EXPORTER to support a new new well known value.

This is only an issue discussion and not spec, but it implies an intent that when/if we add support for PluginComponentProvider, then the env-based configuration code path should not use that component provider. If we have shared create<Thing>FromConfig() functions for both env-based and config-based, then there is another if-guard needed. This is another leak.


Q1: What options should startNodeSdk() offer?

As a replacement for new NodeSDK(...) (ignoring file-based config for a moment), we likely want to offer similar options like opts.spanProcessors, opts.views, opts.resourceDetectors. The idea here is that @opentelemetry/sdk-node provides a convenient way to setup the SDK mostly with envvars, but you can add some logic that the envvars don't support. "Convenient" because you don't have to resort to using sdk-trace, sdk-logs, sdk-metrics, and all the exporter-* packages.

Q2: Assuming we offer similar options, should those options be used when file-based config is used?
I.e. if I set OTEL_CONFIG_FILE=./my-sdk-config.yaml and call startNodeSdk({ spanProcessors: [...] }), should opts.spanProcessors get used?

I think no. If using a config file, then the config should only come from
that config file.
This is how it is implemented in the current PR.

Assuming you agree with that call, then a single create<Thing>FromConfig() code path for env-based and file-based config is complicated by opts handling. Separate create<Thing>FromOptsAndEnv() and create<Thing>FromConfig() functions are shorter and clearer.


Regarding maintenance.

[...] because the maintanance is harder the one you're doing now.

I disagree. Or, at the least, I think it is a wash deciding ahead of time which way is less maintenance.

Above I've argued that error handling mode, good error messages, PluginComponentProvider usage, and handling startNodeSdk options (for backward compat with new NodeSDK()) would make unified create<Thing>FromConfig() functions more complex than separate create<Thing>FromOptsAndEnv() and create<Thing>FromConfig() functions.

The current PR adds full support for startNodeSdk options (type StartSdkOptions), adds complete support for env-based config via startNodeSdk, adds some to the resources package to solve the Resource creation questions, and the code size is no larger. (The PR reduces the total number of lines, but a large part of that is the removal of the "EnvironmentConfigFactory.test.ts".)


const instrumentations = opts?.instrumentations?.flat();
if (instrumentations) {
registerInstrumentations({ instrumentations });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question]: any reason to register the instrumentations before providers are resolved? AFAIK the method sets the different providers (logs, traces, metrics) for each instrumentation so if providers are set 1st the instrumentations will get the right tracer, logger and meter then the if in L156 is not necessary

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

David,
A similar point was brought up recently here: #6989 (comment)
See my response on that thread.

tl;dr: Yes, you are right. I do want to change to calling registerInstrumentations(...) after the providers have been registered. However, as with the PR above, I don't want to add this change to this PR because:

  1. It keeps this (already controversial) PR a little less busy. Both startNodeSDK() and new NodeSDK() before this change are doing registerInstrumentations(...) before creating and registering the providers.
  2. My proposed change to the ConfigProvider PR -- feat(sdk-node,instrumentation,instrumentation-http,api-config,configuration): add declarative config support for instrumentation/development #6868 (comment) -- is doing this move of registerInstrumentations() to be after creating/registering the providers. See here: 45e9a42...trentm:opentelemetry-js:inst-config-provider#diff-44b3cba79d8ee04c3efd823142cf926e5effe45f31c1199ff8eae607c3500885R86-R116

(For historical interest: This PR is where registerInstrumentations() was moved to be before creating/registering the providers. This was because, at the time, sdk.start() was async to deal with async resource detectors. That's no longer relevant. #3502)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

Replace serviceInstanceDetector with serviceDetector add warning for invalid values Add all fallback options for config model

4 participants