ClassCircularityError: java/lang/invoke/MethodHandle$1 during agent premain on JDK 17 when OTEL_JAVAAGENT_CONFIGURATION_FILE is set (1.8.0 / 1.10.0)

[ericmustin]

Summary

EDOT Java 1.8.0 and 1.10.0 crash during agent premain on JDK 17 with java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1 when OTEL_JAVAAGENT_CONFIGURATION_FILE is set to a properties file containing at least one real otel.* entry. 1.7.0, 1.9.0, and 1.11.0 are unaffected. An empty file or delivering the same properties via env vars does not trigger.

The bug is rooted in upstream opentelemetry-java-instrumentation PR #15091 and surfaces specifically in distributions (including EDOT) due to additional SPI / autoconfigure class-loading during premain. The bare upstream agent jar does not reproduce.

Stack trace

Picked up JAVA_TOOL_OPTIONS: -javaagent:/javaagent.jar
[main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.8.0
java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1
    at java.base/java.lang.invoke.MethodHandle.customize(MethodHandle.java:1741)
    at java.base/java.lang.invoke.MethodHandle.maybeCustomize(MethodHandle.java:1731)
    at java.base/java.lang.invoke.Invokers.maybeCustomize(Invokers.java:634)
    at java.base/java.lang.invoke.Invokers.checkCustomized(Invokers.java:628)
    at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
    at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
    at io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties.<init>(DefaultConfigProperties.java:79)
    at io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties.create(DefaultConfigProperties.java:54)
    at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.buildImpl(AutoConfiguredOpenTelemetrySdkBuilder.java:452)
    ...
    at co.elastic.otel.agent.ElasticAgent.premain(ElasticAgent.java:39)

The visible crash site (DefaultConfigProperties.<init>:79) is incidental — the actual race is on the lambda sites in upstream ConfigurationFile.loadConfigFile() (see Mechanism).

Reproducer

Parameterized by EDOT version. Dockerfile/README below

# syntax=docker/dockerfile:1.4
ARG EDOT_VERSION=1.8.0
FROM docker.elastic.co/observability/elastic-otel-javaagent:${EDOT_VERSION} AS otel-agent
FROM eclipse-temurin:17-jdk
COPY --from=otel-agent /javaagent.jar /javaagent.jar
RUN mkdir -p /opt/otel-repro/extensions \
 && printf 'otel.javaagent.extensions=/opt/otel-repro/extensions\n' > /opt/otel-repro/config.properties
ENV OTEL_JAVAAGENT_CONFIGURATION_FILE=/opt/otel-repro/config.properties
ENV JAVA_TOOL_OPTIONS=-javaagent:/javaagent.jar
CMD ["java", "-version"]

# Build
docker build --platform=linux/amd64 --build-arg EDOT_VERSION=1.8.0 -t repro:1.8.0 .

# Crash repro — fires the ClassCircularityError above
docker run --platform=linux/amd64 --rm repro:1.8.0

# Workaround — same property delivered via env var, JVM starts cleanly
docker run --platform=linux/amd64 --rm \
  -e OTEL_JAVAAGENT_CONFIGURATION_FILE= \
  -e OTEL_JAVAAGENT_EXTENSIONS=/opt/otel-repro/extensions \
  repro:1.8.0

Crash 3/3 deterministic on linux/amd64 (Apple Silicon via Rosetta 2; native arm64 doesn't reliably reproduce due to JIT timing). Cycle EDOT_VERSION through 1.7.0 / 1.9.0 / 1.10.0 / 1.11.0 to verify the matrix below.

Affected versions

EDOT	Upstream agent	Result
1.7.0	2.21.0	clean
1.8.0	2.22.0	crash
1.9.0	2.24.0	clean
1.10.0	2.26.1	crash
1.11.0	2.27.0	clean

What rules out other causes

Hypothesis	Test	Result
Indy advice mode is the trigger	OTEL_JAVAAGENT_EXPERIMENTAL_INDY=false on 1.8.0	still crashes
File I/O alone is the trigger	OTEL_JAVAAGENT_CONFIGURATION_FILE=/dev/null (empty file)	clean
The extensions feature itself is the trigger	Drop file, deliver as OTEL_JAVAAGENT_EXTENSIONS env var	clean
Multi-agent transformer pressure required	Single javaagent	crashes
Application code matters	java -version, no app	crashes
Bare upstream agent reproduces	Replace EDOT image with v2.22.0 opentelemetry-javaagent.jar from upstream releases, identical setup	clean

The trigger is the configuration-file code path processing real properties during premain on the affected agent versions, in combination with EDOT's SPI loadout.

Mechanism

ConfigurationFile.loadConfigFile() (in upstream, unchanged across all five tags) ends with:

return properties.entrySet().stream()
    .collect(toMap(
        e -> e.getKey().toString(),
        e -> e.getValue().toString()));

→ https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/v2.22.0/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/config/ConfigurationFile.java#L80

When the file has content, those two lambda sites need to link via LambdaMetafactory → BootstrapMethodInvoker → Invokers.checkCustomized → MethodHandle.maybeCustomize → MethodHandle.customize → load MethodHandle$1. On JDK 17 they race with MethodHandle$1's ongoing definition.

In upstream v2.21.0, AgentInstaller.start() called setupUnsafe(inst) before reaching the SDK autoconfigure pipeline. That call ran UnsafeInitializer.initialize() → SunMiscUnsafeGenerator.generateUnsafe(), which generated a replacement sun.misc.Unsafe class via classloader machinery and incidentally finished defining MethodHandle$1 before any later code asked the JDK to link a lambda.

Upstream PR #15091 "Don't use unsafe on jdk23+" (in 2.22.0) removed setupUnsafe(inst) and deleted UnsafeInitializer entirely:

@@ -107,7 +107,6 @@ public static void installBytebuddyAgent(
     logVersionInfo();
     if (earlyConfig.getBoolean(JAVAAGENT_ENABLED_CONFIG, true)) {
-      setupUnsafe(inst);
       List<AgentListener> agentListeners = loadOrdered(AgentListener.class, extensionClassLoader);
       installBytebuddyAgent(inst, extensionClassLoader, agentListeners, earlyConfig);
     }

After 2.22.0 nothing on the agent's premain path reliably pre-warms MethodHandle$1. Whether the lambdas in loadConfigFile() link cleanly now depends on what other class-loading happens during premain.

The bare upstream agent doesn't load enough classes during premain to surface the race — MethodHandle$1 happens to already be defined by the time the lambdas link. EDOT's distribution adds enough SPI / autoconfigure class-loading work (ElasticAutoConfigurationCustomizerProvider, ConfigLoggingAgentListener, ElasticDistroResourceProvider, etc., all auto-discovered via META-INF/services) to push the linker into the bad window. That's why the bug is EDOT-visible but bare-upstream-invisible.

The clean upstream releases (2.24.0, 2.27.0) didn't fix the race — setupUnsafe did not come back, UnsafeInitializer.java is still gone in main. Unrelated startup-ordering refactors (signature changes in installBytebuddyAgent, setBootstrapPackages reordering, #17241 / #16638 / #17824) happen to inject other class loading earlier and incidentally warm the linker. 2.26.0's package-rename refactor shifted ordering back into the bad window. Any future startup refactor can re-surface the race.

Suggested fix

EDOT-side (workaround): add a one-line MethodHandle$1 preload at the top of co.elastic.otel.agent.ElasticAgent.premain, before OpenTelemetryAgent.premain is called:

// preload MethodHandle internals to avoid JDK 17 ClassCircularityError on
// the first invokedynamic site that runs during premain (regression in
// upstream agent 2.22.0+ — see <UPSTREAM_PR_LINK_IF_FILED>)
((java.util.function.Function<Object, Object>) (x -> x)).apply(null);

This makes EDOT robust regardless of which upstream agent version it pins. Same shape as upstream's existing ThreadLocalRandom.current() preload (#14030).

Upstream-side (better long-term): sibling preload in AgentInstaller.installBytebuddyAgent adjacent to the existing ThreadLocalRandom.current() preload at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/AgentInstaller.java#L114-L120. Fixes this for every distribution, not just EDOT. Open question whether to escalate to upstream after the EDOT-side fix lands.

Workaround for affected users

Deliver configuration-file properties as env vars or -D system properties instead. Replace otel.javaagent.extensions=/path in the file with OTEL_JAVAAGENT_EXTENSIONS=/path (env var translation: lowercase, replace dots with underscores, uppercase, prepend OTEL_). Functionally identical input path to the agent.

OTEL_JAVAAGENT_EXPERIMENTAL_INDY=false does not prevent this — confirmed empirically.

Environment

JDK 17.0.18 Temurin, linux/amd64. Reproduced on Apple Silicon via Rosetta 2 (--platform=linux/amd64); native linux/arm64 does not reliably reproduce due to JIT timing differences.

[SylvainJuge]

Hi @ericmustin , did you manage to reproduce this with the upstream opentelemetry instrumentation agent ? Before asking upstream I think it would be better to clarify if it's only upstream or EDOT specific.

There was a related fix merged (but not included in changelog) that is included in the 1.11.0 EDOT release: #1064. I thought that it was only happening since the upstream update for that version, so it was not technically a regression.

What you report here seems to indicate that it might be an issue that is affecting 1.8.0 and 1.10.0, but since the latest version fixes it there is nothing for us to do (users should use the latest available version).

What I would suggest as next steps for investigation:

• try to reproduce this with upstream instrumentation, in particular with the versions mentionned in the table
• if the issue is not reproducible with upstream, then it means it's only EDOT specific and we know that it's already fixed in the latest version, so there is nothing to do. We could add this to the release notes to indicate there is a known issue with the affected EDOT versions.
• if the issue is reproducible with upstream, then we need to open an issue upstream with a way to reproduce it as you provided here.

[ericmustin]

try to reproduce this with upstream instrumentation, in particular with the versions mentionned in the table

I was unable to repro with merely upstream

I believe this was a regression in 1.8/1.10 but given that this is fixed in 1.11 there's nothing further to do except upgrade. It might be useful to know whether moving OTEL_JAVAAGENT_CONFIGURATION_FILE to env vars would be an effective workaround on 1.8, but I'll leave that up to you and go ahead and close this issue.

[SylvainJuge]

This issue was closed because the 1.11.0 version fixes the issue:

• workaround 1: update to 1.11.0 EDOT or later

I'm adding this for further reference if needed:

• upstream 2.22.0 version (which is used in 1.8.0) is not impacted by this, so we can assume this is limited to EDOT and not part of upstream.
• multiple EDOT versions are impacted: 1.8.0 1.10.0, however surprisingly the 1.9.0 is not.
• issue is triggered when using Java 17 combined with OTEL_JAVAAGENT_CONFIGURATION_FILE environment variable
  • workaround 2: use another JDK version (tested with JDK 21), but that may not be always possible
  • workaround 3: use JVM arguments to provide the otel configuration through java system properties.

[ericmustin]

@SylvainJuge fwiw I believe 1.11.0 does not fully resolve the issue. will try to provide further details as I have them

[SylvainJuge]

@SylvainJuge fwiw I believe 1.11.0 does not fully resolve the issue. will try to provide further details as I have them

We haven't had any reports of a similar issue either through public issues and forum nor on our internal customer support, so having more details would be more than welcome here.

Also, this is the kind of issue that fails at startup, what are the symptoms that you observed on your side that make you believe it could be this ? For example, did your application failed to start or enter a restart loop ?

[jameyg42]

was able to reproduce w/ java-17 and 1.11.0. Other near-identical setups are not recreating the problem (e.g. updating to 1.11.0 did resolve the issue for some), so we're trying to reduce the problem down to a simpler form and will submit up through the customer support ticket. There's a possibility (based on the single notable difference between working and non-working) the Error is actually happening w/ the keytool command run at ENTRYPOINT and not the java launcher since the javaagent is being injected using the JAVA_TOOL_OPTIONS environment variable (we're currently switching things up to use JDK_JAVA_OPTIONS instead)

[SylvainJuge]

This is one of the main downsides of using JAVA_TOOL_OPTIONS, it is applied broadly to all programs running on the JVM, including jps, keytool which are commonly used in automation scripts.

The symptom of this is usually seeing more than one instance of Picked up JAVA_TOOL_OPTIONS message in the standard output, meaning multiple JVM instances are using this environment variable.

Edit: if the issue is actually due to trying to trigger instrumentation on JVM/JDK binaries, then we had a defensive check against that in the Elastic APM Java agent (see test ) that we could attempt to port to upstream opentelemetry instrumentation.

[jameyg42]

i'm not sure it this helps, but i was able to confirm that with
openjdk version "17.0.19" 2026-04-21
OpenJDK Runtime Environment Temurin-17.0.19+10 (build 17.0.19+10)
OpenJDK 64-Bit Server VM Temurin-17.0.19+10 (build 17.0.19+10, mixed mode, sharing)
[main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.11.0

simply running keytool or jshell causes the java.lang.ClassCircularityError. We don't even need to set
OTEL_JAVAAGENT_CONFIGURATION_FILE (or any other OTEL_* variable) to trigger. Unfortunately, this is NOT in fact the cause of the issue we're experiencing (my previous hunch was wrong).

[SylvainJuge]

I can confirm this issue happens with the following:

• EDOT Java, including latest version
• Java 17
• jps command line tool

So the first conclusion is that the ClassCircularityError is specific to the EDOT Java and is likely triggered for all the JDK command line tools.

So far, changing a single item in the list above seems to NOT trigger the issue:

• using upstream opentelemetry instrumentation
• using a more recent java version (tested with 25)
• using a "real" application, not a command-line tool from the JDK (tested with Tomcat 10.x)

Are there any cases where this issue is triggered on a "real application" and not a JDK CLI tool ?
If not, then implementing an early detection might be a possible work-around.

On the other side, the fact that the JAVA_TOOL_OPTIONS with -javaagent: is being set before invoking tools like jps is something that should be considered to be a setup anomaly, as it makes the instrumentation also applied to any of those tools invocations.

For example, with jps it adds a ~10s startup time overhead due to instrumentation, which is something you definitely want to avoid, in particular if it's called in a loop.

If there is no alternative to using a broad JAVA_TOOL_OPTIONS environment variable, then making the instrumentation detect and self-disable on startup is a possible improvement.

[SylvainJuge]

I've opened the following upstream PR to bypass instrumentation for all JDK tools by default, so this issue might be solved once this is merged and the upstream version is updated.