OtelJulHandler permanently disables log export on first emit exception; never resets on reconfiguration

[bixu]

Provenance. This report was investigated and written collaboratively by a human maintainer and Claude (Anthropic's coding agent). The findings are based on reproduction in a local lab and on direct observation of a production Jenkins controller.

Versions

• opentelemetry plugin 3.1589.ve81b_b_fa_927d5 (latest release at time of writing)
• OpenTelemetry Java SDK 1.54.1
• Jenkins 2.541.2

Summary

OtelJulHandler installs a latching circuit breaker on its JUL→OTLP log bridge. The first RuntimeException thrown from the emit path permanently disables all log forwarding for the life of the JVM, with no retry and no reset. A transient failure (for example the OTLP endpoint being briefly unavailable, or a reconfigure/shutdown race) therefore silently stops all controller and pipeline log export. Traces are unaffected (separate exporter), which masks the outage.

Details

In OtelJulHandler.java:

• private boolean disabled = false; (the "circuit breaker" field)
• publish() returns immediately when disabled is true (early guard)
• the surrounding try/catch (RuntimeException e) prints "Exception sending logs to OTLP endpoint, disable OTelJulHandler" to System.err and sets disabled = true
• disabled is never set back to false anywhere

postConstruct() captures loggerProvider = openTelemetry.getLogsBridge() once. The class implements OpenTelemetryLifecycleListener but does not override afterConfiguration(...), so a JCasC/SDK reconfigure neither re-fetches the provider nor clears the breaker. The only thing that resets it is a brand-new handler instance (a JVM restart or plugin reload).

Impact observed in production

After a controller restart, OTLP log export stopped and never resumed, while traces kept flowing over the same endpoint. Attempting to recover it, both a safe restart (fresh JVM) and a live JCasC reload failed to restore log export. We ultimately had to route controller logs through journald instead to get them back.

Deterministic reproduction

This test isolates the defect: a single emit failure permanently disables the handler, even after the endpoint recovers. It compiles and passes green against 3.1589 (BUILD SUCCESS), needs no Collector, and has no timing dependence.

package io.jenkins.plugins.opentelemetry.init;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

import io.opentelemetry.api.logs.LogRecordBuilder;
import io.opentelemetry.api.logs.Logger;
import io.opentelemetry.api.logs.LoggerProvider;
import java.lang.reflect.Field;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.logging.Level;
import java.util.logging.LogRecord;
import org.junit.jupiter.api.Test;
import org.mockito.Answers;

class OtelJulHandlerCircuitBreakerTest {

    @Test
    void handlerNeverRecoversAfterASingleEmitFailure() throws Exception {
        // A LoggerProvider whose emit() throws on the first call, then succeeds -
        // i.e. a transient failure such as the OTLP endpoint being briefly unavailable.
        AtomicInteger emitAttempts = new AtomicInteger();
        LogRecordBuilder builder = mock(LogRecordBuilder.class, Answers.RETURNS_SELF);
        doAnswer(inv -> {
                    if (emitAttempts.getAndIncrement() == 0) {
                        throw new IllegalStateException("simulated transient OTLP emit failure");
                    }
                    return null;
                })
                .when(builder)
                .emit();

        Logger otelLogger = mock(Logger.class);
        when(otelLogger.logRecordBuilder()).thenReturn(builder);
        LoggerProvider provider = mock(LoggerProvider.class);
        when(provider.get(anyString())).thenReturn(otelLogger);

        OtelJulHandler handler = new OtelJulHandler();
        Field loggerProviderField = OtelJulHandler.class.getDeclaredField("loggerProvider");
        loggerProviderField.setAccessible(true);
        loggerProviderField.set(handler, provider);

        // 1st record: emit() throws -> the circuit breaker latches disabled = true.
        handler.publish(new LogRecord(Level.INFO, "first record - endpoint transiently down"));

        // 2nd record: the endpoint has recovered (emit() would now succeed) ...
        handler.publish(new LogRecord(Level.INFO, "second record - endpoint back up"));

        // ... but the handler short-circuits on `disabled` and never attempts emit again.
        assertEquals(
                1,
                emitAttempts.get(),
                "emit() was attempted only once; after a single failure the handler drops every "
                        + "subsequent log forever, even though the endpoint recovered");

        Field disabledField = OtelJulHandler.class.getDeclaredField("disabled");
        disabledField.setAccessible(true);
        assertTrue(
                (boolean) disabledField.get(handler),
                "disabled latched true and there is no code path (retry, backoff, or "
                        + "afterConfiguration reset) that ever sets it back to false");
    }
}

Suggested fix

Either (or both):

• Reset the breaker in afterConfiguration(ConfigProperties): clear disabled and re-fetch loggerProvider from openTelemetry.getLogsBridge(), so a reconfigure recovers a previously-tripped handler.
• Replace the permanent latch with bounded retry/backoff (or a periodic half-open probe), so a transient emit failure does not permanently disable logging.

Happy to open a PR if that would help.