Skip to content

fix: reset OtelJulHandler circuit breaker on reconfiguration - #1295

Open
om7057 wants to merge 1 commit into
jenkinsci:mainfrom
om7057:fix/oteljulhandler-circuit-breaker-reset-1291
Open

fix: reset OtelJulHandler circuit breaker on reconfiguration#1295
om7057 wants to merge 1 commit into
jenkinsci:mainfrom
om7057:fix/oteljulhandler-circuit-breaker-reset-1291

Conversation

@om7057

@om7057 om7057 commented Aug 9, 2026

Copy link
Copy Markdown

Fixes #1291

OtelJulHandler installs a circuit breaker on its JUL to OTLP log bridge: the first RuntimeException thrown from the emit path sets disabled = true, and publish() short-circuits on that flag from then on. Nothing in the class ever set it back to false. A single transient failure, for example the OTLP endpoint being briefly unavailable, or a reconfigure/shutdown race, permanently stops all controller and pipeline log export for the remaining life of the JVM. Traces are unaffected since they go through a separate exporter, which masks the outage.

The class already implements OpenTelemetryLifecycleListener but did not override afterConfiguration(ConfigProperties), so a JCasC reload or any other SDK reconfiguration never re-fetched the logger provider or reset the breaker. The only thing that previously cleared it was a brand new handler instance, meaning a JVM restart or plugin reload.

This change overrides afterConfiguration to re-fetch the logger provider from openTelemetry.getLogsBridge() and clear disabled, so a reconfigure recovers a previously tripped handler. The disabled field is also made volatile since publish() can run on arbitrary logging threads while afterConfiguration() runs on the reconfiguration thread.

Testing done

Added OtelJulHandlerTest with two tests:

  • handlerNeverRecoversWithoutAfterConfiguration: reproduces the original bug, a single emit failure permanently disables the handler and every subsequent publish() is a no-op.
  • afterConfigurationResetsTheCircuitBreaker: asserts calling afterConfiguration() clears the breaker and a subsequent publish() attempts to emit again.

Ran the full existing test suite locally on JDK 21 (matching this repo's Jenkinsfile): 238 tests run, 0 failures, 0 errors, 1 skipped (pre-existing skip, unrelated to this change). Also verified ./mvnw spotless:check and ./mvnw spotbugs:check pass clean.

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests that demonstrate the feature works or the issue is fixed

OtelJulHandler latches a disabled flag on the first RuntimeException
thrown while emitting a log record, and nothing ever cleared it. A
single transient failure, such as the OTLP endpoint being briefly
unreachable, silently and permanently stopped all controller and
pipeline log export for the remaining life of the JVM, with traces
unaffected since they use a separate exporter, which masked the
outage.

The class already implements OpenTelemetryLifecycleListener but did
not override afterConfiguration, so neither a JCasC reload nor any
other SDK reconfiguration ever re-fetched the logger provider or
reset the breaker.

Override afterConfiguration to re-fetch the logger provider and clear
disabled, so a reconfigure recovers a previously tripped handler.

Fixes jenkinsci#1291
@om7057
om7057 requested a review from a team as a code owner August 9, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OtelJulHandler permanently disables log export on first emit exception; never resets on reconfiguration

1 participant