Skip to content

fix: bound preOnline timeout so RTT-sensitive agents are not kicked offline - #1294

Open
om7057 wants to merge 1 commit into
jenkinsci:mainfrom
om7057:fix/preonline-unbounded-timeout-1285
Open

fix: bound preOnline timeout so RTT-sensitive agents are not kicked offline#1294
om7057 wants to merge 1 commit into
jenkinsci:mainfrom
om7057:fix/preonline-unbounded-timeout-1285

Conversation

@om7057

@om7057 om7057 commented Aug 9, 2026

Copy link
Copy Markdown

Fixes #1285

preOnline() waited on the agent's OpenTelemetry SDK configuration RPC with an unbounded Future.get(), while the sibling afterConfiguration() method already bounds the same kind of call to 10 seconds. Since preOnline() runs synchronously inside SlaveComputer.setChannel(), blocking there for an unbounded amount of time gives unrelated NAT idle timeouts or network disturbances a much bigger window to close the channel before the online handshake finishes. This mostly affects agents connecting over higher RTT links, where the configuration RPC naturally takes longer to complete.

This change bounds the wait to a configurable timeout, defaulting to 10 seconds to match afterConfiguration(), and logs a warning instead of blocking indefinitely when it elapses. The agent is still allowed online even if its OpenTelemetry configuration RPC has not completed in time.

The timeout is configurable via otel.instrumentation.jenkins.agent.pre_online.timeout (ConfigurationKey.OTEL_INSTRUMENTATION_JENKINS_AGENT_PRE_ONLINE_TIMEOUT), read the same way other OTel SDK properties are, through ConfigProperties.getDuration(...).

Testing done

Added OpenTelemetryConfigurerComputerListenerTest with two tests:

  • preOnlineReturnsPromptlyWhenAgentConfigurationRpcNeverCompletes: uses a hudson.remoting.Future that never completes to simulate a slow or RTT-bound channel, and asserts preOnline() returns well within a few seconds once the configured timeout elapses, instead of blocking indefinitely.
  • preOnlineDoesNotPropagateExecutionExceptionFromAgent: asserts a failed configuration RPC does not propagate out of preOnline().

Ran the full existing test suite locally on JDK 21 (matching this repo's Jenkinsfile): 238 tests run, 0 failures, 0 errors, 1 skipped (pre-existing skip, unrelated to this change). Also verified ./mvnw spotless:check and ./mvnw spotbugs:check pass clean.

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests that demonstrate the feature works or the issue is fixed

…ffline

preOnline() called Future.get() with no timeout while waiting for a
build agent to acknowledge its OpenTelemetry SDK configuration. This
blocked SlaveComputer.setChannel() for however long the remote call
took, giving unrelated NAT idle timeouts or network disturbances a
much bigger window to close the channel before the online handshake
finished, especially for agents connecting over higher RTT links.

Bound the wait to a configurable timeout (default 10 seconds, matching
the existing timeout in afterConfiguration), and log a warning instead
of blocking indefinitely when it elapses. The agent is still allowed
online even if its OpenTelemetry configuration RPC has not completed.

Fixes jenkinsci#1285
@om7057
om7057 requested a review from a team as a code owner August 9, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

preOnline: channel.call(…).get() blocks indefinitely, kicks RTT-sensitive agents when channel closes

1 participant