fix: bound preOnline timeout so RTT-sensitive agents are not kicked offline - #1294
Open
om7057 wants to merge 1 commit into
Open
fix: bound preOnline timeout so RTT-sensitive agents are not kicked offline#1294om7057 wants to merge 1 commit into
om7057 wants to merge 1 commit into
Conversation
…ffline preOnline() called Future.get() with no timeout while waiting for a build agent to acknowledge its OpenTelemetry SDK configuration. This blocked SlaveComputer.setChannel() for however long the remote call took, giving unrelated NAT idle timeouts or network disturbances a much bigger window to close the channel before the online handshake finished, especially for agents connecting over higher RTT links. Bound the wait to a configurable timeout (default 10 seconds, matching the existing timeout in afterConfiguration), and log a warning instead of blocking indefinitely when it elapses. The agent is still allowed online even if its OpenTelemetry configuration RPC has not completed. Fixes jenkinsci#1285
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1285
preOnline()waited on the agent's OpenTelemetry SDK configuration RPC with an unboundedFuture.get(), while the siblingafterConfiguration()method already bounds the same kind of call to 10 seconds. SincepreOnline()runs synchronously insideSlaveComputer.setChannel(), blocking there for an unbounded amount of time gives unrelated NAT idle timeouts or network disturbances a much bigger window to close the channel before the online handshake finishes. This mostly affects agents connecting over higher RTT links, where the configuration RPC naturally takes longer to complete.This change bounds the wait to a configurable timeout, defaulting to 10 seconds to match
afterConfiguration(), and logs a warning instead of blocking indefinitely when it elapses. The agent is still allowed online even if its OpenTelemetry configuration RPC has not completed in time.The timeout is configurable via
otel.instrumentation.jenkins.agent.pre_online.timeout(ConfigurationKey.OTEL_INSTRUMENTATION_JENKINS_AGENT_PRE_ONLINE_TIMEOUT), read the same way other OTel SDK properties are, throughConfigProperties.getDuration(...).Testing done
Added
OpenTelemetryConfigurerComputerListenerTestwith two tests:preOnlineReturnsPromptlyWhenAgentConfigurationRpcNeverCompletes: uses ahudson.remoting.Futurethat never completes to simulate a slow or RTT-bound channel, and assertspreOnline()returns well within a few seconds once the configured timeout elapses, instead of blocking indefinitely.preOnlineDoesNotPropagateExecutionExceptionFromAgent: asserts a failed configuration RPC does not propagate out ofpreOnline().Ran the full existing test suite locally on JDK 21 (matching this repo's Jenkinsfile): 238 tests run, 0 failures, 0 errors, 1 skipped (pre-existing skip, unrelated to this change). Also verified
./mvnw spotless:checkand./mvnw spotbugs:checkpass clean.Submitter checklist