fix: Add timeout to IMDS requests to prevent shutdown hang - #1010
Open
folouiseAWS wants to merge 2 commits into
Open
fix: Add timeout to IMDS requests to prevent shutdown hang#1010folouiseAWS wants to merge 2 commits into
folouiseAWS wants to merge 2 commits into
Conversation
The IMDSv2 token, spot instance-action, and ASG lifecycle-state requests had no timeout and only caught ConnectionError. On hosts where the IMDS address (169.254.169.254) is blackholed rather than refused, the token request in Worker.run() blocks the main thread indefinitely. The scheduler keeps running in the executor, so the agent appears healthy, but SIGTERM shutdown can never complete and the worker is never transitioned to STOPPED. Apply the same 0.5 second timeout already used by startup.bootstrap._get_metadata, and treat request timeouts the same as connection errors: IMDS is unavailable. Verified on a macOS host on a network that blackholes the IMDS address: without this change the agent hung indefinitely after SIGTERM; with it, the agent transitions to STOPPED and exits in under one second. Signed-off-by: Louise Fox <208544511+folouiseAWS@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was the problem/requirement?
The IMDSv2 token request in
Worker._get_ec2_metadata_imdsv2_token()(and the spot instance-action and ASG lifecycle-state requests) had no timeout and only caughtrequests.ConnectionError.On hosts where the IMDS address (169.254.169.254) is silently dropped rather than actively refused — possible on any non-EC2 host depending on network configuration — the token request in
Worker.run()blocks the main thread indefinitely. The scheduler keeps running in the executor, so the agent appears healthy and processes work, but SIGTERM shutdown can never complete: the worker is never transitioned toSTOPPEDand must be SIGKILLed, leaving the service-side worker resource inSTARTEDuntil it ages out toNOT_RESPONDING.What was the solution?
timeout=0.5to all three IMDS requests inworker.py, matching the timeout and rationale already used bystartup.bootstrap._get_metadata.requests.Timeoutthe same asrequests.ConnectionError: IMDS is unavailable.Worker._IMDS_REQUEST_TIMEOUT_SECONDSwith documentation.What was the testing done?
test/unit/test_worker.pymodule passes (36 tests).hatch run lint(ruff + mypy) passes.requests.putatworker.py:391) and the worker stayedSTARTED; with this change, the agent transitioned the worker toSTOPPEDand exited in under one second.