Skip to content

fix: Add timeout to IMDS requests to prevent shutdown hang - #1010

Open
folouiseAWS wants to merge 2 commits into
aws-deadline:mainlinefrom
folouiseAWS:fix/imds-token-request-timeout
Open

fix: Add timeout to IMDS requests to prevent shutdown hang#1010
folouiseAWS wants to merge 2 commits into
aws-deadline:mainlinefrom
folouiseAWS:fix/imds-token-request-timeout

Conversation

@folouiseAWS

@folouiseAWS folouiseAWS commented Jul 15, 2026

Copy link
Copy Markdown

What was the problem/requirement?

The IMDSv2 token request in Worker._get_ec2_metadata_imdsv2_token() (and the spot instance-action and ASG lifecycle-state requests) had no timeout and only caught requests.ConnectionError.

On hosts where the IMDS address (169.254.169.254) is silently dropped rather than actively refused — possible on any non-EC2 host depending on network configuration — the token request in Worker.run() blocks the main thread indefinitely. The scheduler keeps running in the executor, so the agent appears healthy and processes work, but SIGTERM shutdown can never complete: the worker is never transitioned to STOPPED and must be SIGKILLed, leaving the service-side worker resource in STARTED until it ages out to NOT_RESPONDING.

What was the solution?

  • Add timeout=0.5 to all three IMDS requests in worker.py, matching the timeout and rationale already used by startup.bootstrap._get_metadata.
  • Treat requests.Timeout the same as requests.ConnectionError: IMDS is unavailable.
  • Extracted as a class constant Worker._IMDS_REQUEST_TIMEOUT_SECONDS with documentation.

What was the testing done?

  • Unit tests added for timeout behavior of all three IMDS queries; existing call assertions updated. Full test/unit/test_worker.py module passes (36 tests).
  • hatch run lint (ruff + mypy) passes.
  • Reproduced live on an arm64 macOS host on a network where the IMDS address is silently dropped, running the agent against a real Deadline Cloud farm: without this change, the agent hung indefinitely after SIGTERM (thread dump showed the main thread blocked in requests.put at worker.py:391) and the worker stayed STARTED; with this change, the agent transitioned the worker to STOPPED and exited in under one second.

The IMDSv2 token, spot instance-action, and ASG lifecycle-state
requests had no timeout and only caught ConnectionError. On hosts
where the IMDS address (169.254.169.254) is blackholed rather than
refused, the token request in Worker.run() blocks the main thread
indefinitely. The scheduler keeps running in the executor, so the
agent appears healthy, but SIGTERM shutdown can never complete and
the worker is never transitioned to STOPPED.

Apply the same 0.5 second timeout already used by
startup.bootstrap._get_metadata, and treat request timeouts the same
as connection errors: IMDS is unavailable.

Verified on a macOS host on a network that blackholes the IMDS
address: without this change the agent hung indefinitely after
SIGTERM; with it, the agent transitions to STOPPED and exits in
under one second.

Signed-off-by: Louise Fox <208544511+folouiseAWS@users.noreply.github.com>
@folouiseAWS
folouiseAWS requested a review from a team as a code owner July 15, 2026 23:32
@github-actions github-actions Bot added the waiting-on-maintainers Waiting on the maintainers to review. label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-maintainers Waiting on the maintainers to review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant