Skip to content

ec2 plugin: add retries / more resilient fetch for individual IMDS metadata keys #1946

Description

@gmarciani

Describe the Enhancement

In function metadata_get retry the retrieval of metadata that returns http code != 200.
At every retry the IMDS v2 token should be refreshed to make sure the token expiration is not the source of failures.

Describe the Need

The ec2 Ohai plugin's metadata fetch is best-effort per key: if a single IMDS GET for a given metadata path returns anything other than HTTP 200, the plugin either sets the attribute to nil (for http code 404) or aborts (non-404/non-200 http codes). There is no retry, no backoff, and no per-key resilience.

This works most of the time, but when IMDS occasionally returns transient errors, individual keys (e.g. local-hostname, local-ipv4, network interface sub-attributes) can end up nil with impact on the cookbook that assumes those attribute to get populated.

In our case, we've seen intermittent failures on EC2 instances where node['ec2']['local_hostname'] ends up nil, causing downstream recipes to fail.

Current Alternative

Replace Ohai entirely, in favor of custom logic just to make this retrieval more robust.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Status: UntriagedAn issue that has yet to be triaged.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions