[DO-rrao]

Issue

• Temporary SSH key POST /v2/account/keys succeeds, then POST /v2/droplets immediately uses the numeric key ID and returns 422 … are invalid key identifiers for Droplet creation.
• Cleanup DELETE /v2/account/keys/{id} then 404s; the key later exists and is left behind (replica lag between key write and droplet-create validation). HTTPRetryMax only retries 429/5xx, not 422.
• After the droplet is active, a single GET can still have empty networks.v4 (IPv4 address not found for droplet). Polling GET /v2/account/keys/{id} is not a valid consistency check: Keys GET can succeed while droplet create still 422s.

Solution

• Retry Droplets.Create only on 422 invalid key identifiers with exponential backoff (1s → cap 15s), overall deadline state_timeout (default 6m). Other 422s fail immediately.
• Retry Keys.DeleteByID on 404 with the same backoff so lag does not orphan keys. Store keyId as soon as create succeeds so cleanup still runs if a later step fails.
• After active, poll until the expected public/private IPv4 exists (retry empty networks and 404). Do not poll Keys GET or use a blind 60s sleep.

Reproduction

• Packer DigitalOcean build that imports a temporary SSH key (do not set ssh_key_id / ssh_private_key_file).
• Immediately after key create, droplet create fails 422 … are invalid key identifiers; key delete 404s.
• Timing-dependent; sequential creates often succeed, concurrent CI-like creates reproduce it.

Testing

• go test ./builder/digitalocean/ unit tests for backoff cap, retryable vs permanent errors, 422/404 classifiers, and IPv4 selection.
• Prod api.digitalocean.com: 40 sequential key→droplet creates did not 422 (lag is load-dependent).
• Prod: 8 parallel creates in tor1/sfo3/nyc3 all 422'd on first droplet create; backoff recovered all 8 (reproduced_422=8, recovered_with_backoff=8). Several also retried empty IPv4 after active. All droplets/keys deleted.

CI pins macos-14 instead of macos-latest because GitHub’s latest macOS is 15, and Go 1.21/1.22 -race binaries lack LC_UUID so dyld rejects them.
The required check test (1.21.x, macos-latest) will stay pending on this PR until branch protection is updated to macos-14 (or the pin is reverted).

Test plan

• go test ./builder/digitalocean/
• Optional prod: RUN_PROD_API_TEST=1 DIGITALOCEAN_TOKEN=$(doctl auth token) PROD_API_ATTEMPTS=64 PROD_API_PARALLEL=8 go test ./builder/digitalocean -run TestProdSSHKeyLagAgainstAPI -v -timeout 25m
• Packer build with a generated temp SSH key (no ssh_key_id) against a real account

Fixes ESC-26331

Made with Cursor

[anup-deka]

LGTM