bug(init): `init --up` leaks db instances out of state.json on an EC2 eventual-consistency race

[rustyrazorblade]

Problem: init --up aborted when the db group's status-check wait hit a transient InvalidInstanceID.NotFound (EC2 eventual consistency), and the 3 created db instances were left out of state.json — alive in AWS but invisible to status and down (orphaned spend).

Root cause: EC2InstanceService.waitForInstanceStatusOk() uses a raw Ec2Waiter with no RetryUtil.withEc2InstanceRetry wrapper (unlike the already-hardened describeInstances()), and partially-created instances aren't persisted.

Fix: retry the status-check waiter through InvalidInstanceID.NotFound; persist partially-created instances so down/status can reclaim them.

Found during the TiDB×sysbench load-test investigation (2026-07-21). See the run's final report for full context.

[rustyrazorblade]

Deprioritized to P3 after design review.

Key finding that lowers the severity: down reclaims orphaned instances by VPC, not from state.json — vpcId is persisted before any instance is created, and teardown discovers instances via findInstancesInVpc(vpcId). So there is no real billing leak: down already terminates the orphaned instances. The only genuinely-affected command is status, which lists nodes purely from state.hosts. Net: this is a status-accuracy / truthful-state.json / defense-in-depth issue, not a cost risk.

Design captured for if/when revisited:

• Part 1 (reliability): route EC2InstanceService.waitForInstanceStatusOk() through the existing RetryUtil.withEc2InstanceRetry (its predicate already matches InvalidInstanceID.NotFound via the "does not exist" message; optionally add an explicit error-code match). Wrap the waiter only — never create+wait, or it re-issues RunInstances and creates duplicates.
• Part 2 (state truthfulness): persist created instances early via the existing onHostsCreated callback (keeps EC2InstanceService free of state concerns), switching the merge to upsert-by-instanceId (also fixes a latent re-run up duplicate-append).