fix: stamp Go's base-identity tags, or refuse to launch (#51) - #60
Merged
Conversation
A spawn-ts launch was visible in `spawn list` yet invisible AND unterminatable in the portal. The portal filters its instance list, single-instance lookup and terminate on `spawn:iam-user` (lambda/dashboard-api/instances.go:60/:168/:285 — the last 403s on a mismatch); `spawn list` filters on `spawn:managed` alone. So the divergence only surfaced when someone tried to clean up, and by then the instance had been billing the whole time. buildLaunchTags now takes an optional `LaunchIdentity` and stamps the block Go writes: root, created-by, version, account-id, account-base36, iam-user, account-name (slugified, #121), plus `os` and `local-username`. Identity arrives as data so the tag builder stays pure; EC2Provider resolves it once via GetCallerIdentity and caches it, or takes it from the caller — the federated BYOA path already has the ARN and account id back from AssumeRoleWithWebIdentity. EC2Provider **refuses to launch** when it can't resolve the identity, including the 200-with-empty-fields case a bare try/catch would sail past. Omitting the tag and launching anyway produces an orphaned billable instance nobody can terminate from the portal — strictly worse than a failed launch, and the #63 invariant in its most expensive form: the error must not look like an absence of identity. `local-username` is resolved once and used for both the tag and user-data. They must agree or `spawn connect` SSHes to a user that doesn't exist (cmd/connect.go:135 prefers the tag, falls back to ec2-user), so a custom username with no tag silently sent it to ec2-user. Also adds `spawn:active-ports`, which spored consumes as the sibling of active-processes in one expression (pkg/agent/agent.go:402). Absent, a port-based activity signal was silently unenforceable: a box serving a live RStudio connection on 8787 with no matching process name idled out underneath the user. Fixes a tag-limit bug found while measuring the above, present in Go too. AWS caps a resource at 50 tags and exceeding it fails RunInstances outright rather than truncating. Go allows a flat 35 spawn:param:* tags with a comment claiming that stays "under AWS 50-tag limit" (pkg/aws/tags.go:247); the arithmetic doesn't hold — a maximal sweep member reached 73, and 64 before this change, so the bug predates the identity block and this would have worsened it. The cap is now a budget computed against what the launch actually consumed (49 on a maximal launch, one slot reserved for local-username), and surplus params drop in sorted key order so the surviving subset is deterministic. docs/integration.md's "wire-compatible" claim now names what is *not* stamped, tiered like the rest of that section: the fsx/efs/dcv/app-name keys as D, `spawn:command` and `slack-workspace-id` as B, `job-array-created` as a deliberate E. It also records the two tags spawn-ts writes that Go's launcher doesn't — `compute-seconds` (Go lets spored create it) and `idle-cpu`, which Go *decodes* (pkg/provider/ec2.go:508) but never writes, making a Go-launched instance's --idle-cpu threshold unreachable by the reader that wants it. Tests: 26 files, 395 passed. tags.test.ts's old "caps at 35" test asserted the bug, so it now asserts the invariant instead, and a new maximal-launch test is the one that would actually have caught it. Part of #57.
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #51. Part of #57.
The bug
A spawn-ts-launched instance was visible in
spawn listyet invisible and unterminatable in the portal.The portal filters on
spawn:iam-userin three places — the instance list (lambda/dashboard-api/instances.go:60), single-instance lookup (:168), and terminate (:285, which 403s on a mismatch).spawn listfilters onspawn:managedalone. So the divergence only surfaced when someone tried to clean up, and by then the instance had been billing the whole time.spawn cleanup --only-mineskips it too (pkg/aws/cleanup.go:93).The fix
buildLaunchTagstakes an optionalLaunchIdentityand stamps Go's base-identity block:root,created-by,version,account-id,account-base36,iam-user,account-name(slugified, #121), plusosandlocal-username.Identity arrives as data so the tag builder stays pure.
EC2Providerresolves it once viaGetCallerIdentityand caches it for the provider's lifetime, or accepts it from the caller — the federated BYOA path already hasAssumedRoleUser.Arnand the account id back fromAssumeRoleWithWebIdentity, so no extra call there.It refuses to launch rather than degrade
Including the sneakier
200-with-empty-fields case, which a baretry/catchsails past and stampsundefined. Omitting the tag and launching anyway produces an orphaned billable instance nobody can terminate from the portal — strictly worse than a failed launch, and the #63 invariant in its most expensive form: the error must not look like an absence of identity. Both refusal paths assert nothing was launched, i.e. it fails beforeRunInstances, not after.local-usernameis resolved once, for both the tag and user-dataThey must agree or
spawn connectSSHes to a user that doesn't exist — Go prefers the tag and falls back toec2-user(cmd/connect.go:135), so a custom username with no tag silently sent it toec2-user. (Caught by a test: the first cut snapshottedtagListbefore the mutation.)spawn:active-portsspored consumes it as the sibling of
active-processesin one expression —countActiveSessions() + countActivePortConnections(config.ActivePorts)(pkg/agent/agent.go:402). Absent, a port-based activity signal was silently unenforceable: a box serving a live RStudio connection on 8787 with no matching process name idled out underneath the user.Side-discovery: the tag budget was over AWS's hard limit
Found by measuring, not by reading. AWS caps a resource at 50 tags and exceeding it fails
RunInstancesoutright — it doesn't truncate.Go allows a flat 35
spawn:param:*tags with a comment claiming that stays "under AWS 50-tag limit" (pkg/aws/tags.go:247). The arithmetic doesn't hold — the sweep block is 4 tags and a configured launch carries ~30 more:local-username)So the bug predates this change and the identity block would have worsened it. Rather than swap in a smaller guessed number, the cap is now a budget against what the launch actually consumed, so it stays correct as tags are added. Surplus params drop in sorted key order, so the surviving subset is deterministic rather than dependent on object insertion order. Dropping parameters is itself lossy — they're how a sweep member records which point in the space it is — but a truncated tag set beats a launch that fails.
The old
tags.test.tstest asserted the flat 35, i.e. it asserted the bug; it now asserts the invariant. The new end-to-end maximal-launch test is the one that would actually have caught it.docs/integration.md
#51 asks for this explicitly: the "wire-compatible" claim now names what is not stamped, in the same tier language as the rest of that section.
buildTagscan write 54 distinct keys;buildLaunchTagswrites 43.fsx-*(7),efs-id,efs-mount-point,dcv-session-id,app-namespawn:commandlen <= 256guard orRunInstancesfails (spawn#214/#246)slack-workspace-idjob-array-createdlaunch-timecovers itIt also records the two tags spawn-ts writes that Go's launcher doesn't:
spawn:compute-seconds— seeded at0; Go lets spored create it (pkg/agent/agent.go:404), andpkg/provider/ec2.go:486reads it either way.spawn:idle-cpu— Go decodes it (pkg/provider/ec2.go:508) but never writes it at launch, so a Go-launched instance's--idle-cputhreshold is unreachable by the reader that wants it. spawn-ts writing it is the fix, not the divergence.spawn:created-byis deliberately"spawn-ts", not"spawn"— no reader compares the value, and an operator benefits from knowing which launcher produced an instance.Verification
No AWS calls; the STS paths are stubbed via
STSClient.prototype.send.