Skip to content

feat(array): --min-viable cost guard + MPI declaration tags (#52) - #67

Merged
scttfrdmn merged 1 commit into
mainfrom
feat/min-viable-mpi-tags
Aug 1, 2026
Merged

feat(array): --min-viable cost guard + MPI declaration tags (#52)#67
scttfrdmn merged 1 commit into
mainfrom
feat/min-viable-mpi-tags

Conversation

@scttfrdmn

Copy link
Copy Markdown
Contributor

Closes #52. Last substantive item in the v0.7.0 — Go parity (documented divergences) milestone.

--min-viable — a threshold on the set, not a policy per member

onFailure decides whether to keep launching, and "stop" leaves the members that already came up running. So under onFailure alone, a 100-member array that reaches 2-of-100 is two instances billing indefinitely for a job that cannot be done. --min-viable N states the count below which the whole array is pointless, and spawn-ts then does what Go's cohort.Reconciler does, for the reason Go itself gives — "Drain surviving instances so nothing idles and bills" (cohort/reconcile.go:298):

  • fast-fail — the moment the threshold becomes unreachable, unstarted members are skipped rather than launched (cohort's fastFailCancel, reconcile.go:243). Without it, --min-viable 50 on an array that lost 51 members would go on launching the other 49.
  • drainJobArray.enforceViability() terminates the survivors, and SpawnClient.pumpFanOuts calls it every tick, so the wind-down needs no caller action. A survivor that could not be terminated emits a warning rather than being swallowed: that is precisely the case where money keeps being spent.
  • reportFanOutSummary gains minViable / viableCandidates / nonViable / missingIndexes.

The gate is a monotone latch (skipping raises the lost count, which keeps it non-viable), which is what lets applyGating's fixpoint loop still terminate. nonViable: false means "not yet ruled out", never "confirmed viable" — counting only running members would call a healthy array doomed on its first pump, and the caller's response to non-viability is termination. completed counts toward viability for the same reason: otherwise a fully successful array turns non-viable as it drains.

Out-of-range values clamp to [1, size] as Go clamps them (cmd/launch_jobarray.go:576-582) — --min-viable 200 on a 100-member array is an obvious "all of them". A malformed value is rejected instead: Number("hlaf") is NaN and would land on the no-op 1, silently disabling the guard the user explicitly asked for.

Enforcement lives in JobArray, not FanOut. FanOut is shared with sweeps and queues, every other state change it makes is a launch, and a shared engine that silently terminated instances would surprise its other two callers. --min-viable is also a job-array concept — a sweep has no viability threshold.

Sparse indexes

A threshold alone trades one wrong answer for another: "97 of 100 running" hides which three slices have no worker. missingIndexes follows Go's rule (cmd/arraygroup.go:100, :228), whose live set is running/pending only — so a terminated member's index counts as missing too.

MPI: tags only, deliberately

--mpi / --mpi-processes-per-node N stamp spawn:mpi-enabled / spawn:mpi-processes-per-node on every member, decoded onto ManagedInstance.mpi and shown in status, so a spawn-ts-launched array is recognisable as MPI by the Go CLI and the portal.

The boundary is a decision, not an unfinished port. Go's pkg/mpicohort is a self-declared spike whose header states the unresolved problem: cohort's Placement is per-entity while a placement group and an EFA fabric are collective constraints. Porting a spike would commit spawn-ts to a shape Go is still deciding. EFA validation must run in the launch region (blocked on truffle-ts#33) and --auto-placement-group creates a real AWS resource.

Absence stays absence: no mpi-enabled=false is ever written, decodeMpiTags returns undefined rather than {enabled: false}, and status never prints "mpi: no".

Two defects found by driving the app, not by the tests

Both passed the suite before being found, so each has a test now.

  1. Duplicate terminate events. The drain emitted two terminate events for one instance. startJobArray kicks a pump without awaiting it, so two enforceViability() calls were in flight at once. A mark-on-success guard still duplicated — both calls cleared it before either terminate resolved — so the claim is taken before the await and released on failure, keeping a throttled instance retryable. One instance, one event: a duplicate reads as two instances wound down.

  2. The dashboard never mentioned --min-viable at all, and showed a green "done" chip on an array whose survivors had just been terminated. A user watched running members disappear with no reason given, and a torn-down array looked like one that succeeded. The card now states the threshold, shows the shortfall in red with what follows, names the missing indexes, and reads non-viable.

Verification

  • npm run typecheck clean; npm test 591 passing / 30 files (586 before); npm run build succeeds (one pre-existing unrelated TypeDoc warning).
  • 20 targeted mutations, all caught — including mark-on-success vs mark-before-await, dropping the dedupe check, never releasing a failed claim, and each of the three new dashboard lines.
  • Driven, not just tested. The CLI in a tmux REPL over the built dist/ boundary: the happy path, clamp reporting (--min-viable 2003 of 3 (adjusted from 200)), both malformed-value rejections (hlaf and a valueless flag), the --mpi-processes-per-node without --mpi guard, and the status MPI line. The dashboard in real Chromium with screenshots inspected, including the non-viable card.
  • New minviable.harness.html (dev-only, following the existing expiry.harness.html convention) reaches the non-viable state by making two members unlaunchable. No control in the UI can produce a capacity failure, which is what makes that state the one most likely to rot unnoticed. It builds the real Dashboard over a real SpawnClient and stubs only MockProvider.launch.

Docs

New docs/execution-shapes.md: the three shapes, what --min-viable guarantees (latch / fast-fail / drain, clamping vs. rejection), sparse indexes, "MPI: tags, deliberately" quoting the mpicohort spike header, "Absence is not a negative claim", and an out-of-reach table for logs / collect / retry --failed — quoting Go's own reason for the last (a local launch record under ~/.config/spore/arrays/ that "must run from the machine that launched the array").

`--min-viable N` states the count below which a job array is not worth
running. That is a different thing from `onFailure`, which decides whether
to keep *launching* and leaves already-launched members running — so a
100-member array reaching 2-of-100 was, under `onFailure` alone, two
instances billing indefinitely for a job that cannot be done.

Three parts, matching what Go's cohort.Reconciler does and why ("Drain
surviving instances so nothing idles and bills", cohort/reconcile.go:298):

- fast-fail: once the threshold is unreachable, unstarted members are
  skipped rather than launched (cohort's fastFailCancel, reconcile.go:243).
- drain: JobArray.enforceViability() terminates the survivors, called by
  SpawnClient.pumpFanOuts on every tick so it needs no caller action. Each
  survivor is reported exactly once even across overlapping pumps; a failed
  terminate stays retryable and emits a warning rather than being swallowed.
- report: FanOutSummary gains minViable/viableCandidates/nonViable/
  missingIndexes. The last is Go's sparse-index view — "97 of 100 running"
  hides which three slices have no worker.

The gate is a monotone latch, so applyGating's fixpoint loop still
terminates, and `nonViable: false` means "not yet ruled out", never
"confirmed viable". Out-of-range values clamp to [1, size] as Go clamps
them; a *malformed* value is rejected instead, because Number("hlaf") would
land on the no-op 1 and silently disable the guard.

Enforcement lives in JobArray rather than FanOut: FanOut is shared with
sweeps and queues and every other state change it makes is a launch.

The dashboard card states the threshold, and on going non-viable shows the
shortfall in red with what follows, its chip reading "non-viable" instead
of the green "done" — an array that was torn down must not look like one
that succeeded. minviable.harness.html reaches that state (it needs capacity
failures, which no UI control can produce).

MPI is tags only: --mpi / --mpi-processes-per-node stamp spawn:mpi-* on
every member so the array is recognisable as MPI by the Go CLI and the
portal. That boundary is deliberate — Go's pkg/mpicohort is a self-declared
spike whose unresolved problem is that cohort's Placement is per-entity
while a placement group and an EFA fabric are collective constraints.
Porting a spike would commit spawn-ts to a shape Go is still deciding. EFA
validation needs the launch region (truffle-ts#33) and
--auto-placement-group creates a real AWS resource. Absence stays absence:
no mpi-enabled=false is written, and status never prints "mpi: no".

Docs: docs/execution-shapes.md, incl. an out-of-reach table for
logs/collect/retry --failed quoting Go's own reason.
@scttfrdmn
scttfrdmn merged commit 767ea5b into main Aug 1, 2026
1 check passed
@scttfrdmn
scttfrdmn deleted the feat/min-viable-mpi-tags branch August 1, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Execution shapes: job-array --min-viable is missing, MPI is tag-only (and MPI orchestration should not be chased)

1 participant