feat(array): --min-viable cost guard + MPI declaration tags (#52) - #67
Merged
Conversation
`--min-viable N` states the count below which a job array is not worth
running. That is a different thing from `onFailure`, which decides whether
to keep *launching* and leaves already-launched members running — so a
100-member array reaching 2-of-100 was, under `onFailure` alone, two
instances billing indefinitely for a job that cannot be done.
Three parts, matching what Go's cohort.Reconciler does and why ("Drain
surviving instances so nothing idles and bills", cohort/reconcile.go:298):
- fast-fail: once the threshold is unreachable, unstarted members are
skipped rather than launched (cohort's fastFailCancel, reconcile.go:243).
- drain: JobArray.enforceViability() terminates the survivors, called by
SpawnClient.pumpFanOuts on every tick so it needs no caller action. Each
survivor is reported exactly once even across overlapping pumps; a failed
terminate stays retryable and emits a warning rather than being swallowed.
- report: FanOutSummary gains minViable/viableCandidates/nonViable/
missingIndexes. The last is Go's sparse-index view — "97 of 100 running"
hides which three slices have no worker.
The gate is a monotone latch, so applyGating's fixpoint loop still
terminates, and `nonViable: false` means "not yet ruled out", never
"confirmed viable". Out-of-range values clamp to [1, size] as Go clamps
them; a *malformed* value is rejected instead, because Number("hlaf") would
land on the no-op 1 and silently disable the guard.
Enforcement lives in JobArray rather than FanOut: FanOut is shared with
sweeps and queues and every other state change it makes is a launch.
The dashboard card states the threshold, and on going non-viable shows the
shortfall in red with what follows, its chip reading "non-viable" instead
of the green "done" — an array that was torn down must not look like one
that succeeded. minviable.harness.html reaches that state (it needs capacity
failures, which no UI control can produce).
MPI is tags only: --mpi / --mpi-processes-per-node stamp spawn:mpi-* on
every member so the array is recognisable as MPI by the Go CLI and the
portal. That boundary is deliberate — Go's pkg/mpicohort is a self-declared
spike whose unresolved problem is that cohort's Placement is per-entity
while a placement group and an EFA fabric are collective constraints.
Porting a spike would commit spawn-ts to a shape Go is still deciding. EFA
validation needs the launch region (truffle-ts#33) and
--auto-placement-group creates a real AWS resource. Absence stays absence:
no mpi-enabled=false is written, and status never prints "mpi: no".
Docs: docs/execution-shapes.md, incl. an out-of-reach table for
logs/collect/retry --failed quoting Go's own reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #52. Last substantive item in the
v0.7.0 — Go parity (documented divergences)milestone.--min-viable— a threshold on the set, not a policy per memberonFailuredecides whether to keep launching, and"stop"leaves the members that already came up running. So underonFailurealone, a 100-member array that reaches 2-of-100 is two instances billing indefinitely for a job that cannot be done.--min-viable Nstates the count below which the whole array is pointless, and spawn-ts then does what Go'scohort.Reconcilerdoes, for the reason Go itself gives — "Drain surviving instances so nothing idles and bills" (cohort/reconcile.go:298):fastFailCancel,reconcile.go:243). Without it,--min-viable 50on an array that lost 51 members would go on launching the other 49.JobArray.enforceViability()terminates the survivors, andSpawnClient.pumpFanOutscalls it every tick, so the wind-down needs no caller action. A survivor that could not be terminated emits a warning rather than being swallowed: that is precisely the case where money keeps being spent.FanOutSummarygainsminViable/viableCandidates/nonViable/missingIndexes.The gate is a monotone latch (skipping raises the lost count, which keeps it non-viable), which is what lets
applyGating's fixpoint loop still terminate.nonViable: falsemeans "not yet ruled out", never "confirmed viable" — counting only running members would call a healthy array doomed on its first pump, and the caller's response to non-viability is termination.completedcounts toward viability for the same reason: otherwise a fully successful array turns non-viable as it drains.Out-of-range values clamp to
[1, size]as Go clamps them (cmd/launch_jobarray.go:576-582) —--min-viable 200on a 100-member array is an obvious "all of them". A malformed value is rejected instead:Number("hlaf")isNaNand would land on the no-op1, silently disabling the guard the user explicitly asked for.Enforcement lives in
JobArray, notFanOut.FanOutis shared with sweeps and queues, every other state change it makes is a launch, and a shared engine that silently terminated instances would surprise its other two callers.--min-viableis also a job-array concept — a sweep has no viability threshold.Sparse indexes
A threshold alone trades one wrong answer for another: "97 of 100 running" hides which three slices have no worker.
missingIndexesfollows Go's rule (cmd/arraygroup.go:100,:228), whose live set isrunning/pendingonly — so a terminated member's index counts as missing too.MPI: tags only, deliberately
--mpi/--mpi-processes-per-node Nstampspawn:mpi-enabled/spawn:mpi-processes-per-nodeon every member, decoded ontoManagedInstance.mpiand shown instatus, so a spawn-ts-launched array is recognisable as MPI by the Go CLI and the portal.The boundary is a decision, not an unfinished port. Go's
pkg/mpicohortis a self-declared spike whose header states the unresolved problem: cohort'sPlacementis per-entity while a placement group and an EFA fabric are collective constraints. Porting a spike would commit spawn-ts to a shape Go is still deciding. EFA validation must run in the launch region (blocked on truffle-ts#33) and--auto-placement-groupcreates a real AWS resource.Absence stays absence: no
mpi-enabled=falseis ever written,decodeMpiTagsreturnsundefinedrather than{enabled: false}, andstatusnever prints "mpi: no".Two defects found by driving the app, not by the tests
Both passed the suite before being found, so each has a test now.
Duplicate terminate events. The drain emitted two
terminateevents for one instance.startJobArraykicks a pump without awaiting it, so twoenforceViability()calls were in flight at once. A mark-on-success guard still duplicated — both calls cleared it before eitherterminateresolved — so the claim is taken before the await and released on failure, keeping a throttled instance retryable. One instance, one event: a duplicate reads as two instances wound down.The dashboard never mentioned
--min-viableat all, and showed a green "done" chip on an array whose survivors had just been terminated. A user watched running members disappear with no reason given, and a torn-down array looked like one that succeeded. The card now states the threshold, shows the shortfall in red with what follows, names the missing indexes, and reads non-viable.Verification
npm run typecheckclean;npm test591 passing / 30 files (586 before);npm run buildsucceeds (one pre-existing unrelated TypeDoc warning).dist/boundary: the happy path, clamp reporting (--min-viable 200→3 of 3 (adjusted from 200)), both malformed-value rejections (hlafand a valueless flag), the--mpi-processes-per-nodewithout--mpiguard, and thestatusMPI line. The dashboard in real Chromium with screenshots inspected, including the non-viable card.minviable.harness.html(dev-only, following the existingexpiry.harness.htmlconvention) reaches the non-viable state by making two members unlaunchable. No control in the UI can produce a capacity failure, which is what makes that state the one most likely to rot unnoticed. It builds the realDashboardover a realSpawnClientand stubs onlyMockProvider.launch.Docs
New
docs/execution-shapes.md: the three shapes, what--min-viableguarantees (latch / fast-fail / drain, clamping vs. rejection), sparse indexes, "MPI: tags, deliberately" quoting the mpicohort spike header, "Absence is not a negative claim", and an out-of-reach table forlogs/collect/retry --failed— quoting Go's own reason for the last (a local launch record under~/.config/spore/arrays/that "must run from the machine that launched the array").