fix(lifecycle): publish the live run before announcing StatusRunning (#2806)

[devarismeroxa]

Fixes #2806. Tier 1 — data path, the engine that ships by default.

Process, as requested: plan → adversarial review of the plan (which overturned the recommendation) → implement → adversarial review of the commit (which returned merge with changes, all applied) → merge on green.

The bug

Between UpdateStatus(StatusRunning) and Start's runningPipelines.Set — which only happened after runPipeline returned — the map still pointed at the previous run. On a recovery restart the old entry is deliberately left in place until that swap, so the window is real and reachable. WaitPipeline joined the dead tomb and returned the pre-recovery error; StopAll then swallowed the resulting error at shutdown and let the runtime quiesce the persister and close the DB while the recovered run was still live. Invariant 7, silently.

The fix

Publication moves into runPipeline, immediately before the status write, with an explicit rollback on the error path, and every delete becomes a compare-and-delete serialized by a new publishMu.

What it deliberately does not do is reorder the cleanup-goroutine registration to match v2's shape. Registering cleanup before the status write lets a fast-failing pipeline's terminal status be stomped by the still-pending Running write, since UpdateStatus is last-writer-wins with no versioning. Today's ordering is load-bearing.

Two things the reviews caught that I had wrong

My recommended design was wrong. The plan review rejected it: reordering cleanup registration reintroduces the same bug class through the recovery chain, where the cleanup goroutine runs the nested restart synchronously and a transient status-write failure in the nested run unwinds into the outer cleanup's delete.

My failure-mode analysis was wrong. I wrote that Stop "reports success" against the dead run. It doesn't — SourceNode.Stop returns "source node is not running" deterministically, so StopAndWait surfaces an error and ApplyPlanLive does not proceed. The harm arrives via StopAll instead. Corrected here, in the commit messages, and in issue #2806 itself.

The commit review also caught that my compare-and-delete was leaky when it didn't need to be — demonstrated, not argued: 4 erasures of a live entry in 200,000 races, 0 with publishMu. And reachable without a recovery chain at all, since an operator Stop leaves the status UserStopped, which admits a concurrent Start.

Evidence

Verified against the exact pre-fix service.go from main, keeping the new tests:

Test	Pre-fix
Recovery_LiveEntryPublishedBeforeRunningStatus	FAIL (4.84s)
Recovery_StopDuringWindowTargetsLiveRun	FAIL (19.63s)
RunPipeline_UpdateStatusRunningFails_RollsBackPublication	FAIL once strengthened
StopAll_Recovering (binds an old run pre-failure by design)	still passes

Worth recording: my first perturbation attempt reported zero failures, which would have meant the tests were vacuous. It was my error — the edit didn't compile, so go test printed a build error and my grep '^--- FAIL' matched nothing. A build failure looked exactly like a passing perturbation. Redone with a git-exact revert.

Tests hold the window open via a PipelineService wrapper that blocks inside the status update rather than racing it — a pre-fix -count=60 sweep of the equivalent v2 package passed while that bug was live, so repeat runs are not evidence.

go test ./pkg/lifecycle/ -race -count=3 -shuffle=on green (65s), golangci-lint 0 issues.

Filed, deliberately not fixed here

• pipeline: UpdateStatus mutates in-memory status before the store write and never rolls it back #2809 — UpdateStatus mutates the shared in-memory status before the store write and never rolls it back. This is why the rollback test asserts only map state, not that a later Start succeeds.
• lifecycle: the recovery-backoff window holds a dead run for up to 10 minutes while Stop admits StatusRecovering #2810 — the recovery-backoff window holds a dead run for 1s–10m while Stop admits StatusRecovering, and a stop during backoff doesn't prevent the restart. Far larger than the window this PR closes, entirely pre-existing.
• lifecycle-poc (arch-v2): cleanup still deletes runningPipelines by key, so a superseded run can erase a live one #2811 — lifecycle-poc still deletes by key in its recovery-failed arm: the delete-side half of this bug in v2.

Follow-ups, not merge gates: StopAll holds the map's read lock across stopGraceful, and named coverage gaps around StopAll/reconfigure during the window.