test(lifecycle-poc): wait for the live status on the recovery path, don't instant-read it - #2832
Open
devarismeroxa wants to merge 1 commit into
Open
test(lifecycle-poc): wait for the live status on the recovery path, don't instant-read it#2832devarismeroxa wants to merge 1 commit into
devarismeroxa wants to merge 1 commit into
Conversation
…on't instant-read it TestServiceLifecycle_Recovery_TransientErrorRecovers failed on a DOCS-ONLY push to main (run 32785888309, sha 7486372; the next docs commit c073543 passed). The failure was: service_test.go:778: Running != Recovering panic.go:615: persister did not drain within 10s (see #2746) controller.go:97: missing call(s) to *mock.SourcePlugin.Teardown controller.go:97: missing call(s) to *mock.DestinationPlugin.Teardown x2 This is a test-synchronization defect, not a recovery bug. Evidence: at the failing assertion the recorded status sequence is exactly [Running Recovering Running] - ONE recovery, not two - while the live instance still reads Recovering. Both dispensers are .Times(2) and neither reported an over-call, and the three missing Teardowns are precisely the recovered run's own plugins, i.e. run 2 was alive and healthy when the assertion aborted the test. Everything after line 778 (the persister drain panic, the "run didn't finish" lines, the missing Teardowns) is fallout from is.Equal calling FailNow while the pipeline was still running. Root cause: neither watermark the test waits on is ordered after the status write it then asserts. - statusRecorder.UpdateStatus appends the status and runs its hook BEFORE delegating to pipeline.Service, which is what calls Instance.SetStatus. So waitForRecovered's "Running after a Recovering" fires strictly before pl.GetStatus() returns Running, with a whole UpdateStatus call (Get, two metrics updates, store write) still to go. - runPipeline closes `registered` - releasing every worker goroutine - and only then calls UpdateStatus(StatusRunning). The recovered run can therefore read, write and ack its entire record set while that status write is still in flight, so waitForRecordsAcked returns immediately and adds no delay either. Unlike the initial start (ls.Start returns only after runPipeline's status write completes, which is why the same instant assertion is sound there and in pkg/lifecycle), a recovery restart runs on the tomb's cleanup goroutine: nothing in the test goroutine is synchronized with it. The assertion was passing on the ~microsecond margin between the recorder entry and SetStatus, which a 4-core CI runner under `-race -shuffle=on` with docker sidecars can lose to a single goroutine preemption. Fix (test-only, no production change): waitForRecovered now also polls the live pl.GetStatus() until it reports Running, so it means what its name says. A genuine second recovery is still caught - the helper does not latch past it, and the AC-7 exact-sequence assertion at the end of each test would fail on [Running Recovering Running Recovering ...]. Four call sites carried the same unsound pattern, all fixed: - TestServiceLifecycle_Recovery_TransientErrorRecovers - TestServiceLifecycle_NSource_TransientErrorOneSource_Recovers - TestServiceLifecycle_NxM_TransientErrorOneSource_RecoversAllSourcesAndDestinations - TestServiceLifecycle_Recovery_LiveEntryPublishedBeforeRunningStatus The last one has no waitForRecovered call - it freezes the window with the onUpdate hook - and is the MOST exposed of the four, because close(release) only lets the status write proceed while the records it then waits on were already acked during the freeze. It gets waitForStatus instead. It was added by 34bf97d ("...and one async assertion behind three flaky tests"), which shipped a new assertion of the same class it was fixing; waitForRecovered itself dates to a61d4bc (#2718). This is polling a watermark, not a widened timeout, a retry or a sleep: the deadline (5s, unchanged from the existing helpers) only bounds the failure message. Perturbation proof. The flake could NOT be reproduced naturally on an M-series 16-core box: 3840 runs of the race-enabled binary (24x40 at -cpu=1,4 and 64x30 at GOMAXPROCS=1) were all green, so the CI signature was reproduced by injecting the scheduler preemption directly - a sleep in statusRecorder between recording the status and delegating to the wrapped service. With 200ms injected, across all four tests at -race -count=10: before fix: 40/40 FAIL, byte-identical to the CI output after fix: 40/40 PASS At 20ms the same 80-run matrix went 46/80 FAIL -> 0/80 PASS. Package green at `go test ./pkg/lifecycle-poc/... -race -count=3 -shuffle=on`; golangci-lint clean. The injection is not committed; it is reproducible by setting rec.onUpdate to sleep on the nth==2 StatusRunning, the seam that already exists for TestServiceLifecycle_Recovery_LiveEntryPublishedBeforeRunningStatus. Tier 3 (test-only). No existing issue tracks this test; #2534 is the umbrella flaky-suite issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016xE861dwb3MLgqEdWRECLY
4 tasks
This was referenced Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier 3 (test files only — no production code touched).
Fixes the
TestServiceLifecycle_Recovery_TransientErrorRecoversfailure seen onmainatrun 32785888309
(
service_test.go:778: Running != Recovering). The commit that failed was docs-only and the nextcommit passed, so this is a flake, not a regression.
Root cause
Neither watermark the test waits on is ordered after the status write it then asserts.
statusRecorder.UpdateStatusappends the status and runs its hook before delegating topipeline.Service, which is what actually callsInstance.SetStatus. SowaitForRecovered's"Running after a Recovering" fires strictly before
pl.GetStatus()can returnRunning, with awhole
UpdateStatus(aGet, two metrics updates, a store write) still in flight.waitForRecordsAckedadds no delay either:runPipelinedoesclose(registered)— releasing everyworker goroutine — before it calls
UpdateStatus(StatusRunning), so the recovered run can read,write and ack its whole record set while the status write is still pending.
The identical instant assertion is sound on the initial start (and in
pkg/lifecycle) becausels.Startreturns only afterrunPipeline's status write completes. A recovery restart runs onthe tomb's cleanup goroutine, and nothing in the test goroutine is synchronized with it. The
assertion was passing on a ~microsecond margin that a 4-core CI runner under
-race -shuffle=onloses to a single goroutine preemption.
Ruled out: this is not a second recovery. Both dispensers are
.Times(2)with no over-callreported (a second recovery needs a third dispense), and the three missing
Teardowns are therecovered run's own plugins — run 2 was alive and healthy when
is.EqualcalledFailNow.Everything after line 778 is cascade.
Fix
waitForRecoverednow also polls the livepl.GetStatus()until it reportsRunning, so the helpermeans what its name says. A genuine second recovery is still caught: the helper does not latch past
it, and the AC-7 exact-sequence assertion would fail on
[Running Recovering Running Recovering …].Four sites carried the same unsound pattern; all are fixed. The fourth
(
TestServiceLifecycle_Recovery_LiveEntryPublishedBeforeRunningStatus) has nowaitForRecoveredcall and was the most exposed, because
close(release)only lets the status write proceed while therecords it then waits on were already acked during the freeze.
Perturbation proof
Could not reproduce naturally on a 16-core M-series box: 3840 runs of the race binary
(24x40 at
-cpu=1,4; 64x30 at GOMAXPROCS=1) all green. So the preemption was injected directly — asleep in
statusRecorderbetween recording the status and delegating (injection not committed):-race -count=10Package green at
-race -count=3 -shuffle=on;golangci-lintreports 0 issues.Relationship to prior fixes
Not a recurrence of the #2746/#2806 production bugs — those were about
runningPipelinesholdinga dead run, and 4ec2b5f is intact. It is the same class as the async assertion fixed in 34bf97d,
and that commit is where the fourth site was introduced: the PR that fixed one async assertion
shipped a new one of the same kind.
waitForRecovereditself dates to a61d4bc (#2718).Not fixed here
A Tier-1 production ordering finding surfaced during this work and is filed separately rather than
folded in — see the linked issue. It needs its own PR with a deterministic regression test.
Roadmap: v0.20 WS9-B (flaky tests fixed at cause, never masked).