Skip to content

test: Remove timing dependencies from worker tests - #1674

Merged
Kampfmoehre merged 1 commit into
mainfrom
fix/flaky-worker-tests
Aug 28, 2026
Merged

test: Remove timing dependencies from worker tests#1674
Kampfmoehre merged 1 commit into
mainfrom
fix/flaky-worker-tests

Conversation

@Kampfmoehre

Copy link
Copy Markdown
Member

Follow-up to #1673. Fixes the flaky test that failed on #1667, plus three more with the same defect found while verifying.

Note

The coverage job on this PR will stay red until #1673 is merged — that's the unrelated coveralls timezone crash, not this change.

The flake from CI

AddFailedProgressAsync_ShouldAddFailedProgres let the process function sleep 100 ms, then called AddFailedProgressAsync after StartAsync returned:

sut.SetProcessFunction(async (_j, ct) => { await Task.Delay(100, ct); return null; });
await sut.StartAsync(ct);
await sut.CallAddFailedProgressAsync(7);

StartAsync returns as soon as ExecuteAsync suspends — which is inside that Task.Delay(100). If the test thread isn't scheduled again within 100 ms, HandleJobRunAsync finishes and sets _jobRepository = null (JobWorkerBase.cs:360), so the call throws:

System.InvalidOperationException : Unable to set job items because no job repository is set.

Reproduced deterministically by inserting a 300 ms delay after StartAsync — byte-identical to the CI failure.

The two sibling tests (SetTotalItemsAsync_ShouldSetTotalItems, AddProgressAsync_ShouldAddProgress) had already been converted to a TaskCompletionSource handshake. This one was missed; it now uses the same pattern.

Three more with the same defect

The AddInitialJob tests waited a fixed Task.Delay(10)"Give some time to add job" — before asserting. They now wait on a new TestWorker.JobRunCompleted, which completes when the worker's first iteration has actually finished.

This also removes a false negative: AddInitialJob_ShouldNotAddInitialJob_IfThereIsAnyInDueRange asserts AddJobAsync was never called, which previously could pass simply because the worker hadn't reached that code yet. Waiting for the iteration to complete makes the assertion meaningful.

I used the PostJobRunHookAsync seam rather than a callback on the AddJobAsync mock: a mock callback can't work for the negative test (you can't wait for a call that must never happen), and one uniform mechanism beats two.

Un-skipped test

SetTotalItemsAsync_ShouldThrowInvalidOperationException_WhenNoCurrentJobIsSet was marked:

Skip = "Find a way to setup state for this without repo being nulled after a run"

It needs _jobRepository != null && _currentJob == null. That window exists: DeleteOldJobs runs after _currentJob is reset but before _jobRepository is nulled. Blocking inside the DeleteJobsAsync mock holds the worker there deterministically.

Verified it isn't passing vacuously — with the wait removed it fails on the other branch:

Expected: ···"to set job items because no current job exists."
Actual:   ···"to set job items because no job repository is set."

Evidence

All soaks on 16 cores with 2× busy-loop load:

Before After
AddFailedProgressAsync_ShouldAddFailedProgres 0/15 pass 15/15 pass
AddInitialJob_ShouldAddAnInitialJob_IfConfigured 2/6 pass
Whole JobService.Test project 20/20 pass

Full solution green 3× (17 + 17 + 19 passed, 0 skipped — was 1 skipped).

No production code changed; TestWorker gains one test-only seam.

🤖 Generated with Claude Code

https://claude.ai/code/session_015BX2NY9ozmWKHsifii1KYP

Four tests in JobWorkerBaseTest raced the worker instead of synchronising with
it, which made them fail on a loaded CI runner.

AddFailedProgressAsync_ShouldAddFailedProgres let the process function sleep
100ms and called AddFailedProgressAsync afterwards. If the test thread was not
scheduled within that window, HandleJobRunAsync had already completed and set
_jobRepository to null, so the call threw:

  System.InvalidOperationException : Unable to set job items because no job
  repository is set.

Under 2x CPU load this test failed 15 out of 15 runs. Its two siblings
(SetTotalItemsAsync/AddProgressAsync) were already converted to a
TaskCompletionSource handshake; this applies the same pattern to the one that
was missed.

The three AddInitialJob tests waited a fixed 10ms ("Give some time to add job")
before asserting. AddInitialJob_ShouldAddAnInitialJob_IfConfigured failed 4 out
of 6 runs under load. They now wait on TestWorker.JobRunCompleted, which
completes when the worker's first iteration is actually finished. That also
removes a false-negative risk in the ShouldNotAddInitialJob case, which could
previously pass simply because the worker had not got there yet.

Un-skips SetTotalItemsAsync_ShouldThrowInvalidOperationException_WhenNoCurrent
JobIsSet, which was marked "Find a way to setup state for this without repo
being nulled after a run". Deleting old jobs runs while the repository is still
set but the current job has been reset, so blocking inside DeleteJobsAsync holds
the worker in exactly that state. Verified by mutation: without the wait the
test fails on the "no job repository is set" message instead.

After the change the whole test project passed 20 out of 20 runs under 2x load.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015BX2NY9ozmWKHsifii1KYP
@Kampfmoehre
Kampfmoehre merged commit 97102f7 into main Aug 28, 2026
3 of 4 checks passed
@Kampfmoehre
Kampfmoehre deleted the fix/flaky-worker-tests branch August 28, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant