Skip to content

Add failure-tolerant run mode (ContinueOnIterationFailure)#408

Draft
lilydoar wants to merge 1 commit into
mainfrom
lilydoar/continue-on-iteration-failure
Draft

Add failure-tolerant run mode (ContinueOnIterationFailure)#408
lilydoar wants to merge 1 commit into
mainfrom
lilydoar/continue-on-iteration-failure

Conversation

@lilydoar

Copy link
Copy Markdown
Contributor

What

Adds an opt-in, failure-tolerant run mode to the generic executor so a caller can let some iterations fail and judge the result itself, instead of the run aborting on the first terminal failure.

  • loadgenRunConfiguration.ContinueOnIterationFailure (default false, behavior unchanged when unset): when set, an iteration that terminally fails (after exhausting MaxIterationAttempts) no longer aborts the run — the executor keeps starting new iterations until the duration/iteration limit is reached.
  • loadgenRunConfiguration.OnIterationFailure hook: invoked when an iteration terminally fails, mirroring the existing OnCompletion hook. It is how a caller tallies failures. The executor also tracks succeeded/failed counts and reports them in the end-of-run log.
  • scenarios/throughput_stress: tallies terminal failures into a new FailedIterations field carried in the snapshot (alongside CompletedIterations), and skips the zero-failed-workflow verification (VerifyNoFailedWorkflows) when failures are tolerated — in that mode some workflow failures are expected, so that judgment belongs to the caller.

omes provides the mechanism only. The policy — how many failures are acceptable — lives with the caller. For Temporal Cloud release validation the caller reads failedIterations/completedIterations from the snapshot and applies its own threshold.

Why

Part of the bench-go → omes migration (RE-257). The release-validation load engine needs a failure-tolerance signal: "the load stayed up and fewer than X% of iterations failed" rather than the current all-or-nothing "any failure aborts the run." The completed-iteration count is already readable from the throughput_stress snapshot; this adds the failed-iteration count and the continue-on-failure behavior that make a tolerance verdict possible.

⚠️ Not yet consumable by saas-cicd (do not merge to unblock that work)

saas-cicd cannot bump omes today: omes@main requires go.temporal.io/api ≥ 1.62.12, whose workflowservice gencode registers the nexus-rpc proto annotations (ext 8233), which collide at init with the bergundy proto annotations still embedded in saas-infra-plane/api (all versions ≤ v1.80.0). Any binary linking both panics at startup. This PR is intentionally a draft until that conflict is resolved (saas-infra-plane regenerating against nexus-rpc/nexus-proto-annotations). The saas-cicd side that consumes this (relaxing the disabled throughput verdict, reading failedIterations) is a follow-up.

Testing

  • go test ./loadgen/ -run 'TestRun|TestExecutor' — all executor tests pass, including a new TestRunContinueOnIterationFailure (failed iterations are tolerated, the run completes, and both hooks fire for the right iterations).
  • go build ./..., go vet ./loadgen/... ./scenarios/... — clean (pre-existing unrelated vet notes only).
  • Default-path behavior is unchanged: with ContinueOnIterationFailure unset, a terminal failure aborts the run exactly as before (covered by existing TestRunFailIterations/TestRunFailDuration).
  • Note: TestKitchenSink (a devserver integration test) fails identically on clean main in this environment — unrelated to this change.

…runs

The generic executor aborts a run on the first iteration that terminally
fails (after exhausting MaxIterationAttempts). Add an opt-in mechanism for
callers that want to tolerate some failures and judge the result themselves.

- loadgen: RunConfiguration gains ContinueOnIterationFailure (keep starting
  iterations after a terminal failure instead of aborting) and an
  OnIterationFailure hook so callers can tally failures. The executor tallies
  succeeded/failed iterations and reports them in the end-of-run log. Default
  behavior is unchanged.
- throughput_stress: tally terminal failures into FailedIterations (carried in
  the snapshot alongside CompletedIterations) and skip the zero-failed-workflow
  verification when failures are tolerated, leaving that policy to the caller.

omes provides the mechanism only; the policy for how many failures are
acceptable lives with the caller (e.g. saas-cicd release validation).
@lilydoar lilydoar force-pushed the lilydoar/continue-on-iteration-failure branch from 44150c3 to 099d98a Compare June 22, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant