Add failure-tolerant run mode (ContinueOnIterationFailure)#408
Draft
lilydoar wants to merge 1 commit into
Draft
Conversation
…runs The generic executor aborts a run on the first iteration that terminally fails (after exhausting MaxIterationAttempts). Add an opt-in mechanism for callers that want to tolerate some failures and judge the result themselves. - loadgen: RunConfiguration gains ContinueOnIterationFailure (keep starting iterations after a terminal failure instead of aborting) and an OnIterationFailure hook so callers can tally failures. The executor tallies succeeded/failed iterations and reports them in the end-of-run log. Default behavior is unchanged. - throughput_stress: tally terminal failures into FailedIterations (carried in the snapshot alongside CompletedIterations) and skip the zero-failed-workflow verification when failures are tolerated, leaving that policy to the caller. omes provides the mechanism only; the policy for how many failures are acceptable lives with the caller (e.g. saas-cicd release validation).
44150c3 to
099d98a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in, failure-tolerant run mode to the generic executor so a caller can let some iterations fail and judge the result itself, instead of the run aborting on the first terminal failure.
loadgen—RunConfiguration.ContinueOnIterationFailure(defaultfalse, behavior unchanged when unset): when set, an iteration that terminally fails (after exhaustingMaxIterationAttempts) no longer aborts the run — the executor keeps starting new iterations until the duration/iteration limit is reached.loadgen—RunConfiguration.OnIterationFailurehook: invoked when an iteration terminally fails, mirroring the existingOnCompletionhook. It is how a caller tallies failures. The executor also tracks succeeded/failed counts and reports them in the end-of-run log.scenarios/throughput_stress: tallies terminal failures into a newFailedIterationsfield carried in the snapshot (alongsideCompletedIterations), and skips the zero-failed-workflow verification (VerifyNoFailedWorkflows) when failures are tolerated — in that mode some workflow failures are expected, so that judgment belongs to the caller.omes provides the mechanism only. The policy — how many failures are acceptable — lives with the caller. For Temporal Cloud release validation the caller reads
failedIterations/completedIterationsfrom the snapshot and applies its own threshold.Why
Part of the bench-go → omes migration (RE-257). The release-validation load engine needs a failure-tolerance signal: "the load stayed up and fewer than X% of iterations failed" rather than the current all-or-nothing "any failure aborts the run." The completed-iteration count is already readable from the throughput_stress snapshot; this adds the failed-iteration count and the continue-on-failure behavior that make a tolerance verdict possible.
saas-cicd cannot bump omes today: omes@main requires
go.temporal.io/api ≥ 1.62.12, whoseworkflowservicegencode registers the nexus-rpc proto annotations (ext 8233), which collide at init with the bergundy proto annotations still embedded insaas-infra-plane/api(all versions ≤ v1.80.0). Any binary linking both panics at startup. This PR is intentionally a draft until that conflict is resolved (saas-infra-plane regenerating againstnexus-rpc/nexus-proto-annotations). The saas-cicd side that consumes this (relaxing the disabled throughput verdict, readingfailedIterations) is a follow-up.Testing
go test ./loadgen/ -run 'TestRun|TestExecutor'— all executor tests pass, including a newTestRunContinueOnIterationFailure(failed iterations are tolerated, the run completes, and both hooks fire for the right iterations).go build ./...,go vet ./loadgen/... ./scenarios/...— clean (pre-existing unrelated vet notes only).ContinueOnIterationFailureunset, a terminal failure aborts the run exactly as before (covered by existingTestRunFailIterations/TestRunFailDuration).TestKitchenSink(a devserver integration test) fails identically on cleanmainin this environment — unrelated to this change.