Skip to content

Add worker versioning override storm scenario#394

Open
Shivs11 wants to merge 4 commits into
mainfrom
shivam/worker-versioning-current-override-storm
Open

Add worker versioning override storm scenario#394
Shivs11 wants to merge 4 commits into
mainfrom
shivam/worker-versioning-current-override-storm

Conversation

@Shivs11

@Shivs11 Shivs11 commented Jun 4, 2026

Copy link
Copy Markdown
Member

What

Adds an OMES scenario, worker_versioning_current_override_storm, for validating the following use-case:

  • If a user were to start a large number of workflows using VersioningOverride, does the signal reactivation path (which reactivates drained/inactive versions to draining OR skips this path when a version is 'active' (current/ramping)) work well.

The scenario:

  • waits for worker deployment versions v1 and v2 to register
  • sets v1 as current and waits for routing propagation to complete
  • starts many KitchenSink workflows with a pinned versioning override to v1
  • sets v2 as current and waits for routing propagation to complete (this failed one time for a customer)
  • verifies v1 reaches DRAINING
  • signals the pinned workflows so they complete

Why

This targets the customer-reported pattern from temporalio/temporal#9778: a large number of workflows started with a versioning override should not cause worker deployment internal workflows / routing updates to get stuck.

Local validation

  • go test ./scenarios
  • Source-built Temporal server from local temporal checkout, server version reported 1.32.0
  • Ran two Go workers on the same task queue with deployment build IDs v1 and v2
  • Ran scenario with workflow-count=100
  • Confirmed executor completed successfully
  • Confirmed final deployment describe had v2 current and v1 draining

This is a draft while we prepare the omes-deploy side for larger Cloud load runs.

@Shivs11 Shivs11 marked this pull request as ready for review June 4, 2026 19:42
@Shivs11 Shivs11 requested review from a team as code owners June 4, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant