Skip to content

Commit 11891cb

Browse files
committed
Poll to terminal instead of fixed sleeps in the last two flaky spots
Task-level diagnostics from a red run showed these are two different problems, not the one I assumed: scenario_decorated_workers: task SCHEDULED, pollCount=0, workerId=None after sleep(15), while both decorated workers were demonstrably active and polling every 100ms. The server simply had not handed the task out inside that window. This is the same false negative the batch-completion budget in this file was already raised to fix, so use that budget here and poll to terminal. test_v2_fallback_intg: 4 of a workflow's 5 tasks COMPLETED with the last IN_PROGRESS on a live worker (pollCount=1, workerId set) when the 60s budget expired -- it was progressing, not stuck. Raised to 120s. A task still IN_PROGRESS after that is a real lost update rather than slowness, and the diagnostic prints it.
1 parent 759dd28 commit 11891cb

2 files changed

Lines changed: 21 additions & 4 deletions

File tree

tests/integration/test_v2_fallback_intg.py

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -142,8 +142,13 @@ def _run_workers():
142142

143143
print(f"\n Submitted {len(workflow_ids)} workflows")
144144

145-
# Wait for completion
146-
deadline = time.time() + 60 # 60s timeout
145+
# Wait for completion. 60s was marginal: a red run showed 4 of a
146+
# workflow's 5 tasks COMPLETED and the last one still IN_PROGRESS on
147+
# a live worker, i.e. the run was progressing when the budget ran
148+
# out. Give the shared server the same headroom the other suites
149+
# use. A task still IN_PROGRESS after this is a genuine stuck
150+
# update, and the diagnostic below prints it.
151+
deadline = time.time() + 120
147152
pending = set(workflow_ids)
148153
completed = 0
149154
failed = 0

tests/integration/workflow/test_workflow_execution.py

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -156,8 +156,20 @@ def scenario_decorated_workers(
156156
start_wf_req = StartWorkflowRequest(name=workflow_name, task_to_domain=td_map)
157157
workflow_id_2 = workflow_executor.start_workflow(start_wf_req)
158158

159-
logger.debug(f'started TestPythonDecoratedWorkerWf with domain:cool and id: {workflow_id_2}')
160-
sleep(15)
159+
logger.info('started TestPythonDecoratedWorkerWf %s (no domain) and %s (domain:cool)',
160+
workflow_id, workflow_id_2)
161+
162+
# Poll to terminal instead of sleeping a fixed 15s. Both of these run a
163+
# single decorated-worker task, and on a loaded shared server the task can
164+
# sit SCHEDULED well past 15s before the server hands it to a poller --
165+
# observed as status=SCHEDULED pollCount=0 while the worker was demonstrably
166+
# alive and polling every 100ms. Same false-negative the batch-completion
167+
# budget above was raised to fix; use that budget here too.
168+
for wf_id in (workflow_id, workflow_id_2):
169+
wait_for_workflow_terminal(
170+
workflow_executor, wf_id,
171+
timeout_seconds=WORKFLOW_COMPLETION_MAX_WAIT_SECONDS,
172+
)
161173

162174
_run_with_retry_attempt(
163175
validate_workflow_status,

0 commit comments

Comments
 (0)