Skip to content

Commit 1737b4d

Browse files
wshallwshallclaude
andcommitted
ci(ubuntu): give the pytest watchdog room to tell a hang from a slow run
The ubuntu leg's step timeout was 13 minutes. On 2026-07-31 the suite finished GREEN in 775s -- 9598 passed, 855 skipped -- against a 780s cap, and the step was killed anyway. The run is reported as a failure whose last line of output is a clean pass summary. A watchdog five seconds from firing on a healthy run is not distinguishing a deadlock from a slow day; it is a coin flip that reddens PRs at random and trains people to re-run without reading. That is worse than no watchdog, because a real hang now looks like the usual flake. Raises the ubuntu budget to job 22 / step 19, preserving the property the surrounding comment is built on: the STEP must expire before the JOB, so a missed hang still fails with step attribution rather than as a silent job-level kill. Verified on all three legs -- ubuntu 19<22, windows-2022 and windows-2025 unchanged at 26<30. Windows is deliberately untouched. It runs the same suite against a 26-min step cap, which is still roughly double the observed runtime; it has the headroom ubuntu lost. Not a fix for anything slow. The suite grew into a cap that was set when it was shorter, and the comment now says to re-check the margin as it grows rather than leaving the next person to rediscover this from a green-but-failed run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 9b03057 commit 1737b4d

1 file changed

Lines changed: 12 additions & 3 deletions

File tree

.github/workflows/ci.yml

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -212,8 +212,17 @@ jobs:
212212
# per-test dump is attributed first; this is the last-resort stack when the thread method is
213213
# itself out-raced). NB the plugin exposes this as the `faulthandler_timeout` ini key, not a
214214
# `--faulthandler-*` CLI flag, so it is passed with `-o`; it only DUMPS (never kills), safe.
215-
# The step `timeout-minutes: 13` is the outer backstop, well under the job's 15-min cap, so a
216-
# hang both watchdogs somehow miss still fails the STEP (not the whole job at the silent cap).
215+
# The step `timeout-minutes` (matrix `step_timeout`) is the outer backstop, held well under the
216+
# job's own cap, so a hang both watchdogs somehow miss still fails the STEP (not the whole job at
217+
# the silent cap). Keep the gap: the step must expire BEFORE the job, or the failure surfaces as
218+
# an uninformative job-level kill with no step attribution.
219+
#
220+
# It only works while the cap stays clear of a HEALTHY run. On 2026-07-31 the ubuntu leg finished
221+
# green in 775s against a 780s cap -- 5s of margin -- and the step was killed anyway, reported as
222+
# a failure on a run whose last line was "9598 passed, 855 skipped". A watchdog that cannot
223+
# separate "deadlocked" from "slow today" stops being a watchdog and becomes a coin flip, so the
224+
# ubuntu budget is raised here. Re-check the margin when the suite grows; the Windows legs are
225+
# unchanged because 26 min against the same suite is still ~2x headroom.
217226
- name: Tests (pytest)
218227
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
219228
timeout-minutes: ${{ matrix.step_timeout }}
@@ -371,7 +380,7 @@ jobs:
371380
# $GITHUB_REPOSITORY is a built-in runner env var, read here as plain shell (NOT a workflow-
372381
# expression interpolation into the run body), so it is zizmor-safe and cannot be misparsed as
373382
# an Actions expression the way a literal double-brace token in a run: block would be.
374-
U='{"os":"ubuntu-latest","python-version":"3.14","hosted":["ubuntu-latest"],"job_timeout":15,"step_timeout":13,"pytest_timeout":60,"fault_timeout":90}'
383+
U='{"os":"ubuntu-latest","python-version":"3.14","hosted":["ubuntu-latest"],"job_timeout":22,"step_timeout":19,"pytest_timeout":60,"fault_timeout":90}'
375384
W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":30,"step_timeout":26,"pytest_timeout":120,"fault_timeout":150}'
376385
W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":30,"step_timeout":26,"pytest_timeout":120,"fault_timeout":150}'
377386
if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then

0 commit comments

Comments
 (0)