ci: don't treat "waiting" runs as active in the bot-CI priority gate - #12738
ci: don't treat "waiting" runs as active in the bot-CI priority gate#12738szihs wants to merge 3 commits into
Conversation
A run in GitHub Actions "waiting" status is parked on an environment protection rule, not consuming a runner. The only environment gate in this repo is falcor-ci (a manual ci-approvers approval on the Falcor bridge test), so a "waiting" run is blocked on a human, not on runner capacity. Since #12614 the falcor-build-approval-gate job (needs: [filter]) makes every run enter "waiting" on the falcor-ci approval almost immediately. With "waiting" in ACTIVE_STATUSES, the priority gate then saw CI as permanently busy so every nv-slang-bot dispatch yielded and ran zero tests, and the retry saw CI as never quiet so yielded runs were never rerun -- a deadlock (observed: 29/29 currently-waiting bot dispatch runs had executed no builds). Drop "waiting" from ACTIVE_STATUSES. Runs that truly hold or await runners are queued/in_progress/requested/pending and remain counted; approval-parked runs flip to those states once approved and are counted then.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe shared CI priority status set no longer treats GitHub Actions runs waiting for ChangesCI status filtering
Suggested reviewers: Merge Risk: ⚪ Minimal · up to This localized change stops approval-parked CI runs from being treated as active contention; no actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Motivation
nv-slang-bot draft CI has been running zero tests and piling up in a
waitingdeadlock. Over the last 600ci.ymlruns (8 days), of 90 botworkflow_dispatchruns: 45 completed having run nothing, 29 are stuck inwaitinghaving run nothing (29/29 verified:wait-for-human-priorityfailed and every build job skipped), and only 5 succeeded.Root cause is an interaction between two mechanisms:
wait-for-priority.py(the gate, first job of every bot dispatch) yields the run when higher-priority CI is "active".retry-yielded-bot-ci.py(the retry) only reruns a yielded bot run when no CI is "active".Both use
ACTIVE_STATUSESinextras/ci/ci_priority_common.py, which included"waiting".In GitHub Actions,
waitingmeans a job is parked on an environment protection rule, not that it is consuming a runner. The only environment gate in this repo isfalcor-ci(a manualci-approversapproval on the Falcor bridge test). Since #12614 addedfalcor-build-approval-gate(a job thatneeds: [filter]and carriesenvironment: falcor-ci), every run now enterswaitingalmost immediately and stays there until a human approves.With
"waiting"counted as active, the result is a deadlock:Before #12614,
environment: falcor-cisat only on the leaftest-falcorjob (#11915), so runs reachedwaitinglate and briefly, and the hazard stayed latent.Proposed solution
Drop
"waiting"fromACTIVE_STATUSES. A run parked on the falcor-ci human approval is idle build-wise and must not count as runner contention. Runs that genuinely hold or await runners arequeued/in_progress/requested/pendingand remain counted; an approval-parked run flips to those states once approved and is counted then.test-falcorstays human-gated exactly as intended — this only changes how the priority scripts account for thewaitingstate.Change summary
extras/ci/ci_priority_common.py"waiting"fromACTIVE_STATUSES; add a comment explaining the falcor-ci reasoning and the single-environment assumption.Blast radius
ACTIVE_STATUSES/fetch_active_runsare read in exactly two places, both intended:wait-for-priority.py::classify_blockers— stops yielding to approval-parked runs.retry-yielded-bot-ci.py::any_active_ci— lets the retry drain the backlog.Not affected:
ci-queue-status.py(uses its own independent"waiting"literal for reporting; does not import this module), the analytics suite and its tests, the anti-starvation aging logic, and all workflow YAML (no status literals passed).Residual note captured in the code comment: this is correct because
falcor-ciis the only environment. If a second environment gate is ever added whose pending state should count as contention, this must be revisited.