Skip to content

ci: don't treat "waiting" runs as active in the bot-CI priority gate - #12738

Open
szihs wants to merge 3 commits into
masterfrom
fix/ci-priority-waiting-deadlock
Open

ci: don't treat "waiting" runs as active in the bot-CI priority gate#12738
szihs wants to merge 3 commits into
masterfrom
fix/ci-priority-waiting-deadlock

Conversation

@szihs

@szihs szihs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Motivation

nv-slang-bot draft CI has been running zero tests and piling up in a waiting deadlock. Over the last 600 ci.yml runs (8 days), of 90 bot workflow_dispatch runs: 45 completed having run nothing, 29 are stuck in waiting having run nothing (29/29 verified: wait-for-human-priority failed and every build job skipped), and only 5 succeeded.

Root cause is an interaction between two mechanisms:

  • wait-for-priority.py (the gate, first job of every bot dispatch) yields the run when higher-priority CI is "active".
  • retry-yielded-bot-ci.py (the retry) only reruns a yielded bot run when no CI is "active".

Both use ACTIVE_STATUSES in extras/ci/ci_priority_common.py, which included "waiting".

In GitHub Actions, waiting means a job is parked on an environment protection rule, not that it is consuming a runner. The only environment gate in this repo is falcor-ci (a manual ci-approvers approval on the Falcor bridge test). Since #12614 added falcor-build-approval-gate (a job that needs: [filter] and carries environment: falcor-ci), every run now enters waiting almost immediately and stays there until a human approves.

With "waiting" counted as active, the result is a deadlock:

  • the priority gate always sees "CI busy" -> every bot dispatch yields and runs nothing;
  • the retry always sees "CI not quiet" -> yielded runs are never rerun.

Before #12614, environment: falcor-ci sat only on the leaf test-falcor job (#11915), so runs reached waiting late and briefly, and the hazard stayed latent.

Proposed solution

Drop "waiting" from ACTIVE_STATUSES. A run parked on the falcor-ci human approval is idle build-wise and must not count as runner contention. Runs that genuinely hold or await runners are queued / in_progress / requested / pending and remain counted; an approval-parked run flips to those states once approved and is counted then. test-falcor stays human-gated exactly as intended — this only changes how the priority scripts account for the waiting state.

Change summary

File Change
extras/ci/ci_priority_common.py Remove "waiting" from ACTIVE_STATUSES; add a comment explaining the falcor-ci reasoning and the single-environment assumption.

Blast radius

ACTIVE_STATUSES / fetch_active_runs are read in exactly two places, both intended:

  • wait-for-priority.py::classify_blockers — stops yielding to approval-parked runs.
  • retry-yielded-bot-ci.py::any_active_ci — lets the retry drain the backlog.

Not affected: ci-queue-status.py (uses its own independent "waiting" literal for reporting; does not import this module), the analytics suite and its tests, the anti-starvation aging logic, and all workflow YAML (no status literals passed).

Residual note captured in the code comment: this is correct because falcor-ci is the only environment. If a second environment gate is ever added whose pending state should count as contention, this must be revisited.

A run in GitHub Actions "waiting" status is parked on an environment
protection rule, not consuming a runner. The only environment gate in
this repo is falcor-ci (a manual ci-approvers approval on the Falcor
bridge test), so a "waiting" run is blocked on a human, not on runner
capacity.

Since #12614 the falcor-build-approval-gate job (needs: [filter]) makes
every run enter "waiting" on the falcor-ci approval almost immediately.
With "waiting" in ACTIVE_STATUSES, the priority gate then saw CI as
permanently busy so every nv-slang-bot dispatch yielded and ran zero
tests, and the retry saw CI as never quiet so yielded runs were never
rerun -- a deadlock (observed: 29/29 currently-waiting bot dispatch runs
had executed no builds).

Drop "waiting" from ACTIVE_STATUSES. Runs that truly hold or await
runners are queued/in_progress/requested/pending and remain counted;
approval-parked runs flip to those states once approved and are counted
then.
@szihs
szihs requested a review from a team as a code owner August 25, 2026 14:04
@szihs
szihs requested review from bmillsNV and removed request for a team August 25, 2026 14:04
@szihs szihs added the pr: non-breaking PRs without breaking changes label Aug 25, 2026
@jhelferty-nv
jhelferty-nv removed the request for review from bmillsNV August 25, 2026 14:04
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 3af5c49c-e1ae-4ec0-99f1-85cb0e85e2ce

📥 Commits

Reviewing files that changed from the base of the PR and between eb20c14 and de8b608.

📒 Files selected for processing (1)
  • extras/ci/ci_priority_common.py

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

The shared CI priority status set no longer treats GitHub Actions runs waiting for falcor-ci environment approval as active runner contention. Queued, in-progress, requested, and pending runs remain active.

Changes

CI status filtering

Layer / File(s) Summary
Active status contract
extras/ci/ci_priority_common.py
ACTIVE_STATUSES excludes waiting and retains queued, in_progress, requested, and pending statuses.

Suggested reviewers: bmillsnv

Merge Risk: ⚪ Minimal · up to de8b6

This localized change stops approval-parked CI runs from being treated as active contention; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description check ✅ Passed The description clearly explains the deadlock caused by counting GitHub Actions waiting runs as active and documents the proposed change to ACTIVE_STATUSES.
Title check ✅ Passed The title clearly and concisely describes the main change: excluding waiting runs from the bot-CI priority gate's active-run accounting.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@szihs
szihs deployed to falcor-ci August 25, 2026 14:10 — with GitHub Actions Active
@szihs
szihs deployed to falcor-ci August 27, 2026 07:48 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr: non-breaking PRs without breaking changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants