Skip to content

CI priority jobs dispatch - #327

Open
varun-jaiswal17 wants to merge 6 commits into
opencv:mainfrom
varun-jaiswal17:ci-intel-priority-dispatch
Open

CI priority jobs dispatch#327
varun-jaiswal17 wants to merge 6 commits into
opencv:mainfrom
varun-jaiswal17:ci-intel-priority-dispatch

Conversation

@varun-jaiswal17

@varun-jaiswal17 varun-jaiswal17 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Score-driven CI priority dispatch for 5.x PRs

opencv/opencv's PR-5.x.yaml calls one reusable workflow per platform. Each gated reusable workflow gets a lightweight priority-gate job (on ubuntu-latest); the heavy self-hosted jobs needs: it:

PR run ─▶ priority-gate (cheap hosted runner)
             1. score this (branch, platform) from history
             2. enrol in a per-pool queue on the ci-priority-data branch
             3. block until: running_in_pool < max-concurrent
                             AND this run is the top-scored waiter
             4. release  (fail-open after a timeout — never blocks a build)
          ─▶ heavy build jobs (self-hosted)   ← only grab a scarce runner once released
          ─▶ release-priority-slot            ← frees the slot AND records the run outcome

The scarce runner is never held while waiting — only the cheap gate job waits.

  • Scoring is keyed on (head_branch, platform) — opencv PRs are fork pull_request runs with no PR number. Factors: failed_on_prev_run (×10), flake_penalty (×2), short_job_first (×1), branch_stability (×1). A cancelled run is not treated as a failure.
  • Queue is partitioned by pool = runner label and keyed on (run_id, pool) — one PR:5.x run fans out to many platform jobs that share a run_id, and platforms sharing self-hosted runners must contend in one queue.
  • Storage is zero-infrastructure: NDJSON on a dedicated ci-priority-data branch (clone → append → push-retry).
  • A committed 90-day seed (ci_priority/seed/runs.ndjson) makes initial scoring base; the store then self-updates from each run's release job (no separate ingest).

Safety

  • Fail-open everywhere: a missing/read-only token, a git error, or a timeout releases the build unordered — it never blocks or skips a build.
  • Inert by default: with the PRIORITY_GATE_RUNNER variable unset, every gate runs on GitHub-hosted ubuntu-latest and fails open, so CI behaves exactly as today. Merging is a no-op until it's enabled — safe to review in the wild first. No opencv/opencv change is required.

Current state & follow-up

This PR runs the gate on GitHub-hosted ubuntu-latest — no physical/self-hosted runner. As shipped:

  • Merging is a safe no-op (fail-open everywhere; CI behaves exactly as today).
  • It orders runs whose token can write the store — i.e. trusted runs.

(The workflows carry a PRIORITY_GATE_RUNNER variable hook that defaults to ubuntu-latest; it's unused for now and exists only so the follow-up needs no code change.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant