Skip to content

ci: relieve queue via macOS test de-dup and cancel-in-progress on all platforms - #5096

Merged
nanangizz merged 1 commit into
masterfrom
ci-mac-dedup
Jul 20, 2026
Merged

nanangizz merged 1 commit into
masterfrom
ci-mac-dedup

Conversation

@nanangizz

@nanangizz nanangizz commented Jul 20, 2026 •

Copy link
Copy Markdown
Member

The CI queue — especially macOS — is frequently very long: GitHub caps concurrent runners (org-wide, shared across all repos), and the macOS job matrix re-ran several suites that a given variant can't actually affect. A single flaky timing test (e.g. sleep_test) then failed a whole matrix of jobs at once and forced a full re-queue behind that long queue.

1. Cancel superseded runs — all platforms (macOS, Ubuntu, Windows)

Add a concurrency group with cancel-in-progress to ci-mac.yml, ci-linux.yml, ci-win.yml, so a force-push to a PR stops the superseded run from holding scarce runner slots (which return to the shared pool immediately).

2. macOS test de-dup

Principle: a variant job should run only the tests its variant actually changes. The full pjlib-test, pjlib-util-test and pjnath-test suites run once in the Default jobs; the OpenSSL/GnuTLS jobs keep only the parts that genuinely differ by SSL backend:

  • pjlib-test → Default runs the full suite (incl. the timing tests sleep/timer/timestamp). The OpenSSL/GnuTLS jobs run only ssl_sock_test + ssl_sock_stress_test (the backend-dependent parts), run directly as ioq-cb-with-lock already does.
  • pjnath-test → Default runs the full suite. The OpenSSL/GnuTLS jobs run only turn_sock_test, which covers TURN-over-TLS (turn_sock_test variant 2 → PJ_TURN_TP_TLS, exercising pj_ssl_sock_t) — the only SSL-backend-dependent surface in pjnath.
  • pjlib-util-test → dropped from the SSL jobs entirely; it has no SSL/https path (verified in http_client.c) and is codec-independent, so the single Default run suffices.
  • Openh264+VPX / pjlib,util,pjnath job ran only those three suites, all already covered by the Default jobs and unaffected by the video codec → disabled (commented out), freeing a runner slot.

Removed/narrowed macOS steps are commented out with the rationale inline, so they're trivial to restore. pjmedia-test, pjsip-test, pjsua-test runs are intentionally left unchanged (conservative scope).

The initial revision over-narrowed pjlib/pjnath; Copilot flagged the dropped ssl_sock_stress_test and TURN-over-TLS coverage, both verified and restored in the current revision. Thanks @copilot.

Effect

  • The flaky timing tests (sleep/timer/timestamp, which live in the full pjlib-test) go from 4 runs → 1 per macOS CI run — so one timing blip can no longer fail the whole matrix.
  • Removes most of ~2 h of redundant full-suite runner time per macOS CI run: pjnath-test (~17 min) and pjlib-test (~10 min) no longer run 3× each — the Default run stays, the two SSL jobs run only small backend-specific subsets, and the fully-redundant job is dropped.
  • One fewer macOS job (16 → 15). Under the slot-limited scheduler, less slot-occupancy = the queue drains faster on every platform.

Note on status checks (no action needed)

I checked the repo: master has no branch protection and no required status checks (the only ruleset, "Copilot review for default branch", is disabled). So disabling the Openh264+VPX / pjlib,util,pjnath job does not block any PR — nothing to update. Recording it only for the future: if required status checks are ever enabled, that job's check name should not be added to the required list (it no longer reports); all other check names are unchanged.

Related: the sleep_test tolerance fix is #5095 (merged) — complementary: that reduces the failure rate, this reduces the blast radius.

Co-Authored-By Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Reduces CI queue pressure by cancelling superseded runs and removing redundant macOS test executions.

Changes:

  • Adds concurrency cancellation to Linux, macOS, and Windows CI.
  • Narrows SSL-backend macOS jobs and disables one redundant codec job.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/ci-win.yml Cancels superseded Windows runs.
.github/workflows/ci-mac.yml Cancels superseded runs and deduplicates tests.
.github/workflows/ci-linux.yml Cancels superseded Linux runs.

Comment thread .github/workflows/ci-mac.yml Outdated
Comment thread .github/workflows/ci-mac.yml
… platforms

The CI queue (especially macOS) is frequently very long because GitHub
caps the number of concurrent runners (org-wide, shared across all repos),
and the macOS job matrix re-ran several test suites that a given job
variant cannot actually affect. A single flaky timing test (e.g.
sleep_test) therefore failed a whole matrix of jobs at once and forced a
full re-queue.

Concurrency (all platforms: macOS, Ubuntu, Windows):
- Add a concurrency group with cancel-in-progress so a force-push to a PR
  stops the superseded run from holding scarce runner slots.

macOS test de-dup (principle: a variant job should run only the tests its
variant changes):
- The full pjlib-test, pjlib-util-test and pjnath-test suites already run
  once in the Default jobs. In the OpenSSL and GnuTLS jobs we now run only
  the SSL-backend-dependent tests per backend: ssl_sock_test and
  ssl_sock_stress_test (from pjlib-test) and turn_sock_test (pjnath's
  TURN-over-TLS coverage), run directly as ioq-cb-with-lock already does.
  pjlib-util-test has no SSL dependency and is dropped from these jobs.
- The "Openh264+VPX / pjlib,util,pjnath" job ran only codec-independent
  suites already covered by the Default jobs and was entirely redundant;
  it is disabled (commented out) to free a macOS runner slot.

Removed/narrowed macOS steps are commented out (not deleted) with the
rationale inline, so they are easy to restore. pjmedia-test, pjsip-test
and pjsua-test runs are intentionally left unchanged.

This cuts the flaky timing tests (sleep/timer/timestamp) from 4 runs to 1
per macOS CI run while preserving per-backend ssl_sock and TURN-over-TLS
coverage.

Co-Authored-By Claude Code
@nanangizz

Copy link
Copy Markdown
Member Author

Good catches from @copilot — both verified against the source and now fixed in ec7ff17:

  1. ssl_sock_stress_test (pjlib/src/pjlib-test/test.c:351-352): confirmed INCLUDE_SSLSOCK_TEST registers both ssl_sock_test and ssl_sock_stress_test. The SSL jobs now run both, so backend-specific pending-I/O / renegotiation stress coverage is preserved.

  2. pjnath-test / TURN-over-TLS (turn_sock_test.c:544, variant n==2 → PJ_TURN_TP_TLS, gated by USE_TLS = PJ_HAS_SSL_SOCK): confirmed it exercises pj_ssl_sock_t, so pjnath-test is not SSL-backend-independent. Rather than dropping it, the SSL jobs now run turn_sock_test (which covers the UDP/TCP/TLS variants), keeping per-backend TURN/TLS coverage while still skipping the long backend-independent ICE/STUN bulk (that runs once in the Default job).

pjlib-util-test remains dropped from these jobs — verified http_client.c has no SSL/https path, so it is genuinely backend-independent.

@nanangizz
nanangizz merged commit 845befc into master Jul 20, 2026
57 checks passed
@nanangizz
nanangizz deleted the ci-mac-dedup branch July 20, 2026 13:53
weslleymurdock pushed a commit to weslleymurdock/pjproject that referenced this pull request Sep 9, 2026
… platforms (pjsip#5096)

The CI queue (especially macOS) is frequently very long because GitHub
caps the number of concurrent runners (org-wide, shared across all repos),
and the macOS job matrix re-ran several test suites that a given job
variant cannot actually affect. A single flaky timing test (e.g.
sleep_test) therefore failed a whole matrix of jobs at once and forced a
full re-queue.

Concurrency (all platforms: macOS, Ubuntu, Windows):
- Add a concurrency group with cancel-in-progress so a force-push to a PR
  stops the superseded run from holding scarce runner slots.

macOS test de-dup (principle: a variant job should run only the tests its
variant changes):
- The full pjlib-test, pjlib-util-test and pjnath-test suites already run
  once in the Default jobs. In the OpenSSL and GnuTLS jobs we now run only
  the SSL-backend-dependent tests per backend: ssl_sock_test and
  ssl_sock_stress_test (from pjlib-test) and turn_sock_test (pjnath's
  TURN-over-TLS coverage), run directly as ioq-cb-with-lock already does.
  pjlib-util-test has no SSL dependency and is dropped from these jobs.
- The "Openh264+VPX / pjlib,util,pjnath" job ran only codec-independent
  suites already covered by the Default jobs and was entirely redundant;
  it is disabled (commented out) to free a macOS runner slot.

Removed/narrowed macOS steps are commented out (not deleted) with the
rationale inline, so they are easy to restore. pjmedia-test, pjsip-test
and pjsua-test runs are intentionally left unchanged.

This cuts the flaky timing tests (sleep/timer/timestamp) from 4 runs to 1
per macOS CI run while preserving per-backend ssl_sock and TURN-over-TLS
coverage.

Co-Authored-By Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants