Unify quarantine and outerloop CI with the main test pipeline by radical · Pull Request #16448 · microsoft/aspire

radical · 2026-04-24T22:33:31Z

Description

Migrates the specialized test runner (quarantine/outerloop) from the old MSBuild-based runsheet generation to the new metadata-driven pipeline, and extracts a shared run-tests-core.yml workflow to eliminate duplicated test job definitions.

Key changes

New run-tests-core.yml — reusable workflow containing the 6 dependency-based test bucket jobs + results gate. Shared by both tests.yml and specialized-test-runner.yml to avoid duplicating job definitions.
Refactored specialized-test-runner.yml — now uses the same pipeline as tests.yml: TestEnumerationRunsheetBuilder → build-test-matrix → expand-test-matrix-github → split-test-matrix-by-deps → run-tests-core. Added grep-based project scoping and positive trait filtering (DiscoveryTraitFilter) to build a lean matrix of only relevant test classes.
Simplified tests.yml — replaced 6 inline test job definitions with a single call to run-tests-core.yml.
Updated tests-outerloop.yml / tests-quarantine.yml — added run-tests-core.yml to paths: triggers so workflow changes are validated on PRs.
Deleted old MSBuild targets — removed SpecializedTestRunsheetBuilderBase.targets, OuterloopTestRunsheetBuilder.targets, QuarantinedTestRunsheetBuilder.targets, and cleared the old runsheet generation from AfterSolutionBuild.targets.
split-test-projects-for-ci.ps1 — added IncludeTraitFilter parameter for positive trait filtering in class discovery mode. When set and zero classes are found, the script gracefully skips instead of erroring.
tests/Directory.Build.targets — passes DiscoveryTraitFilter through to the split script as -IncludeTraitFilter.

Tests added

IncludeTraitFilterGracefullySkipsWhenNoClassesFound — verifies graceful skip with empty output JSON
WithoutIncludeTraitFilterFailsWhenNoClassesFound — verifies error behavior is preserved without the filter

Checklist

Is this feature complete?
- Yes. Ready to ship.
- No. Follow-up changes expected.
Are you including unit tests for the changes and scenario tests if relevant?
- Yes
- No
Did you add public API?
- Yes
- No
Does the change make any security assumptions or guarantees?
- Yes
- No
Does the change require an update in our Aspire docs?
- Yes
- No

github-actions · 2026-04-24T22:33:42Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16448

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16448"

github-actions · 2026-04-25T01:05:26Z

Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

Tests / run_tests / Hosting-1 / Hosting-1 (windows-latest) - Job-level runner or infrastructure failure matched the transient allowlist.

Copilot

Pull request overview

This PR migrates the quarantine/outerloop “specialized test runner” off the legacy MSBuild runsheet generation and onto the same metadata-driven test matrix pipeline used by the main CI, while extracting a shared reusable workflow to eliminate duplicated test-bucket job definitions.

Changes:

Added a new reusable workflow (run-tests-core.yml) that executes the 6 dependency-based test buckets plus a results gate.
Refactored specialized-test-runner.yml to generate/split matrices via the canonical pipeline and then call run-tests-core.yml (including a new positive trait discovery mode for class splitting).
Simplified tests.yml to call run-tests-core.yml, and removed legacy MSBuild runsheet targets/logic.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/Infrastructure.Tests/PowerShellScripts/SplitTestProjectsTests.cs	Adds tests for new `IncludeTraitFilter` “graceful skip” behavior.
tests/Directory.Build.targets	Plumbs `DiscoveryTraitFilter` through to the split script as `-IncludeTraitFilter`.
eng/scripts/split-test-projects-for-ci.ps1	Adds `IncludeTraitFilter` support and “skip instead of fail” behavior when no matching classes are found.
eng/SpecializedTestRunsheetBuilderBase.targets	Deletes legacy specialized runsheet generator base targets.
eng/QuarantinedTestRunsheetBuilder/QuarantinedTestRunsheetBuilder.targets	Deletes quarantined runsheet builder targets.
eng/OuterloopTestRunsheetBuilder/OuterloopTestRunsheetBuilder.targets	Deletes outerloop runsheet builder targets.
eng/AfterSolutionBuild.targets	Removes legacy combined runsheet generation target (keeps canonical matrix generation).
.github/workflows/tests.yml	Replaces 6 inline bucket jobs with a single call to `run-tests-core.yml`.
.github/workflows/tests-quarantine.yml	Updates `paths:` triggers and switches to new specialized runner inputs.
.github/workflows/tests-outerloop.yml	Updates `paths:` triggers and switches to new specialized runner inputs.
.github/workflows/specialized-test-runner.yml	Rebuilds specialized test execution around canonical matrices + `run-tests-core.yml`.
.github/workflows/run-tests.yml	Ensures `ignoreTestFailures` exits 0 reliably and always verifies TRX presence when enabled.
.github/workflows/run-tests-core.yml	New shared workflow implementing the 6 bucket jobs + results gate.

Comments suppressed due to low confidence (1)

eng/scripts/split-test-projects-for-ci.ps1:50

The script header still states it "Fails fast if zero test classes discovered when in class mode", but with IncludeTraitFilter set the script now intentionally succeeds and writes an empty partitions file. Update the .NOTES (and/or description) to reflect the conditional behavior so callers understand when zero classes is an error vs a skip.

.NOTES
  PowerShell 7+
  Fails fast if ExtractTestPartitions cannot be built or run.
  Fails fast if zero test classes discovered when in class mode.
  Only runs --list-tests when no partitions are found in the assembly.

JamesNK

Review: 2 issues flagged (1 correctness, 1 CI infrastructure). Both relate to the unification trade-offs — one about cross-OS build failure isolation regression, one about the auto-rerun script not accounting for the new results-gate job name.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

github-actions · 2026-07-09T02:38:04Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

github-actions · 2026-07-09T22:39:41Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.

github-actions · 2026-07-10T04:19:05Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 3 comments.

github-actions · 2026-07-10T19:16:13Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

github-actions · 2026-07-10T20:56:08Z

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Move the repeated six-bucket test job layout into a reusable run-tests-core workflow so the regular and specialized test pipelines can share the same execution shape. Keep the lower-level run-tests workflow responsible for per-project test execution details, including preserving ignore-test-failure result verification and avoiding cmd.exe parsing for MTP filter arguments on Windows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b15291c7-3448-460b-b9c5-33d6c5d3f668

Replace the quarantined and outerloop bespoke runsheet builders with the same metadata-driven test matrix pipeline used by the main test workflow. The specialized runner now scopes restore/build to projects containing the requested attribute, generates the canonical matrix, splits it into dependency buckets, and delegates execution to the shared run-tests-core workflow. Carry the specialized trait into matrix entries so downstream class and partition shards keep running only the requested quarantined or outerloop tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b15291c7-3448-460b-b9c5-33d6c5d3f668

Add and update infrastructure tests for the shared workflow wiring, specialized project discovery, trigger mapping, and matrix splitting. These tests assert that quarantined and outerloop runs keep the positive specialized trait filters when class or partition shards are generated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b15291c7-3448-460b-b9c5-33d6c5d3f668

Update the CI docs and PR testing guidance to describe the shared test workflow, the specialized quarantined/outerloop matrix path, and the MTP argument handling that workflow changes must preserve. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b15291c7-3448-460b-b9c5-33d6c5d3f668

Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

+      extraTestArgs: "${{ inputs.extraTestArgs }}"
      ignoreTestFailures: ${{ inputs.ignoreTestFailures }}


radical changed the title ~~Migrate specialized test runners to metadata-driven pipeline and extract run-tests-core.yml~~ Unify quarantine and outerloop CI with the main test pipeline Apr 25, 2026

github-actions Bot mentioned this pull request Apr 25, 2026

[repo-pulse] 📊 Repo Pulse — Daily Activity Dashboard #16404

Open

radical marked this pull request as ready for review April 26, 2026 00:48

Copilot AI review requested due to automatic review settings April 26, 2026 00:48

Copilot started reviewing on behalf of radical April 26, 2026 00:50 View session

Copilot AI reviewed Apr 26, 2026

View reviewed changes

Comment thread .github/workflows/specialized-test-runner.yml

Comment thread .github/workflows/specialized-test-runner.yml Outdated

github-actions Bot mentioned this pull request Apr 26, 2026

[pull requests] Review required #13834

Closed

This comment was marked as duplicate.

Sign in to view

radical requested review from JamesNK, davidfowl, eerhardt, joperezr and mitchdenny April 26, 2026 06:38

JamesNK reviewed Apr 27, 2026

View reviewed changes

Comment thread .github/workflows/tests.yml Outdated

Comment thread .github/workflows/run-tests-core.yml

radical mentioned this pull request Jun 9, 2026

Tracking: CI improvements #18036

Open

15 tasks

radical marked this pull request as draft June 9, 2026 21:10

Copilot AI review requested due to automatic review settings July 9, 2026 02:09

Copilot started reviewing on behalf of radical July 9, 2026 02:10 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread eng/scripts/split-test-projects-for-ci.ps1

Copilot AI review requested due to automatic review settings July 9, 2026 02:58

Copilot started reviewing on behalf of radical July 9, 2026 02:59 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread .github/workflows/run-tests-core.yml

Copilot AI review requested due to automatic review settings July 9, 2026 18:53

Copilot started reviewing on behalf of radical July 9, 2026 18:56 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 21:06

Copilot started reviewing on behalf of radical July 9, 2026 21:06 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread eng/scripts/generate-specialized-test-projects-list.sh Outdated

Comment thread .github/workflows/specialized-test-runner.yml

Comment thread docs/ci/TestingOnCI.md Outdated

Copilot AI review requested due to automatic review settings July 9, 2026 22:03

Copilot started reviewing on behalf of radical July 9, 2026 22:03 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread .github/workflows/run-tests-core.yml

Copilot AI review requested due to automatic review settings July 10, 2026 03:15

Copilot started reviewing on behalf of radical July 10, 2026 03:15 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 18:42

Copilot started reviewing on behalf of radical July 10, 2026 18:43 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread eng/scripts/build-test-matrix.ps1

Comment thread .github/workflows/tests-quarantine.yml

Comment thread .github/workflows/tests-outerloop.yml

Copilot AI review requested due to automatic review settings July 10, 2026 20:08

Copilot started reviewing on behalf of radical July 10, 2026 20:08 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 20:21

Copilot started reviewing on behalf of radical July 10, 2026 20:22 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

radical and others added 4 commits July 11, 2026 17:25

Copilot AI review requested due to automatic review settings July 11, 2026 22:02

radical force-pushed the update-specialized-test-runner branch from f2184cb to 2cc0cf4 Compare July 11, 2026 22:02

Copilot started reviewing on behalf of radical July 11, 2026 22:02 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

Comment thread .github/workflows/specialized-test-runner.yml

Comment on lines +254 to 255

extraTestArgs: "${{ inputs.extraTestArgs }}"

ignoreTestFailures: ${{ inputs.ignoreTestFailures }}

		extraTestArgs: "${{ inputs.extraTestArgs }}"
		ignoreTestFailures: ${{ inputs.ignoreTestFailures }}

Uh oh!

Conversation

radical commented Apr 24, 2026

Description

Key changes

Tests added

Checklist

Uh oh!

github-actions Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

JamesNK left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 24, 2026 •

edited

Loading