Tracking: CI improvements

[radical]

Goal

Track in-flight and planned CI improvements for the repo, organized into goal-oriented
epics: pipeline performance, pipeline architecture, automated failure monitoring, quality
gates, quarantine automation, and CI-health reporting.

Legend: ✅ shipped · 🔧 in progress · ⬜ planned · 🧩 needs issue

---

Epic 1 — Faster CI

Goal: cut CI wall-clock and runner cost without losing coverage.

• ✅ perf(ci): cut microsoft-aspire pipeline wall-clock from 121min to ~57min #17760 — cut microsoft-aspire pipeline wall-clock from ~121 min to ~57 min
• ✅ [ci] Make internal build robust to download failures #18057 — make internal build robust to download failures
• ⬜ feat(ci): select only affected tests and jobs per PR (audit mode) #18127
• ci: cache repo-local .dotnet in setup_for_tests (restore everywhere, save on main/release) #18244

Epic 2 — CI architecture / pipeline unification

Goal: one maintainable test-pipeline definition; remove duplicated CI plumbing.

• ⬜ Unify quarantine and outerloop CI with the main test pipeline #16448 — unify quarantine + outerloop CI with the main test pipeline (extract run-tests-core.yml)

Epic 3 — Automated CI failure monitoring & triage

Goal: no failure on main/release/* goes unseen; agents do first-pass triage and route
to the right owners. Builds on the repo's existing gh-aw setup.

Producers — standardized failure → issue (every workflow/pipeline files in the same shape,
via shared infra, so the consumer can monitor them uniformly):

• 🔧 ci: file deduplicated GitHub issues for failing CI, scheduled, nightly, outerloop & quarantine runs #18058 — file GitHub issues when scheduled/outerloop/quarantine CI fails (automation-broken); shared issue format + infra — the linchpin for the consumer below
• ⬜ ci(notify): file/update ci-broken GitHub issue when internal AzDO build breaks on main/release/* #17920 — file/update a ci-broken issue when the internal AzDO build breaks on main/release/* 🔴 Blocked on creating a new var group

Consumer — failure-watch agent (gh-aw first, local GitHub App next):

• 🧩 Watch automation-broken + ci-broken issues → post to Teams · maintain a pinned
  status-board issue · ping owners · run initial analysis · dedup. Adapts gh-aw
  ci-doctor + aw-failure-investigator; consumes both GH- and AzDO-filed issues.

Epic 4 — Improved quality gates

Goal: catch breakage before it ships.

• ⬜ Validate published Aspire CLI builds end-to-end from AzDO + GH #17532 — validate published Aspire CLI builds end-to-end from AzDO + GH
• ⬜ Add zizmor GitHub Actions security analysis workflow #15987 — add zizmor GitHub Actions security analysis workflow

Epic 5 — Quarantine system

Goal: data-driven, auditable quarantine / unquarantine.

Open architecture decision: repo-local memory vs ciinsights DB + API — to be resolved
in this epic before building the tracker.

• 🧩 gh-aw quarantine tracker + auto-(un)quarantine — per-OS pass/fail history on per-test
  issues; human-approved quarantine/unquarantine PRs. Implements the "separate process" in
  docs/unquarantine-policy.md.
• ⬜ [tests] Quarantined tests dashboard #8813 — quarantined-tests dashboard (related)

Epic 6 — CI-health reporting

Goal: turn CI signals into a recurring report and act on them.

• 🧩 gh-aw CI-health report — scheduled pinned-issue report (repo-pulse pattern)
• 🧩 gh-aw CI-health remediation — split slow test projects; add recurring flaky/network
  signatures to auto-rerun patterns

General / tooling

• ✅ Teach pr-testing skill to test infra changes #18062 — teach pr-testing skill to test infra changes

---

Agentic guardrails (apply to Epics 3, 5, 6)

Confidence thresholds before acting · keep rolling pass-rate even when a rerun recovers ·
quarantine budget + enforced de-quarantine · cost/recursion caps · global /agent-stop ·
every code or quarantine mutation goes through a human-approved PR.