📈 CI Daily Pulse

[radical]

🟢 rolling main green at tip (33% — 2 infra/flaky reds cleared) · 🟢 internal 100% (▲30) · 🟢 outerloop 100% · 🟢 PR 80% ⭐ — nothing needs you

✅ Nothing needs you — rolling main recovered: green at the tip since Jul 24 00:05 and quiet ~20h

All three earlier main job failures were non-product: a Windows hosted-runner drop (Hosting-1), an ubuntu disk-full (Templates), and one Cli windows timing test that timed out under CPU load — no shared cause, no deterministic regression. Internal + outerloop are green at the tip; PR flakes (2 branches) are being absorbed by the rerun bot (PR-only, 8 rescued). See § 🧩 below.

_{Window: Thu Jul 23 08:06 → Fri Jul 24 20:06 UTC (last 36h) · Δ = vs the 7-day average · Jul 24 2026.}

📊 Last 36h by lane — activity, pass rate & Δ vs 7d

Lane	36h activity	36h pass rate (Δ vs 7d)	Latest build
🌳 GH CI — main	🟩🟥🟥🟥	🔴 33% (1/3, 95% CI 6–79%) · ▼ 19.0	🟢 Jul 24 00:05
🧪 GH CI outerloop — main	🟩🟩	🟢 100% (1/1, 95% CI 21–100%) · ▬ 0.0	🟢 Jul 24 03:02
🏗️ Internal (AzDO) — main	🟩🟩🟩	🟢 100% (2/2, 95% CI 34–100%) · ▲ 30.0	🟢 Jul 24 02:18
🔀 GH CI — PR validation ⭐	🟩🟩🟩🟩🟩🟩🟥	🟢 80% (12/15) · ▼ 4.6	🟢 Jul 24 11:59

🧱 infra share: ~67% (2 of 3 failed main jobs were environmental) — a Windows hosted-runner drop (Hosting-1, "runner lost communication") + an ubuntu disk-full (Templates, "No space left on device"); the 3rd was a Cli windows timing test that timed out under runner CPU pressure (intermittent flake, not a code regression). Internal AzDO main was 2/2 green — no failed internal build to link. (Release lanes omitted — GH release/13.4 and AzDO release/13.4 had no in-window builds.)

_{Legend: bar length = sample size / confidence (short = read its % with care) · Δ vs 7d = 36h rate minus the lane's 7-day average · Latest build = red/green of the lane's most recent in-window run, linked to it · lane names link to the workflow / pipeline.}

🧩 What's broken & why

✅ Nothing needs active attention right now. Rolling main was intermittently red earlier in the window (2 of 3 runs), but it is green at the tip (Jul 24 00:05) and has been quiet ~20h. All three failing jobs were infra runner drops or a load-induced Cli timing flake — no shared product cause, no deterministic regression. The rerun bot is absorbing PR flakes (PR-only).

	What's broken	Who's on it	Since
⚪	rolling main — two earlier reds, no shared cause: a Windows runner drop + an ubuntu disk-full + a Cli windows timing test that timed out under CPU load; green at the tip now	Looks resolved — recovered to green; every cause is infra or a flaky timeout, not a code regression	last failed ~20h ago (Jul 24 00:03), green since 00:05
🟡	PR validation — a few PR runs flake; 2 branches red at their latest attempt, different owners, no common cause	Rerun bot is absorbing these (PR-only); each branch owner owns their fix	ongoing; bot rescued 8 this window; 2 branches left red

🧱 Product vs infra: today's reds are not a product-code regression — 2 of 3 failed main jobs are environmental (Windows hosted-runner drop, ubuntu disk-full) and the 3rd is an intermittent Cli timing test that timed out under runner CPU pressure. Internal AzDO is 2/2 green, so no failed internal build to link.

⏱️ Build-time note (read lightly, n=3): main's three in-window runs ran long — ~67m incl. the green tip vs ~27m 7-day norm — so the script flagged it, but on only 3 samples (2 of them failed) this is not a confirmed lane-wide slowdown.

exact tests, jobs & counts

GH CI — main, in-window: 1 success / 2 failure = 33%. Tip run 30055153746 (Jul 24 00:05) is green; no main push since (~20h quiet). Roll-up "Final Results" / "Final Test Results" aggregator jobs excluded from the failing-job count.

• ⚪ run 30048462581 — Jul 23 22:02 (commit "Add .NET 11 support to project templates" Add .NET 11 support to project templates #18849):
  • Tests / Hosting-1 / Hosting-1 (windows-latest) — hosted-runner drop, annotation: "The hosted runner lost communication with the server." Environmental, no test assertion, no failed step.
• ⚪ run 30055044537 — Jul 24 00:03 (commit "Update dependencies from dcp"):
  • Tests / Cli / Cli (windows-latest) — test Aspire.Cli.Tests.Projects.ProcessGuestLauncherTests.LaunchAsync_WithGracefulServices_BlockingSignalerDoesNotConsumeGracefulBudget threw System.TimeoutException : The operation has timed out. (~20s) — a timing-sensitive test timing out under runner CPU pressure, i.e. intermittent, not deterministic. No open tracking issue found for this test name.
  • Tests / Templates-XUnit_Default_NewUpAndBuildSupportProjectTemplatesTests (ubuntu-latest) — runner disk-full, annotation: "System.IO.IOException: No space left on device". Environmental, no failed test step.
• ⚪ Successful main run in-window (the tip): 30055153746 (Jul 24 00:05, ~2m after the red at 00:03). Auto-rerun is PR-only — this rolling red cleared on the next clean push, not a bot rerun.

PR validation, in-window: 12 success / 15 completed = 80% (excludes 8 cancelled, 12 awaiting-approval, 1 startup_failure). Latest completed run 30091438099 (Jul 24 11:59) is green. Two branches sit red at their latest in-window attempt after the bot exhausted retries — dapine/publisher-output-contract and dapire/security-deps/aspire-lowrisk-batch (distinct owners, no shared cluster; the latter flaked twice); the rerun bot rescued 8 red→green (see § 🔁).

ciinsights: returned 0 builds/failures across 36h and 7d for microsoft/aspire this run (see footer), so the per-test cluster join was unavailable — the failures above were reconstructed directly from the failed GH job annotations / logs.

_{Attention dots (this section): 🔴 needs a human now · 🟡 keep an eye on it · ⚪ looks resolved — distinct from the lane table's 🔴/🟢 run-outcome dots.}

🔁 Reruns & flaky tax

The auto-rerun bot carried the entire rescue load this window — all 8 PR runs rescued red→green were the bot (PR-only); zero human reruns, so manual toil was nil. Low volume overall (quiet window).

rerun actor split (36h, PR validation)

• 🤖 Auto-rerun bot: 22 reruns across 13 PR runs → 8 rescued red→green; 3 still red, 2 cancelled.
• 🧑 Manual (human) reruns: 0 reruns across 0 PR runs (0 branches the bot also reran) → on the 0 human-only branches: 0 rescued, 0 still red, 0 cancelled.
• 🔗 Combined: 13 distinct PR runs needed a rerun in the window (8 went green, 3 still red, 2 cancelled); 0 PR branches needed both bot + human.

_{Sources: ciinsights MCP + gh api …/ci.yml/runs (pass rates + rerun split) + az pipelines build list (def 1602). Window = last 36h. Jul 24 2026. ciinsights degraded this run — the connection reported healthy, yet list_builds / query_test_failures / get_failure_summary / get_health_summary all returned 0 builds/failures for microsoft/aspire across both 36h and 7d (no_data_in_time_window), which blocked the per-test cluster join; the main failures were reconstructed from GH job logs instead. Backlog & trend: see the weekly CI Health report (#18231). Generated 2026-07-24 20:06 UTC.}

[joperezr]

🤖 Aspire triage pass — 2026-06-16 06:54 UTC

Area: �rea-engineering-systems
Kind: docs (informational CI health report — [pipeline trial] per title)
Repro: n/a — this is a generated dashboard, not a bug report
Affected version(s): n/a (CI health snapshot, window: last 36h ending 2026-06-16)
Severity: S4 — informational; no product regression. Body flags real CI breakage (SqlServer conformance cluster, outerloop on main red 7+ days), but those are tracked as separate test-failure issues.

Urgency: ⚪ backlog — no milestone
Reason: informational dashboard / pipeline trial output, not a fix candidate. CI health concerns it surfaces are already being tracked under �rea-engineering-systems work.

Suggested owner(s): @joperezr (score 6)
Evidence:

• joperezr merged Merge release/13.4 to main after v13.4.0 release #17817, Merge release/13.3 to main after v13.3.5 release (supersedes #17337) #17343 in �rea-engineering-systems (last 90d)
• Issue author (@radical) is the originator of this pipeline trial and is excluded from owner suggestions per policy

Confidence: medium
Needs human?: no — informational; @radical is driving the trial and can dismiss/close when the experiment is over.

Dry-run mode — no labels or milestone applied. Reply /triage apply to promote, or edit and reapply.

[radical]

🔧 Post-run review — 2026-06-18 19:48 UTC

First run of the new two-tier Pulse spec (aspire-ci-pulse.prompt.txt + shared aspire-ci-report-common.prompt.txt, both committed today). The prior body here was produced by the older combined prompt, so a straight diff looks like a big regression — most of it is the intended split, but two things are worth a human's eye.

Regressions vs previous run: none that are true regressions — but two intended discontinuities a watcher will notice:

• Dropped sections (intended): the trend PNG, the <details> per-day grid, and the old "Newly…" flaky block are gone. The new spec explicitly moves trend/grid/build-time to the Weekly (🩺 CI Health — Last 7 days #18231); the Pulse is now current-state-only. Lane labels also moved to the canonical 🌳🌿🧪🏗️🔀 set. Deliberate, not a regression.
• Pass rates jumped sharply (methodology correction, NOT a real swing): PR validation reads 83% (76/92) vs the prior body's 38% (76/200); main reads 30% (6/20) vs 26% (6/23). Same passing counts — the denominator shrank because the new shared counting rule excludes cancelled / in-progress / queued runs, and PR ci.yml has heavy cancellation (65 cancelled in 36h alone). The new numbers are spec-correct; the jump reflects the rule, not reality. Calling this out so nobody reads it as "CI suddenly healed overnight."

Readability / improvement ideas:

• The headline blockquote is dense (lane + 3 tests + rerun split + two healthy lanes + watch clause). Skimmable but near the limit — accept: consider trimming the "PR + internal are healthy" clause to a half-sentence, since the table shows it. Reject if the contrast is the point.
• The activity bars are in a fenced code block so the emoji columns align on every client — reads cleanly. Keep.

Data-gathering issues this run:

• ciinsights query_test_failures threw An error occurred invoking 'query_test_failures'; get_failure_summary worked and supplied the top breakages + infra/flaky counts, so the Pulse was unaffected. The connection status also reported a DB index-creation warning (IX_TestResults_FullyQualifiedName_CapturedAt already exists) — API healthy, but flag if failure data ever looks stale.
• ciinsights test data can lag the last ~30 min (noted in-body); the rerun split is computed live from gh api .../ci.yml/runs, so it's current.
• First --paginate | jq pass silently produced per-page arrays (mangled counts); re-running with --slurp fixed it. The 36h ci.yml window held 194 runs across pages — a non-slurped read would have undercounted.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, infra-share rule. It says NN% (failed-infra/failed-total) / "share of failed builds", but the only figure ciinsights exposes is infrastructureFailureCount vs failingTestCount (test-result level: this run, 77/134 = 57%), not a build-level ratio. Pin the metric to one definition + one ciinsights field so successive runs don't silently switch denominators (build-level vs test-result-level). Small wording fix, durable payoff.

[radical]

🔧 Post-run review — 2026-06-18 20:48 UTC

Regressions vs previous run: none. The diff vs the 19:46 run is entirely genuine state movement — window slid ~1h, MongoDB ReplicaSet overtook the CLI TypeScript-apphost smoke as the single loudest test (11 vs 8), rerun split drifted (56/17 → 55/16), infra share recomputed (77/134 → 78/136), and 7d-avg deltas moved a fraction. No lost structure, no flipped statuses, no format churn. (Also trimmed a few trailing blank lines the prior body had accumulated.)

Readability / improvement ideas:

• (accept/reject) The headline now distinguishes the single loudest test (MongoDB ReplicaSet) from the loudest family (Aspire.Cli.*) in the 👀 Watch clause. This reads better than the prior run, which conflated the two. Suggest keeping this single-test-vs-family split as the headline convention.

Data-gathering issues this run (the valuable one):

• The rerun-split fetch via gh api .../ci.yml/runs?...&created=>2026-06-17... silently truncated to 82 of 169 runs on the first attempt because the > in the created filter was not URL-encoded — the shell/gh mangled the operator and the API ignored the lower bound, returning only one partial slice. I caught it only because I cross-checked against the gh run list count (94 completed PR runs in 36h) and against the API's own total_count (169). Without that reconcile, the rerun split would have been undercounted by ~50% (≈27 bot reruns instead of 55), badly distorting the flaky-tax section.
  • Fix that worked: URL-encode the operator (created=%3E2026-06-17T08%3A38%3A49Z) and --paginate, then assert jq -s length == .total_count.

Suggested prompt change (for a human to apply):

• File aspire-ci-report-common.prompt.txt, "Reruns & the flaky tax" section (or the "Fetch-enough-then-filter" section, which today only warns about gh run list --limit undercounts, not the runs-API created filter). Add one line: "When using gh api .../ci.yml/runs with a created=>… bound, URL-encode the operator (%3E) and verify the fetched run count equals the response's total_count — an unencoded > is silently dropped and truncates the window, halving the rerun split." This is a recurring trap (it will bite every run that fetches the rerun split this way), so it clears the "helps every future run" bar.
• Minor: the prompt never states the infra-share denominator. I inferred infra / (infra + failing) from the prior run's 77/134; this run's 78/136 matches the same formula. Worth one explicit line in the "Infra-share line" section so the figure is reproducible and doesn't drift between runs/authors.

[radical]

🔧 Post-run review — 2026-06-18 21:48 UTC

Regressions vs previous run: none. The diff vs the 20:46 run is entirely genuine state movement — window slid ~1h; MongoDB ReplicaSet ticked 11→12; the CLI TypeScript-apphost smoke (LatestCliCanStartStableChannelTypeScriptAppHost) aged out of the 36h window and StartAsync_…GracefulSignaler… took the #3 slot; rerun split drifted (55/16 → 58/20); infra share recomputed (78/136 → 79/135); 7d-avg deltas moved a fraction as the baseline window also slid. No lost structure, no flipped statuses, no format churn.

Readability / improvement ideas: none new — the headline + first table still convey "what's on fire" without scrolling.

Data-gathering issues this run (the new one):

• ciinsights.query_test_failures failed outright this run (An error occurred invoking 'query_test_failures'), and get_connection_status reported the backend DB as unhealthy (index ... 'IX_TestResults_FullyQualifiedName_CapturedAt' already exists — a stuck server-side migration). I fell back to ciinsights.get_failure_summary (its topFailingTests array) for the §3 breakages list, which worked. Without that fallback the loudest-breakages section would have had no source.
• This is plausibly recurring while the DB stays in that migration state, so it clears the "helps every future run" bar.
• I avoided the prior run's gh api .../ci.yml/runs created=> truncation trap by fetching paginated pages (per_page=100&page=N, 6 pages) and filtering by timestamp in code instead of an API-side created bound — confirms that prior finding's value.

Suggested prompt change (for a human to apply):

• (new — common file, "Reruns & the flaky tax" / data-sources) Add one line: "If ciinsights.query_test_failures errors, fall back to get_failure_summary.topFailingTests for the loudest-breakages list (same fail-count ranking, narrower window control)." Codifies a working mitigation for a tool that was down this run.
• (carried forward from 20:46, still unapplied — restated here because that comment is now resolved) (a) "Fetch-enough-then-filter": warn that an unencoded > in a runs-API created=>… bound is silently dropped and halves the rerun split — URL-encode (%3E) and assert fetched count == total_count, or fetch by page and filter in code. (b) "Infra-share line": state the denominator explicitly — infra / (infra + failing) (this run 79/135) — so the figure is reproducible and doesn't drift between authors.

[radical]

🔧 Post-run review — 2026-06-18 22:52 UTC

Regressions vs previous run: none. Diff against the prior body (published ~1h earlier) is all genuine state movement — window slid forward ~1h, main 6/20→6/21, internal main 14/16→14/17, PR validation 82/98→89/104, latest failed internal build 3003323→3003488, rerun counts 58/32→59/33. No section, column, link, or legend was dropped; no frozen value moved (the Pulse has no frozen history); no status emoji flipped without a real change.

Readability / improvement ideas:

• (accept/reject) The Watch clause was sharpened from last run's broad Aspire.Cli.* to the specific graceful-shutdown Signaler…Escalates family (3 named tests, repeating). This is more actionable — a reader can grep three exact test names. Suggest keeping Watch clauses this specific going forward rather than naming a whole namespace.
• The report is skimmable in ~15s: headline → lane table → reruns → breakages. No change needed.

Data-gathering issues this run:

• recentTrends.last24h.newFailingTests is a names-only list with NO per-test counts, and a large block in it is often ONE build's worth of a whole test class failing once — not N independent signals. This run it listed ~33 DistributedApplicationPipelineTests.* entries; cross-checking via search_tests showed every one had totalRuns=1, failures=1, all at the same timestamp (2026-06-17T23:03:34) — a single-build setup/compile blip, not a repeating breakage. Had I trusted the raw block I'd have featured a phantom "loud new family." Always validate newFailingTests entries against search_tests/failCount before featuring them.
• PR-validation 7d denominator needed a much larger fetch than the common prompt's --limit 300. At ~90–150 runs/day, 300 runs only spanned ~2.5 days (back to Jun 16) — short of the 7-day window. Refetched at --limit 1200 (reached Jun 7). The "fetch-enough-then-filter" rule held, but the suggested 300 floor silently undercounts this lane's 7d average.
• ciinsights get_connection_status reported a DB index warning (IX_TestResults_FullyQualifiedName_CapturedAt already exists); all queries still returned complete data. One-off, no impact this run.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, the "Fetch-enough-then-filter" section: add a note that for the 7-day average on PR validation specifically, --limit 300 is insufficient (covers only ~2–3 days); use --limit ~1200 (or paginate to the 7d start) for that lane. The 36h window is fine at 300.
• aspire-ci-report-common.prompt.txt (or the Pulse §3 / headline Watch guidance): add a caution that recentTrends.newFailingTests carries no counts and a bulk block is frequently a single failed run of one test class — verify against search_tests before treating it as a flaky/broken family or featuring it in the Watch clause.

[radical]

🔧 Post-run review — 2026-06-18 23:52 UTC

Regressions vs previous run: none. Diffed the prior body against this one — every section, column, link, the bar-length legend, and both <sub> lines are intact. The only changes are state-driven value updates (window slid ~1h, so the MongoDB ReplicaSet 36h count fell 12→8 as older fails aged out; internal main 82→88%, PR validation 86→85%, infra share 59→60%). Pulse freezes no history, so there is no frozen-cell drift to check.

Readability / improvement ideas:

• (accept/reject) The report can be built from one ciinsights call: get_health_summary returns both failureBreakdown (testFailures + infrastructureFailures + total → the 🧱 infra-share fraction) and topFailingTests (the §3 loudest-breakage list). This run, query_test_failures failed outright (server-side DB index error: IX_TestResults_FullyQualifiedName_CapturedAt already exists), but get_health_summary + get_failure_summary covered everything. Suggest the prompt name get_health_summary as the primary §1-infra/§3-breakages source so a future run doesn't get blocked on the flaky query_test_failures endpoint.

Data-gathering issues this run:

• Rerun split is methodology-sensitive and the prompt under-specifies it. The window moved only ~1h vs the prior run, yet bot reruns jumped 59→89 and human 20→23. Two unpinned choices drive this: (a) what counts as one "rerun" — I used Σ(run_attempt−1) per run, attributed to the latest-attempt actor; (b) how to identify a PR — 38 of 80 rerun-bearing ci.yml runs were fork PRs with an empty pull_requests array, so I fell back to head_branch as the PR-identity key (raising distinct-PR from 23 to 48, much closer to the prior run's 46). Without these pinned, two runs an hour apart can legitimately disagree by ~50%.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, the "Reruns & the flaky tax" section: pin the counting unit and the join key explicitly — (1) a "rerun" = each attempt beyond the first, i.e. Σ(run_attempt−1); (2) attribute a run's reruns to its latest-attempt triggering_actor (github-actions[bot] = bot, else human); (3) when pull_requests is empty (fork PRs), key the PR by head_branch. This is additive — it doesn't contradict the existing "split by who triggered it" rule, just makes it reproducible run-to-run.
• aspire-ci-pulse.prompt.txt / common data sources: note get_health_summary as the single source for both the infra-share fraction and the loudest-breakage list, with query_test_failures flagged as currently erroring.

[radical]

🔧 Post-run review — 2026-06-19 18:52 UTC

Regressions vs previous run: none. All deltas vs the prior body reflect genuine state changes. Structure preserved (header, headline, single lane table, infra line, legend, rerun split, loudest breakages, footer). The PR-validation activity bar shrank 12→11 cells — expected, not a regression: bar length is log-scaled to sample size and the lane's 36h run count dropped (109→72). The Internal AzDO release/13.4 row was omitted correctly (0 builds in window) and the omission is annotated inline so a reader doesn't think the lane was dropped silently.

Readability / improvement ideas:

• (accept/reject) The headline now carries two distinct test clusters (analyzer pair + EF Core Enrich) plus the "loudest by count vs most-recently-active" nuance. It's accurate but dense — at ~5 lines it's near the edge of skimmable. Consider a soft cap: lead with the single loudest cluster, push the "most-recently-active" distinction to §3 only. Reject if the recency caveat is considered load-bearing for "what's on fire now."

Data-gathering issues this run:

• Rerun-split attribution is an approximation. I bucketed reruns by triggering_actor of the latest attempt and counted run_attempt − 1 reruns per run, all assigned to that one actor. A run rerun first by the bot then by a human (or vice-versa) is attributed entirely to whoever triggered last; mixed-actor runs are only partially captured (I could detect "needed both" = 1 only via PR-set overlap, not per-attempt). This likely explains some of the large magnitude swing from the prior run (bot 89/human 23 → bot 43/human 16) — part real drop, part method variance. The prompts do not pin an exact counting method, so two runs can compute this differently.
• Infra-share denominator is ambiguous. The common prompt defines 🧱 infra share = failed-infra/failed-total as "share of failed builds", but ciinsights returns failure-occurrence counts (infrastructureFailureCount vs failingTestCount). I used 62/102 (infra ÷ infra+test signals); the prior run used 83/138 — a different denominator entirely. The metric isn't reproducible run-to-run without pinning the source + denominator.
• ciinsights get_test_history returned empty for an exact FQN (...MSBuildTests.AspireExportAnalyzers...) that get_failure_summary simultaneously reported with 34 fails in the same 36h window. So I could not use it to attribute the loudest test to a specific lane; the headline characterizes the analyzer pair by suite/family from the summary alone, not by lane. get_flagged_tests(problemType=broken) also did not surface the 34-fail pair in its top rows (it ranks multi-context broken tests; the pair appears single-context), so neither tool corroborated a lane for it.
• ciinsights get_connection_status reported the API healthy but the DB "unhealthy" (a duplicate-index warning, IX_TestResults_FullyQualifiedName_CapturedAt). All queries still returned data; flagging in case it foreshadows stale/partial results on a future run.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, "Reruns & the flaky tax" section: pin the exact rerun-counting method — e.g. "count reruns as Σ(run_attempt − 1) over completed ci.yml runs in the window; attribute each run's reruns to the triggering_actor of the latest attempt (a known limitation for mixed bot+human runs); bot = login ending [bot]." Without this, the bot-vs-human magnitudes are not comparable across runs.
• Same file, "Infra-share line" section: pin the denominator and source — state whether it's failed builds or failed signals, and that ciinsights failingTestCount / infrastructureFailureCount are the inputs (or specify the build-level alternative). This run (62/102) and the prior (83/138) used different denominators for the same metric.