Per-commit CI bottleneck analysis across PR, candidate-gate, and preview-release workflows (issue-360 rebase outcome)

[vishaltandale00]

Summary

Records the per-commit CI bottleneck analysis produced while rebasing codex/issue-360-warm-cache-levers onto origin/main (now 601460c, 2026-09-02), the rebase outcome itself, the decision on the one item that needed human judgment, and the post-merge measurements taken on 2026-09-08.

Evidence sources: docs/research/intra-pr-ci-cache.md (hosted timing measurements), docs/agents/ci.md, docs/desktop-release-operations.md, .github/workflows/ci.yml, .github/workflows/desktop-signed-preview.yml, the merged PRs #394, #405, #420, #422, #427, #428, #432, and the hosted runs linked below.

Rebase outcome

• The 26-commit branch was almost entirely squash-merged into main already: Ladybug prebuilt bundle + crash-lane narrowing + precise mappings + flake fixes via Optimize CI: Rust timing artifacts, crash-lane affectedness, precise mappings, flake fixes #394; cache-lever warming, runtime-bundle portfolio-constant keying, and parallel Vitest config via ci: warm the cache levers on top of the parallel Vitest portfolio #420; nearby Seed the locked Cargo closure before the offline preview build #427/Gate desktop candidates on the main CI check, fold the crash lane, run Vitest in parallel #428.
• The only unmerged content was the two-runner Vitest shard with its two review rounds. It was rebased into a single commit 7e4ed3aa ci: shard Vitest across two runners with a dedicated slow job (11 files, +441/-38) directly on origin/main (601460c). An earlier pass sat on 1dc9e50 as bd041af1; the re-rebase carried Stop wall-clock-bounded tests from reporting runner contention as failure #439's 15 s per-test-timeout note into the rewritten shard paragraph and kept Stop wall-clock-bounded tests from reporting runner contention as failure #439's 15 s timeout on the chapter-runner portfolio test.
• Two fixes were needed during the rebase:
  • Removed duplicated portfolio-keying code/test that ci: warm the cache levers on top of the parallel Vitest portfolio #420 had already landed in a better form.
  • Aligned the portfolio manifest's vitest shell with main's package.json (Gate desktop candidates on the main CI check, fold the crash lane, run Vitest in parallel #428 moved worker policy into vitest.config.js); the shard pins --maxWorkers=1 in the chapter runner instead.
• Verification run against the rebased tree: all six CI seam test files pass (101 tests: ci-affected-plan, ci-chapter-runner, ci-required-check, ci-verification-portfolio, ci-lbug-artifact, ci-runtime-artifact, plus desktop-release-main-ci-check); ci.yml YAML parses; JSON manifests and changed scripts pass syntax checks.
• Local state: branch codex/issue-360-warm-cache-levers = 7e4ed3aa; pre-rebase history preserved at backup/pre-rebase-2026-09-02 (e26c845), intermediate rebase at backup/pre-rerebase (bd041af1). Not pushed; the remote branch head (306b25f) was already merged as ci: warm the cache levers on top of the parallel Vitest portfolio #420.

Decision (2026-09-08): keep #428's one-runner parallel Vitest, drop the shard (7e4ed3aa)

7e4ed3aa carried no hosted measurements. The first post-#428 main run supplies them (33999200686, 2026-09-05):

Vitest job	vitest run wall	test/eval-app-integration.test.mjs	process-bound tail
125s	78s (176.6s of test CPU across the workers)	61s	~2s (the browser capture is skipped on CI)

The slow file already overlaps the rest of the isolated group on the one runner; it is the tail. A dedicated vitest-slow runner could bring the Vitest wall from ~78s to ~61s (the file's own duration) at ~40s of extra billed setup per run, and #420 found the shard silently broken on top of projects (CLI --exclude does not propagate into a project's resolved exclude). Decision: drop 7e4ed3aa. docs/agents/ci.md already documents the #428 arrangement, so no documentation change follows. The remaining Vitest headroom is inside the slow file (17 tests, 61s), not in runner topology.

Branch housekeeping: local codex/issue-360-warm-cache-levers (7e4ed3aa), backup/pre-rebase-2026-09-02 (e26c845), and backup/pre-rerebase (bd041af1) can be deleted; remote codex/issue-360-warm-cache-levers (306b25f) is merged as #420.

Post-merge measurements (2026-09-08)

The PR-workflow critical path is now packaging

Run	Vitest chain ends	Packaging ends	check starts
main 33999200686	23:42:31	23:43:19	23:43:56
Rust PR 33998956562	23:37:03	23:37:15	23:37:18

A non-Rust, non-desktop PR (runtime cache hit, packaging not selected) now finishes in ~1m25s (33999557594: plan 17s → Vitest 48s → check 9s).

Prime runtime package (macos-arm64) took 275s on main: npm ci 24s, target-cache restore 16s, desktop:pack 213s. Inside desktop:pack:

Phase	Time
prepare:renderer + build:packages	8s
pinned Ladybug source staging + static OpenSSL 3.5.8 build	~62s
cargo build --release of the four workspace crates (dependencies fresh from the restored target/)	93s
electron-builder --dir + afterPack verification	~50s

The 93s is structural to the target-archive approach: on a fresh checkout Cargo's mtime fingerprints mark every path crate dirty, so the restored target/ only ever saves dependency compilation. Re-keying the archive would not help (as docs/agents/ci.md already notes). The lever with headroom is the #420 pattern applied to packaging: a digest-keyed, trusted-main-seeded bundle of the two aarch64-apple-darwin release binaries, verified through scripts/ci/runtime-artifact.mjs, installed into target/aarch64-apple-darwin/release/, with desktop:pack skipping the Ladybug/OpenSSL preparation and cargo when a verified bundle is installed and failing open to today's build otherwise. Expected: packaging ~4m35s → ~2m, after which the runtime lane → Vitest chain (~3m50s on Rust PRs, ~2m10s on non-Rust desktop PRs) is the critical path again. Plan pending approval.

4 Sep slow runs: an npm audit stall, not runner contention

Runs 33861852333 (10:09Z) and 33862873159 (10:21Z) took 9–11 minutes instead of ~5. Every job that runs npm ci lost 4–7 minutes in Set up Node dependencies (TypeScript builds 433s, Receipt integrity 430s, Quick 332s, Vitest 313s, packaging 323s); the Rust lanes, which do not run npm ci, were normal. In the slow jobs the ~/.npm cache was an exact hit (3s restore) and the runner image was identical to the fast runs (ubuntu-24.04 20260831.293.1). npm ci printed added 612 packages in 7m with no audited clause, where fast runs print added 631 packages, and audited 637 packages in 16s. npm status posted "npm audits endpoint is having issues" at 11:29Z that day; npm's fetch-timeout default is 300s with two retries, which matches a ~7-minute stall on the audit submission. #439's test-timeout changes address the symptom inside tests; the CI-side mitigation is npm ci --no-audit --no-fund in .github/actions/setup-node-dependencies (nothing in the repository consumes npm audit), which also trims a few seconds from every npm ci.

Cache budget

Repository cache stood at 8.61 GB of the 10 GB limit on 2026-09-08. Deleted 16 stale npm download caches (ten npm-* and six node-cache-* entries for superseded lockfile hashes, ~3.7 GB); usage is now 5.98 GB with only the two live npm-* entries for the current package-lock.json. PR-scoped sccache entries for merged PRs still hold ~2 GB and expire on GitHub's schedule.

Workflow definitions used

• PR workflow = ci.yml on pull_request: planner -> chapters -> check aggregate.
• PR release (candidate gate) = Desktop Signed Preview Candidates validate requires a completed successful full-portfolio ci.yml push run on main for the exact candidate commit.
• Preview workflow release = desktop-signed-preview.yml: validate -> per-target signed/notarized package -> tagged Preview publication.

Per-commit bottleneck mapping

Commit	PR workflow bottleneck addressed	PR release	Preview release
24cc6b1 Cargo timing reports	Hidden critical unit: Ladybug build-script = 72-86% of every Rust lane	-	-
b6f4849 narrow crash lane	Crash lane (~12-14 min) fired on non-crash changes	Shortens the exact-commit gate run	-
52f5036 precise ownership maps	Full-mode over-selection: ~11-16 min portfolio where an affected subset suffices	Same gate-run saving	-
a63b63b deterministic shutdown/skill checks	Flake reruns cost a whole lane each	A rerun redoes the entire release-commit gate	-
7be3547 fix escapes + timings mechanism	Wrong timings would select wrong levers	-	-
2eab738 runtime lane read-only sccache	Cache write races: 437-1029 write errors/lane, self-perpetuating cold misses	-	-
c38a130 record convergence, pin action default	Same race; config drift	-	-
6c7aac1 epoch-reset load-tolerant deadline	Load-induced flake -> reruns	Rerun risk on release commit	-
5d7f8e1 portable deterministic checkpoints	Environment-dependent flakes -> reruns	Same	-
b60542c planner load-tolerant timeout	Planner flake under runner contention	Same	-
1c14d34 unified target dir	Ladybug object cache fragmented per lane (281-480 misses -> 116-150)	-	-
67d13cb pin seal dir, ccache premise	Silent ccache launcher switch would destroy cross-run warmth	-	-
d8f20d5 / a730442 review rounds	Mapping, harvest, and rerun integrity	-	-
5d61f9b retrigger run	Ops only	-	-
Docs commits (54fe9f7, 9498bde, e22aa6a, 99ebd4e)	Evidence: lane timings; 244-328 s Ladybug floor is cache overhead not compile (43 s warm local)	-	-
54637fd Ladybug built once (#394)	Single biggest bottleneck removed: Ladybug CMake build 244-328 s x 4 lanes -> build once, verify, restore from cache/artifact	Gate run paid the same floor -> candidates waited on it	- (preview builds on macOS runners, separate path)
e797664 harden prebuilt bundle	New failure mode (bad/stale bundle) must fail open, never block a lane	-	-
fcaba5d + e26c845 cache levers, shard, review round 2	(a) Runtime bundle missed structurally for narrow plans -> full runtime-lane build on every small PR; (b) slow Vitest file extends the Vitest wall; (c) under-covering bundle could store/install; empty-shard guard	Shorter exact-commit gate run	-
e26c845 portfolio-constant keying (#420)	Narrow plans structurally missed the seeded runtime bundle - "the exact small-change PRs the lever accelerates"	Same	-
7e4ed3aa rebased residual (dropped 2026-09-08)	Slow file test/eval-app-integration.test.mjs still ends the Vitest wall after #428's parallel config; a second runner would recover ~17s, see Decision above	-	-

Biggest bottlenecks per workflow today (post-merge, 2026-09-08)

1. PR workflow: (a) the packaging job's release rebuild plus Ladybug/OpenSSL preparation (~155s of a 275s job; the current critical path on main and on Rust PRs); (b) changed-unit Rust misses (~100-114 s per workspace crate; the Ladybug floor is gone); (c) the runtime lane -> Vitest chain (~97s + ~125s), which becomes critical once packaging drops; (d) fail-open full-mode selection for unmapped paths (by design, ~11-16 min); (e) npm ci exposure to registry-side stalls through the audit submission.
2. PR release (candidate gate): structural, not code - the candidate waits for the full-portfolio main CI run on the exact commit (fails closed; ~6 min warm after ci: warm the cache levers on top of the parallel Vitest portfolio #420, longer cold), and any flake reruns the whole gate on the release commit.
3. Preview release: validate chains on that same main run; then cargo fetch --locked network fetch (mitigated by Seed the locked Cargo closure before the offline preview build #427's closure seeding); then the ~3m46s-4m18s package build plus external Apple signing/notarization latency; publication is gated on tag-annotation integrity (Restore the annotated tag object before reading its candidate pin #432), not time.

Suggested next steps

• Decide the Gate desktop candidates on the main CI check, fold the crash lane, run Vitest in parallel #428-vs-shard question: keep Gate desktop candidates on the main CI check, fold the crash lane, run Vitest in parallel #428, drop 7e4ed3aa (2026-09-08).
• Delete the local shard and backup branches listed under Decision.
• Approve and implement the digest-keyed release bundle for packaging (plan pending).
• npm ci --no-audit --no-fund in .github/actions/setup-node-dependencies.
• Track the remaining PR-workflow bottlenecks (changed-unit Rust misses, full-mode over-selection) separately if they need work.

[vishaltandale00]

Rebased onto the latest main (601460c) after #439 and the #404/#423 batch landed. The shard commit is now 7e4ed3aa (was bd041af1 on 1dc9e50). #439 touched two files this branch also touches: the rewritten shard paragraph in docs/agents/ci.md now carries #439's 15 s per-test-timeout note (adapted to the shard model), and #439's 15_000 ms timeout on the chapter-runner portfolio test survives the auto-merge. Re-validated on the new tree: 90 tests pass across the six CI seam files plus desktop-release-main-ci-check (a first local run showed one load-induced contention failure at 153 s total; the identical rerun passed in 8 s — the contention class #439 hardens against). #439 itself joins the flake-hardening row of the analysis: it stops wall-clock-bounded tests from reporting runner contention as failure, which lowers rerun risk on the exact-commit candidate gate (PR release bottleneck).

[vishaltandale00]

Churn analysis redone on current main (601460c)

Method: 14-day window (2026-08-19..2026-09-02) of origin/main, 136 non-merge commits, --numstat file lists classified through the planner's own rules in scripts/ci/affected-modules.v1.json (fullPortfolio.exact + prefixes = full-mode fail-open triggers). Warm-run costs from docs/research/intra-pr-ci-cache.md measurements.

Headline: 54% of main commits force full-portfolio mode

74 of 136 commits match a full-mode trigger. Warm full run ≈ 11 min vs ≈ 3–5 min for an affected npm/test-only run — the conservative fail-open contract is the single biggest CI-time buyer on current main.

Full-mode trigger frequency (commits, any=co-triggered, solo=only trigger):

Trigger	any	solo
package-lock.json	31	24
package.json	23	12
.github/	23	7
docs/evidence/	11	2
Cargo.lock	10	1
scripts/ci/	8	1
infra/	3	1
fixtures/graph-query-v1/	2	1
Cargo.toml / .cargo/ / others	≤4	0

Corrections/refinements to the earlier bottleneck write-up:

1. npm manifest churn, not evidence churn, dominates full-mode selection. package.json/package-lock.json are involved in ~50 of the 74 full commits (68%); 36 commits are full-mode solely because of npm manifests. None of that churn can affect the Rust lanes, yet full mode runs them. docs/evidence/ has huge file volume (318 file touches — evidence dumps batch hundreds of files) but only 11 commits, 2 solo-full.
2. ~19 of the 74 full commits were the issue-360 CI work itself (.github/, scripts/ci/, evidence, infra) — transient. The steady-state full-mode driver is npm lockfile churn.
3. Cargo.lock churn (the cache-invalidation event) has stopped: all 10 events landed 2026-08-20..08-31, none in the last 2 days. Each event invalidates the lbug-prebuilt key, the runtime-bundle Rust-input digest, the dependency archive, and packaging caches → main-push reseeds (Ladybug build ~4 min + runtime build ~6–8 min + multi-GB uploads). We are currently in a warm-key regime; the next dependency bump batch re-pays it.
4. crates/** churn: 37/136 commits (27 of them also full-mode). Every crates touch changes the runtime-bundle digest → the PR restores nothing and the runtime lane builds fresh (~6–10 min warm), then the main push reseeds the trusted bundle.

Affected-only commits (62) by dominant lane cost

Class	Commits	Warm cost shape
vitest/typescript (+packaging when desktop/)	39	Vitest ~1.9–3 min post-#428; packaging ~4.5–5 min when desktop/ (42 of 62 touch desktop/)
docs/prd only	11	prd readability, ~1 min
rust lanes	10	4 lanes ~6.5–10.5 min warm (Ladybug floor removed; residual = changed-unit rebuild ~100–114 s/crate)

High-churn single files (commits touching, 14d)

docs/prd/index.html 45 (cheap: prd chapter) · test/desktop-shell.test.mjs 43 (process-bound Vitest file, main shard) · package-lock.json 31 (full-mode trigger) · desktop/renderer/styles.css 31 · desktop/package.json 29 · codex-basic.ts 26 · workspace.js 26 · README.md 26 · desktop/main/index.mjs 26 · package.json 23 (full-mode trigger) · product_persistence_flow.rs 22 (the known flake site, now hardened by #439).

Note test/eval-app-integration.test.mjs — the slow-shard file this branch isolates — churned in 13/136 commits (~10%). It is both the Vitest wall candidate and a genuinely hot seam, so it will keep re-entering selections. Counterpoint for the #428-vs-shard decision: with #428's parallel config the whole Vitest portfolio already finishes in ~1.9 min, so the shard's upside is bounded well under a minute of wall time at 2× runner count, unless the slow file's runtime grows.

Per-workflow churn conclusions (current main)

1. PR workflow: dominant churn cost = full-mode fail-open selection (54% of commits; npm manifests are the top driver). Second = crates/** digest churn rebuilding/refilling the runtime bundle. Third = the slow Vitest file (10% of commits). The Rust-lane Ladybug floor is gone (post-Optimize CI: Rust timing artifacts, crash-lane affectedness, precise mappings, flake fixes #394); residual Rust churn cost is changed-unit recompiles.
2. PR release (candidate gate): churn does not change scope (the gate is always the full portfolio on the exact commit), but it changes warmth: a candidate whose commit touched Cargo.lock/crates/** (27% of commits in the window: every Cargo.lock bump also touched crates/) pays cold Rust + reseeds (~24 min class) instead of ~11 min warm. Flake-rerun risk on the gate is reduced by Stop wall-clock-bounded tests from reporting runner contention as failure #439.
3. Preview release: churn inputs are desktop/** (102/136 commits touch it — packaging sources), vendor/prime-agent (9 commits — pinned managed runtime the candidate builds; 11 commits touched vendor/ overall), and Cargo.lock (re-fetches the Seed the locked Cargo closure before the offline preview build #427 seeded closure when it moves; currently stable). Signing/notarization latency is churn-independent.

Highest-leverage follow-ups (need ownership review, not obvious wins)

• Map package.json/package-lock.json churn to the npm-consuming chapters (vitest, typescript, packaging) instead of full mode — biggest single saving (~36 commits/14d × ~7 min), but repo policy requires explicit owners for every consumer before a path may leave full; npm manifests plausibly touch Python receipts, PRD tooling, and runtime catalogs too.
• Watch the next Cargo.lock bump batch: it re-pays every cache reseed at once; batching dependency bumps into one main push amortizes it.
• Re-measure the slow shard against Gate desktop candidates on the main CI check, fold the crash lane, run Vitest in parallel #428's 1.9 min baseline before merging 7e4ed3aa (see decision checklist above).