Compute coverage on a single CI leg instead of every leg (part of #2702)

[acoliver]

Compute coverage on a single CI leg instead of every leg

Summary

Coverage instrumentation is enabled unconditionally in every package's Vitest config, so it is computed on all four matrix legs while only one set of coverage artifacts is consumed.

Evidence

coverage: { enabled: true } appears in 11 package configs:

agents, auth, cli, core, ide-integration, mcp,
providers, settings, storage, telemetry, tools

The CI test job uploads a coverage artifact per matrix leg:

coverage-reports-24.x-ubuntu-latest-keyring
coverage-reports-24.x-ubuntu-latest-fallback
coverage-reports-24.x-macos-latest-keyring
coverage-reports-24.x-macos-latest-fallback

post_coverage_comment consumes coverage from a single leg.

Measured instrumentation cost on packages/cli:

baseline (coverage on)   138s
coverage off             119s

Approximately 19s for one package on one leg, repeated across packages and legs.

Proposed work

1. Make coverage instrumentation conditional, enabled on one designated CI leg and disabled elsewhere.
2. Keep coverage on by default for local runs, or provide an obvious opt-in, so developers do not lose the signal.
3. Remove the now-redundant per-leg coverage artifact uploads.

Constraint

Coverage gates and the coverage comment must keep working. This issue reduces redundant computation; it must not reduce coverage visibility or drop an enforced threshold.

Acceptance criteria

• Coverage is computed on exactly one CI leg.
• The coverage comment and any coverage gate continue to function.
• Redundant coverage artifacts are no longer uploaded.
• Before/after timings recorded for at least the largest package.

Note

The matrix-leg count is itself being reduced by the sibling issue that removes the dead secure-store-mode dimension. These compose: fewer legs and coverage on one leg.

Implementation plan

Files to edit

13 coverage.enabled: true sites across 11 packages:

packages/agents/vitest.config.ts:270
packages/auth/vitest.config.ts:53
packages/cli/vitest.config.ts:347
packages/core/vitest.config.ts:224, 231
packages/ide-integration/vitest.config.ts:48
packages/mcp/vitest.config.ts:111, 124
packages/providers/vitest.config.ts:230
packages/settings/vitest.config.ts:113
packages/storage/vitest.config.ts:48
packages/telemetry/vitest.config.ts:113
packages/tools/vitest.config.ts:113, 120

Plus .github/workflows/ci.yml lines 845-849 (artifact upload) and 876 (consumer).

Note core, mcp, and tools each have two enabled: true sites — the second is inside the typecheck block. Check both.

Steps

1. Introduce one shared switch, e.g. enabled: process.env.LLXPRT_COVERAGE !== 'false', defaulting to on so local runs are unaffected. Prefer a single shared helper over 13 independent copies — a divergent 14th will otherwise appear.
2. Set the disabling env var on all but one designated CI leg.
3. Remove the redundant per-leg coverage artifact uploads (849), keeping the designated leg's.
4. Confirm post_coverage_comment (876) still finds its artifact.

Verification

cd packages/cli && npx vitest run --coverage.enabled=false --reporter=dot
# confirm test count unchanged, duration reduced

On the PR: confirm the coverage comment still posts and any coverage gate still evaluates.

Constraint

Do not drop an enforced coverage threshold. This removes redundant computation, not coverage visibility.

Sequencing

Compose with the dead-matrix removal: fewer legs first, then coverage on one of the survivors.

[coderabbitai]

🔗 Related PRs

#1122 - fix(ci): skip E2E tests and coverage comment for docs-only PRs (#1114, #1119) [merged]
#2518 - Focus secure-store backend CI coverage (Fixes #2147) [merged]
#2682 - Fix CI flakiness misdiagnosed as macOS runner faults (Fixes #2668) [merged]
#2689 - Serialize scripts/tests files to prevent cross-file lock collisions (Fixes #2683) [open]
#2690 - Gate ubuntu-22.04 in CI and stop the assign harness hiding failures (Fixes #2688) [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[acoliver]

@coderabbitai please plan

[coderabbitai]

Coding Plan

Summary

• Add one shared, dependency-free coverage-enabled helper (reading LLXPRT_COVERAGE !== 'false', defaulting on) and route all 11 packages' coverage.enabled through it, leaving the unrelated typecheck blocks in core/mcp/tools untouched.
• In CI, set LLXPRT_COVERAGE=false on every leg except the single designated leg (ubuntu-latest + keyring) using the existing matrix-driven env ternary pattern, so coverage is computed exactly once.
• Restrict the coverage-artifact upload to that designated leg and confirm post_coverage_comment still finds its artifact; mirror the toggle into nightly.yml and drop its unconsumed coverage upload.
• Verify unchanged test counts with reduced runtime and record before/after timings for packages/cli, preserving coverage visibility with no threshold changes.

Design Choices

Design Choice 1: Where should the shared coverage toggle live, given no shared Vitest config module exists today?

Options Considered:

1. Create a small root-level helper module (e.g., vitest.shared.ts / scripts/vitest-coverage.ts) exporting a isCoverageEnabled()-style function that each config imports via relative path.
2. Add the helper to packages/test-utils and import it from each package config.
3. Inline the process.env.LLXPRT_COVERAGE !== 'false' check in each config (no shared module).

Chosen Option: 1

Rationale: The ticket explicitly asks for "a single shared helper over 13 independent copies." Package configs currently import only vitest/config and Node builtins, so the helper must be importable without introducing build-order or workspace-dependency complications. A lightweight root-level helper avoids adding an inter-package build/dependency edge (test-utils is not currently consumed by any config) while still satisfying the "single source of truth" requirement and preventing a divergent 14th copy.

Design Choice 2: How should `nightly.yml`'s `windows_ci` coverage be handled?

Options Considered:

1. Set LLXPRT_COVERAGE=false in nightly's test step and drop its now-empty coverage upload.
2. Leave nightly unchanged (coverage still computed and uploaded but unused).

Chosen Option: 1

Rationale: Nightly uploads a coverage artifact but never runs post_coverage_comment, so its coverage is never consumed. Disabling coverage on the never-consumed nightly leg extends the same redundancy-elimination intent consistently, but this is called out as a discrete task so reviewers can confirm nothing downstream relies on the nightly artifact.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Introduce a shared coverage toggle and wire it into package configs

This phase centralizes the on/off decision for coverage instrumentation behind one env-driven helper and replaces every hardcoded coverage.enabled: true with a call to it. Coverage stays on by default (so local vitest run is unaffected) and is only suppressed when the toggle env var is explicitly set to disable it.

Task 1: Create the shared coverage-enabled helper

Provide a single source of truth for whether coverage instrumentation should run.

• Add a small root-level module (e.g., vitest.shared.ts or under scripts/) exporting a function/const that returns process.env.LLXPRT_COVERAGE !== 'false', following the existing === 'true' / !== 'false' env-toggle idiom already used elsewhere in the repo.
• Default behavior must be "enabled" when the var is unset, preserving current local-run coverage signal.
• Keep it dependency-free (only reads process.env) so it is safely importable from every standalone vitest.config.ts.

Task 2: Replace hardcoded coverage.enabled in all 11 package configs

Drive each package's coverage block from the shared helper instead of a literal true.

• Update the coverage.enabled value in each of: packages/agents, auth, cli, core, ide-integration, mcp, providers, settings, storage, telemetry, tools vitest.config.ts to import and call the shared helper.
• Leave all other coverage options (provider, reportsDirectory, include, OS-based coverageReporter, etc.) exactly as they are.
• Critically, do NOT change the second enabled: true in core, mcp, and tools — those belong to the typecheck block and are unrelated to coverage; only the coverage.enabled site in each of those packages should be touched.
• Ensure the relative import path to the shared helper is correct from each package directory.

🤖 Prompt for AI agents

Goal: centralize the coverage on/off decision behind one shared env-driven
helper and wire it into every package's Vitest config, keeping default behavior
unchanged (coverage on) for local runs.

- Create a small root-level module (e.g., `vitest.shared.ts` or under
`scripts/`) that exports a function/const returning `process.env.LLXPRT_COVERAGE
!== 'false'`, following the existing `=== 'true'` / `!== 'false'` env-toggle
idiom used elsewhere in the repo. It must default to enabled when the var is
unset, and must be dependency-free (only reads `process.env`) so it is safely
importable from every standalone `vitest.config.ts`.
- Update `coverage.enabled` in each of the following packages'
`vitest.config.ts` to import and call this shared helper instead of the
hardcoded `true`: `packages/agents`, `auth`, `cli`, `core`, `ide-integration`,
`mcp`, `providers`, `settings`, `storage`, `telemetry`, `tools`.
- Leave all other coverage options (`provider`, `reportsDirectory`, `include`,
OS-based `coverageReporter`, etc.) exactly as they are — only the
`coverage.enabled` value changes.
- Do NOT touch the separate `enabled: true` inside the `typecheck` block present
in `core`, `mcp`, and `tools` — that is unrelated to coverage and must remain
untouched.
- Verify the relative import path to the shared helper resolves correctly from
each package directory.

Phase 2: Compute coverage on one CI leg and remove redundant uploads

This phase makes CI set the disabling env var on every leg except one designated leg, and restricts the coverage-artifact upload to that same leg, so only one set of coverage reports is produced and uploaded — matching what the consumer already downloads.

Task 1: Set LLXPRT_COVERAGE per matrix leg in the test job

Enable coverage only on the designated leg consumed by the coverage comment.

• In .github/workflows/ci.yml, extend the existing step-level env: block on the "Run tests and generate reports" step (alongside LLXPRT_SECURE_STORE_FORCE_FALLBACK) to add LLXPRT_COVERAGE, computed with the established GitHub Actions ternary idiom so it is enabled only on the designated leg (ubuntu-latest + keyring, the leg post_coverage_comment downloads) and set to 'false' on all other legs.
• Keep the designation aligned with the sibling matrix-reduction issue so that after secure-store-mode is removed the designated leg remains valid.

Task 2: Restrict the coverage artifact upload to the designated leg

Stop uploading coverage artifacts that no downstream job consumes.

• Update the "Upload coverage reports" step's if: condition in .github/workflows/ci.yml to run only on the designated leg (matching Task 2.1), rather than unconditionally on always().
• Confirm the artifact name for the designated leg still matches exactly what post_coverage_comment downloads (coverage-reports-24.x-ubuntu-latest-keyring), so the consumer continues to find its artifact.

Task 3: Verify the coverage comment consumer still functions

Ensure coverage visibility is preserved.

• Confirm post_coverage_comment in .github/workflows/ci.yml still references the surviving artifact and that the composite action .github/actions/post-coverage-comment/action.yml receives valid coverage-summary.json / full-text-summary.txt inputs from the designated leg.
• No code change expected here unless the designated leg's naming is adjusted; validate rather than modify.

Task 4: Mirror the toggle in nightly.yml and drop its unused coverage upload

Keep nightly consistent and avoid computing coverage that is never consumed.

• In .github/workflows/nightly.yml's windows_ci job, add LLXPRT_COVERAGE=false to the npm run test step's env (nightly has no post_coverage_comment consumer).
• Remove or disable the corresponding coverage-artifact upload in that job since it would otherwise upload empty/absent coverage; confirm no other nightly job depends on it before removal.

🤖 Prompt for AI agents

Goal: make CI compute coverage on exactly one designated leg and remove the
now-redundant per-leg coverage artifact uploads, without weakening the coverage
comment or any threshold.

- In `.github/workflows/ci.yml`, extend the step-level `env:` block on the "Run
tests and generate reports" step (alongside the existing
`LLXPRT_SECURE_STORE_FORCE_FALLBACK`) to add `LLXPRT_COVERAGE`, computed via the
established GitHub Actions ternary idiom based on matrix values so it evaluates
to enabled only on the designated leg (`ubuntu-latest` + `keyring`, the leg that
`post_coverage_comment` downloads) and to `'false'` on all other legs. Keep the
designation valid even after any related matrix-reduction changes (e.g., removal
of `secure-store-mode`).
- Update the "Upload coverage reports" step's `if:` condition in
`.github/workflows/ci.yml` so it only runs on the designated leg (matching the
env toggle), instead of unconditionally via `always()`.
- Confirm the artifact name for the designated leg still exactly matches what
`post_coverage_comment` downloads
(`coverage-reports-24.x-ubuntu-latest-keyring`).
- Validate (without necessarily modifying) that `post_coverage_comment` in
`.github/workflows/ci.yml` still references the surviving artifact correctly,
and that the composite action `.github/actions/post-coverage-comment/action.yml`
still receives valid `coverage-summary.json` / `full-text-summary.txt` inputs
from the designated leg.
- In `.github/workflows/nightly.yml`'s `windows_ci` job, add
`LLXPRT_COVERAGE=false` to the `npm run test` step's env, since nightly has no
`post_coverage_comment` consumer. Remove or disable the corresponding
coverage-artifact upload step in that job, after confirming no other nightly job
depends on it.

Phase 3: Verification and timing capture

Confirm behavior is unchanged aside from the intended redundancy reduction, and record the required before/after evidence.

Validate test parity and record timings for the largest package

Satisfy the acceptance criterion for recorded before/after timings and unchanged test counts.

• Run packages/cli with coverage on vs. off (e.g., npx vitest run --coverage.enabled=false --reporter=dot) and confirm identical test counts with reduced duration.
• Record the before/after timings in the PR description.
• Confirm on the PR that the coverage comment still posts; since no enforced threshold/gate currently exists, verify none is inadvertently introduced or removed.

🤖 Prompt for AI agents

Goal: confirm the coverage toggle changes do not alter test behavior and capture
required before/after evidence.

- Run `packages/cli` (the largest package) with coverage on vs. off, e.g. using
`npx vitest run --coverage.enabled=false --reporter=dot`, and confirm identical
test counts between the two runs with a reduced duration when coverage is off.
- Record the before/after timings in the PR description.
- Confirm the coverage comment still posts correctly on the PR, and verify that
no enforced coverage threshold/gate is inadvertently introduced or removed,
since none currently exists (reporting remains informational only).

Research

This is a Bun/npm-workspaces monorepo where each packages/* maintains its own fully standalone vitest.config.ts. There is no root or shared Vitest config; the coverage block (coverage: { enabled: true, provider: 'v8', ... }) is copy-pasted across 11 packages, with core, mcp, and tools also carrying an unrelated typecheck: { enabled: true } block that must be left untouched.

CI (.github/workflows/ci.yml) runs a test job across an os × secure-store-mode matrix (4 legs), invoking npm run test (which fans out to per-package vitest run). A step-level env: block already computes per-leg env vars from matrix values (e.g., LLXPRT_SECURE_STORE_FORCE_FALLBACK). Every leg uploads a coverage-reports-* artifact, but only post_coverage_comment consumes exactly one (24.x-ubuntu-latest-keyring); no enforced coverage threshold/gate exists (reporting is informational).

nightly.yml's windows_ci job mirrors the same npm run test invocation and coverage-artifact upload pattern.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Goal: centralize the coverage on/off decision behind one shared env-driven
helper and wire it into every package's Vitest config, keeping default behavior
unchanged (coverage on) for local runs.

- Create a small root-level module (e.g., `vitest.shared.ts` or under
`scripts/`) that exports a function/const returning `process.env.LLXPRT_COVERAGE
!== 'false'`, following the existing `=== 'true'` / `!== 'false'` env-toggle
idiom used elsewhere in the repo. It must default to enabled when the var is
unset, and must be dependency-free (only reads `process.env`) so it is safely
importable from every standalone `vitest.config.ts`.
- Update `coverage.enabled` in each of the following packages'
`vitest.config.ts` to import and call this shared helper instead of the
hardcoded `true`: `packages/agents`, `auth`, `cli`, `core`, `ide-integration`,
`mcp`, `providers`, `settings`, `storage`, `telemetry`, `tools`.
- Leave all other coverage options (`provider`, `reportsDirectory`, `include`,
OS-based `coverageReporter`, etc.) exactly as they are — only the
`coverage.enabled` value changes.
- Do NOT touch the separate `enabled: true` inside the `typecheck` block present
in `core`, `mcp`, and `tools` — that is unrelated to coverage and must remain
untouched.
- Verify the relative import path to the shared helper resolves correctly from
each package directory.
===============================================================================

Task: 2

Goal: make CI compute coverage on exactly one designated leg and remove the
now-redundant per-leg coverage artifact uploads, without weakening the coverage
comment or any threshold.

- In `.github/workflows/ci.yml`, extend the step-level `env:` block on the "Run
tests and generate reports" step (alongside the existing
`LLXPRT_SECURE_STORE_FORCE_FALLBACK`) to add `LLXPRT_COVERAGE`, computed via the
established GitHub Actions ternary idiom based on matrix values so it evaluates
to enabled only on the designated leg (`ubuntu-latest` + `keyring`, the leg that
`post_coverage_comment` downloads) and to `'false'` on all other legs. Keep the
designation valid even after any related matrix-reduction changes (e.g., removal
of `secure-store-mode`).
- Update the "Upload coverage reports" step's `if:` condition in
`.github/workflows/ci.yml` so it only runs on the designated leg (matching the
env toggle), instead of unconditionally via `always()`.
- Confirm the artifact name for the designated leg still exactly matches what
`post_coverage_comment` downloads
(`coverage-reports-24.x-ubuntu-latest-keyring`).
- Validate (without necessarily modifying) that `post_coverage_comment` in
`.github/workflows/ci.yml` still references the surviving artifact correctly,
and that the composite action `.github/actions/post-coverage-comment/action.yml`
still receives valid `coverage-summary.json` / `full-text-summary.txt` inputs
from the designated leg.
- In `.github/workflows/nightly.yml`'s `windows_ci` job, add
`LLXPRT_COVERAGE=false` to the `npm run test` step's env, since nightly has no
`post_coverage_comment` consumer. Remove or disable the corresponding
coverage-artifact upload step in that job, after confirming no other nightly job
depends on it.
===============================================================================

Task: 3

Goal: confirm the coverage toggle changes do not alter test behavior and capture
required before/after evidence.

- Run `packages/cli` (the largest package) with coverage on vs. off, e.g. using
`npx vitest run --coverage.enabled=false --reporter=dot`, and confirm identical
test counts between the two runs with a reduced duration when coverage is off.
- Record the before/after timings in the PR description.
- Confirm the coverage comment still posts correctly on the PR, and verify that
no enforced coverage threshold/gate is inadvertently introduced or removed,
since none currently exists (reporting remains informational only).

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!