Skip to content

Automatically ratchet up coverage thresholds instead of hand-maintaining them #1453

Description

@YuryShkoda

Problem

jest.config.js hardcodes coverageThreshold.global as fixed numbers:

coverageThreshold: {
  global: {
    statements: 73,
    branches: 60.55,
    functions: 73.54,
    lines: 74
  }
}

These only get updated when someone notices npm run test:mocked failing (threshold not met) and manually edits the numbers — usually while already deep in an unrelated PR. This has two failure modes:

  • Silent stagnation: when coverage improves, nothing bumps the threshold up, so the safety net stays looser than it could be indefinitely.
  • Noisy, unrelated diffs: when coverage regresses, whoever's PR happens to trip the threshold has to manually figure out the right new number and edit CI-adjacent config as a side quest to their actual change.

We want this to self-maintain, the same way semantic-release (already used in this repo's npmpublish.yml) self-maintains the package version — no human edits jest.config.js by hand for this again.

Goal

Coverage thresholds should only ever move in the safe direction (up when real coverage improves) automatically, and a genuine regression should still fail CI clearly - never be silently masked.

Two viable approaches — needs a decision before implementation

Option A: custom script + CI auto-commit (self-hosted, more code to own)

  • A script (e.g. scripts/updateCoverageThresholds.js) runs after npm run test:mocked and reads coverage/coverage-summary.json.
  • For each of the four metrics, if actual coverage > current jest.config.js threshold, rewrite jest.config.js with the new value rounded down slightly (e.g. -0.5pt buffer) rather than an exact match, so a trivial/flaky variance between runs doesn't immediately fail the next build.
  • Never auto-lowers a threshold — only ratchets up. A real regression still fails the existing test:mocked step exactly as it does today, forcing a human decision (fix the coverage, or consciously accept + explain the drop in the PR).
  • Runs as a step in a workflow triggered on push to main (mirroring npmpublish.yml's trigger) — after a PR has already merged, not as part of PR checks — and pushes a bot commit like chore: bump coverage thresholds directly to main, the same way semantic-release pushes its version-bump commit.
  • Needs: write access from the Action to main (existing GITHUB_TOKEN with contents:write, or a PAT if branch protection requires it), and a guard so the bot's own commit doesn't re-trigger itself in a loop.

Option B: external coverage service (Codecov / Coveralls) - likely less code to build and maintain

  • These already solve exactly this class of problem: instead of a static number in the repo, they compare each PR's coverage against the base branch and post a status check like "coverage decreased by X% - failing" or "coverage increased - no threshold to maintain at all."
  • No jest.config.js numbers to bump, no bot-commit workflow to build/maintain, PR-level annotations showing exactly which lines lost coverage, and a badge for the README as a bonus.
  • Trade-off: adds a third-party dependency (an account + a small config file, e.g. codecov.yml), and CI needs to upload the coverage report (coverage/lcov.info) to the service each run.
  • This repo already uses artiomtr/jest-coverage-report-action in run-tests.yml for PR coverage comments - worth checking whether it (or a config tweak to it) already covers this need before adding another service.

Suggested next step

  • Decide between Option A and Option B (see trade-offs above) - or confirm neither and reconsider artiomtr/jest-coverage-report-action's existing capabilities first, since it's already wired into run-tests.yml.
  • If Option A: write scripts/updateCoverageThresholds.js, wire it into a new (or existing, post-merge) workflow step, decide the exact buffer value, and add the loop-prevention guard for the bot's own commits.
  • If Option B: pick a service, add the upload step to run-tests.yml, configure its ratcheting/regression rules, and remove the now-redundant coverageThreshold block from jest.config.js.
  • Either way: document the chosen mechanism (e.g. in CONTRIBUTING.md or a code comment near coverageThreshold) so contributors know they should never hand-edit the threshold numbers going forward.

Non-negotiable

Whichever option is picked, a genuine coverage regression must still fail CI loudly - this issue is about removing manual upkeep of the good direction (raising the bar), not about loosening enforcement.

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions