Repository navigation
ci(mutation): test only the changed mutants on pull requests with Stryker's incremental mode - #613
Merged
Merged
Conversation
…yker's incremental mode The whole Stryker run took about 17 minutes on every pull request. Pull requests now run `stryker run --incremental`, restoring reports/stryker-incremental.json from their own last run or from main's, so only the mutants whose code or covering tests changed are tested. Files Stryker cannot track (constants, test helpers, the lockfile, the configs) are hashed into the cache key, so a change to one of them reruns every mutant. main keeps testing every mutant (--force) before publishing the score, and rebuilds the baseline the pull requests start from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLkm9YrtAifc6XCLFVEsdH
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
commit: |
Contributor
Tree-shaking report✅ No bundle size impact. All 185 exports are the same size as on the base branch (full import 1985.9 KB, gzip 378.2 KB). All exports (185)
How this is measuredEvery export is imported alone into an esbuild consumer bundle (minified, tree-shaken) built from the head and from the base of this pull request; the sizes are the resulting bundles, gzip is their gzipped size. 🔴 marks a regression: a pre-existing export that grew more than 20% and more than 256 B, or the bundle importing every pre-existing export growing more than 5%. 🟡 is growth under the threshold, 🟢 a decrease, ⚪ no change, 🆕 an export that does not exist on the base (never a regression), 🗑️ an export that was removed. An intentional increase is accepted with the |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #613 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 218 218
Lines 2252 2252
Branches 673 673
=========================================
Hits 2252 2252
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…file Stryker's incremental mode reuses a mutant's result when the mutant's own code and its covering tests did not change, without looking at the files that code imports: a change to `_internals/format` kept every `format*` mutant's old result (53 of 53 reused on a local run). `scripts/prune-stryker-incremental.ts` compares every mutated file with the source saved in the report, follows the relative imports back to every file that depends on a changed one, and drops those files from the report, so Stryker tests all of their mutants again (0 of 53 reused). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLkm9YrtAifc6XCLFVEsdH
…at reaches a changed file A mutant in an unchanged helper, killed by the tests of a function that changed, kept its "killed" result even when the function no longer calls the helper: the killing test's source did not change. The prune script now also drops every mutant killed by a test in a test file that imports a changed file (or, for a mutant with another status, covered by one). Only the killing test counts for a killed mutant, so `index.test.ts`, which imports every module and covers most mutants, sends back only the few it killed. Whole files are still dropped when they changed or import a changed file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLkm9YrtAifc6XCLFVEsdH
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The
Strykerjob took about 17 minutes on every pull request, because it tested all ~5,300 mutants every time. This PR turns on Stryker's incremental mode in CI.stryker run --incremental. The job restoresreports/stryker-incremental.jsonfrom the cache, trying the PR's own last run first and thenmain's. Only mutants whose results may have changed are tested again; every other result is reused.mainkeeps testing every mutant (--incremental --force) before publishing the score to the dashboard. That run also rebuilds the baseline the pull requests start from, so neither the score nor the baseline carries an old result.Safety: what Stryker can't see
Stryker reuses a mutant's result when the mutant's own code and the source of its covering tests did not change. It looks at nothing else, so three gaps are covered here:
src/**/constants.tsandsrc/_internals/constants/**src/_internals/test/**src/index.tspackage-lock.jsonstryker.config.jsonvite.config.tsscripts/prune-stryker-incremental.tscompares every mutated file with the source saved in the report. Every mutant of a file that changed, or that imports a changed file directly or through other files, is removed from the report and runs again. For example, a change to_internals/formatreruns everyformat*.format-cpftests would stay "killed" afterformat-cpfstops calling that helper, because the test's source did not change. The script also removes every mutant killed by a test in a test file that reaches a changed file. For a mutant with another status (timeout, survived), it removes the mutant if any such test covers it. For a killed mutant, only the killing test counts.index.test.tsimports every module and covers most mutants, so it sends back only the few it killed.The previous sources are already in the report, so no git history is needed.
Checklist
docs/utilities.mdanddocs/pt-br/utilities.md: not applicable.CONTRIBUTING.mddescribes the new behaviour and how to run it locally.npm run checkpasses locally (format, lint, types), andknipis clean.npm run build:llms:docs/untouched.Additional context
On CI:
The time left is mostly the initial run of the whole test suite (perTest coverage), which Stryker always does.
The prune, checked locally with real Stryker runs on
_internals/format+format-pis+format-cnae(53 mutants), after a change to_internals/formatonly:--incrementalalone53 of 53 mutant result(s) are reused: the helper change goes unseen--incremental0 of 53 mutant result(s) are reused: all rerun, score 100Against a report covering every mutated file:
_internals/formatdrops 27 whole files: the helper, everyformat*that uses it, and what imports those in turn (generate-pix-payload,get-pix-key-info,is-valid-pix-key,parse-cnpj). It also drops the other mutants killed by a test that reaches it.The first push to
mainafter merging seeds the baseline. The cache usesactions/cache/restoreandactions/cache/savev6.1.0, pinned by SHA like the other actions.🤖 Generated with Claude Code
https://claude.ai/code/session_01RLkm9YrtAifc6XCLFVEsdH
Generated by Claude Code