Skip to content

Commit 44aa9e2

Browse files
committed
Verify mutation-testing result is reproducible, not stale
1 parent b35eb47 commit 44aa9e2

1 file changed

Lines changed: 26 additions & 9 deletions

File tree

docs/architecture.md

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -527,19 +527,36 @@ alone. Filtered down to the fast, in-memory rule/algorithm fixture tests
527527
(`MissingOverrideRule`, `RedundantMoveReturnRule`, `UseAfterMoveRule`,
528528
`UnbalancedLockRule`, `UncheckedNullResultRule`, `ExpensiveCopyRule`,
529529
`LockOrderInversionRule`, `SCC`, `WorklistDataflowEngine`), a real run
530-
found 69 mutants (mull only generates mutants for lines the running tests
531-
actually cover) and killed 27 of them (39%). Of the 42 survivors, 24 are
530+
found 69 mutants and killed 27 of them (39%). Of the 42 survivors, 24 are
532531
in files whose *own* dedicated tests were excluded from this run for the
533532
memory reason above (`AnalysisDriver.cpp`: 11, `SuppressionState.cpp`: 5,
534533
`SarifFormatter.cpp`: 3, `Baseline.cpp`: 2, plus one each in
535534
`IncrementalCache.cpp`/`RuleFrontendAction.cpp`/`ToolchainDefaults.cpp`)
536-
-- not a real gap, just this run's scope. The other 18, in files whose
537-
tests *were* included, are genuine: e.g. `LockOrderInversionRule.cpp`'s
538-
cycle-reconstruction loop and its `component.size() > 1` cycle/non-cycle
539-
boundary (8 survivors total), `UseAfterMoveRule.cpp`'s assignment-operator
540-
detection (3), and `SCC.cpp`'s shortest-path tie-breaking (2) -- concrete,
541-
file-and-line leads for future fixture work, not chased down further in
542-
this session.
535+
-- not a real gap, just this run's scope: none of the 34 selected tests
536+
call into those files at all, so no mutant there could ever be killed by
537+
this run regardless of how well-tested they are elsewhere. The other 18,
538+
in files whose tests *were* included, are genuine: e.g.
539+
`LockOrderInversionRule.cpp`'s cycle-reconstruction loop and its
540+
`component.size() > 1` cycle/non-cycle boundary (8 survivors total),
541+
`UseAfterMoveRule.cpp`'s assignment-operator detection (3), and
542+
`SCC.cpp`'s shortest-path tie-breaking (2) -- concrete, file-and-line
543+
leads for future fixture work, not chased down further in this session.
544+
545+
**This exact result (69/27/42/39%, identical file-by-file breakdown) was
546+
independently reproduced on a second, completely fresh VM and build
547+
directory**, with the `--gtest_filter` applied from the very first mull
548+
invocation rather than after an earlier unfiltered attempt -- ruling out
549+
the otherwise-plausible concern that the first run's numbers were an
550+
artifact of coverage state left over from the unfiltered `--dry-run` that
551+
OOM'd before the CMake target scoping fix. `--debug-coverage` didn't
552+
produce clean enough output in this environment (missing shared-library
553+
warnings even after pointing `--ld-search-path` at the right directory)
554+
to pin down the *exact* mechanism by which mull decides a line in an
555+
excluded-tests file like `AnalysisDriver.cpp` is mutation-worthy at all;
556+
what's verified is that the result is deterministic and reproducible
557+
across independent runs, not session-contaminated -- the number is real,
558+
even without a fully traced explanation of mull's internal coverage
559+
bookkeeping.
543560

544561
## Build
545562

0 commit comments

Comments
 (0)