Skip to content

architecture.md

Latest commit

 

History

History
607 lines (544 loc) · 33.5 KB

architecture.md

File metadata and controls

607 lines (544 loc) · 33.5 KB
Raw

cpp-sentinel Architecture

Pipeline overview

compile_commands.json
        |
CompilationDatabaseLoader        (JSONCompilationDatabase / loadFromDirectory)
        |
AnalysisDriver + llvm::DefaultThreadPool
        |  one task per TU, each with its own fresh rule instances -- see
        |  "Parallel translation-unit analysis"
        |  IncrementalCache hit? -> skip straight to Rule::restoreState()
        |  + cached diagnostics, no ClangTool at all -- see "Incremental
        |  analysis cache"
        |
   +----+-----------------------------+
   |                                  |
RuleFrontendAction -> ASTConsumer     |
   |  per TU (inside one task,       |
   |  only on a cache miss):         |
   |   - MatchFinder dispatches      |
   |     AST-matcher rules           |
   |     (missing-override,          |
   |      redundant-move-return,     |
   |      unchecked-null-result,     |
   |      expensive-copy)            |
   |   - runOnTranslationUnit()      |
   |     for CFG-based rules:        |
   |      - lock-order-inversion     |
   |        walks the AST directly,  |
   |        accumulating             |
   |        LockOrderFacts into its  |
   |        own LockFactStore        |
   |      - use-after-move and       |
   |        unbalanced-lock each     |
   |        build a clang::CFG per   |
   |        function and run a       |
   |        forward worklist         |
   |        dataflow fixed point     |
   |        over it                  |
   +----+-----------------------------+
        |
 every TU processed
        |
 Rule::finalize() (once per rule, after the whole run)
        |  lock-order-inversion only:
        |   - builds a DirectedGraph from accumulated facts
        |   - Tarjan's SCC finds cycles
        |   - findShortestCycle reconstructs the shortest one
        |   - diagnostics filtered through SuppressionState
        |     (read straight off disk, since per-TU ASTContexts
        |     are already gone by this point)
        |
 Diagnostic vector (all rules' output, merged)
        |
   +----+----+
 Text      JSON  (output/TextFormatter, output/JsonFormatter)

Why rules dispatch two different ways

Rule (include/sentinel/core/Rule.h) derives from clang::ast_matchers::MatchFinder::MatchCallback so an AST-matcher rule can register itself directly with finder.addMatcher(pattern, this) in registerMatchers() and receive matches via the inherited run() override -- no separate callback object needed. Rules that don't fit the match-and-report shape (only lock-order-inversion today) instead override runOnTranslationUnit() and walk the AST themselves; run() defaults to a no-op in the base class for these. One interface, no rule-kind branching anywhere in the driver.

Rule::finalize(std::vector<Diagnostic>&) is the one addition M1's original interface didn't anticipate: it runs once after every translation unit in the run has gone through registerMatchers()/runOnTranslationUnit(). Project-level rules need this because their real analysis (the lock-order graph, in this case) only makes sense once the full cross-TU picture is available -- there's no single TranslationUnitContext that spans multiple TUs. finalize() runs on AnalysisDriver's own "template" rule instances (the ones passed to its constructor), which accumulate facts from every parallel task via Rule::mergeFrom() -- see the next section for why that indirection exists.

Parallel translation-unit analysis

AnalysisDriver::run() (lib/core/AnalysisDriver.cpp) hands each translation unit to its own task on an llvm::DefaultThreadPool (llvm::hardware_concurrency(jobCount); jobCount == 0, the CLI default, means "all available hardware threads"). Each task:

  1. Builds its own fresh set of rule instances via RuleRegistry (the same rule ids AnalysisDriver was constructed with), not the driver's own instances.
  2. If an incremental cache is configured, checks it first (see below) -- a hit skips the rest of this list entirely. Otherwise runs its own clang::tooling::ClangTool against just that one file, with its own local std::vector<Diagnostic> sink.

Fresh instances per task, not shared ones, because an AST-matcher rule stashes its current TU's TranslationUnitContext in a member between registerMatchers() and run() (see Rule.h) -- sharing one instance across two threads processing two different TUs at once would race on that member. clang::tooling::CompilationDatabase is the one object every task reads concurrently; that's safe since compile-command lookup is read-only (the same pattern clangd relies on for concurrent parses).

Once a task finishes, AnalysisDriver merges its diagnostics into the final vector and calls Rule::mergeFrom() on its own template instance for every rule, passing that task's instance as the sibling to merge from -- default a no-op, so most rules need nothing here. Only lock-order-inversion overrides it (merging the task's LockFactStore into its own), because it's the only rule whose finalize() needs the full cross-TU picture: without the merge, a cross-file cycle spanning TUs handled by two different tasks would never be reassembled, since finalize() only runs on the template instance, and it would never have been given the other task's facts (LockOrderInversionProjectTest. CrossFileCycleIsDetectedWithMultipleJobs forces jobCount=4 over exactly two files specifically to prove this). Finally, the merged diagnostics are sorted by (file, line, column, rule id) before returning -- output would otherwise depend on however the thread pool happened to interleave tasks, which is a real, if easy to miss, determinism regression once a driver goes from sequential to parallel.

Incremental analysis cache

IncrementalCache (include/sentinel/core/IncrementalCache.h, lib/core/IncrementalCache.cpp) is deliberately Clang-agnostic, in the same spirit as dataflow::runForwardWorklist and graph/SCC.h: it knows about a "primary key" string, a list of {file, contentHash} dependencies, and a CacheEntry blob; it has no idea what a TU or an AST is. All the Clang-specific glue lives in AnalysisDriver.cpp.

Two problems make a single "content hash of the file" not enough:

  • A TU's result depends on more than the file's own text -- its compile flags, the enabled rule set, --header-filter, and the cpp-sentinel build itself can all change what gets reported without the file changing at all. These fold into the primary key (IncrementalCache::computeKey()), computable before parsing, and changing any of them invalidates every entry that depended on the old value (there's no partial invalidation across a key change -- simplest correct behavior, and a version/flag/rule-set change is rare next to a source edit).
  • The TU's result also depends on every header it transitively includes, and that set isn't knowable until after a real parse. So freshness for a given primary key is checked lazily: AnalysisDriver (collectDependencyFiles() in RuleFrontendAction.cpp, wired through an optional out-parameter on createRuleFrontendAction()) walks the TU's SourceManager::fileinfo_begin()/end() after a real analysis and records every real on-disk file it touched -- the main file plus every header -- each hashed via the same FNV-1a sentinel::hashContent() Fingerprint.cpp already used (extracted to ContentHash.h so both agree on one implementation). IncrementalCache::lookup() re-hashes every recorded dependency's current content on every call; any mismatch (or a deleted file) is a miss.

The trickiest part is lock-order-inversion again: its finalize() needs the full cross-TU picture, but a cache-hit TU is never re-parsed, so there's no fresh LockOrderFact list for it that run. Rule:: serializeState()/restoreState() close that gap generically -- a rule can serialize whatever per-TU state its finalize() needs into an opaque blob IncrementalCache stores alongside a TU's diagnostics, and reconstitute it on a later cache hit into a fresh per-task instance, which then flows into Rule::mergeFrom() exactly like a freshly-computed task's state would (see "Parallel translation-unit analysis" above) -- the cache doesn't need a special code path for this at all, because a cache hit and a cache miss both end up producing "a rule instance with some state," merged the same way either way. Default no-op, like mergeFrom(); only LockOrderInversionRule overrides it, serializing factStore_.facts() to a small JSON array. AnalysisDriverCacheTest.CrossFileCycleStillDetectedWithOneCachedAndOneChangedFile is the test that actually proves this: two files form a lock-order cycle, one is touched (forcing a miss) and one isn't (a hit), and the cycle must still be detected -- exactly the case a cache that only stored diagnostics (and not this per-rule state) would silently break.

A TU that fails to compile is never cached (AnalysisDriver.cpp): its dependency list is likely incomplete (Clang can bail out before touching every header a correct parse would), so caching it risks a stale entry surviving even after the real problem is fixed -- simplest and safest to just retry a compile failure in full on every run.

The flagship rule: lock-order-inversion

Three moving pieces, in lib/rules/LockOrderInversionRule.cpp:

  1. Per-function AST walk (FunctionLockWalker). For each function definition in a TU (collected via a small RecursiveASTVisitor), walks the body tracking currently-held locks in a vector tagged with a scope id. CompoundStmt/IfStmt branches/loop bodies each get their own scope id via enterScope()/exitScope(); RAII lock_guard/unique_lock acquisitions are released when their scope exits, while an explicit unlock() removes a lock immediately regardless of which scope acquired it. Whenever a lock is acquired while others are already held, a LockOrderFact{outerLock, innerLock, outerLoc, innerLoc} is recorded for every currently-held lock, not just the most recent -- consistent with the MVP spec's "emit an edge when B is acquired while A is held."

    This walks the AST directly rather than clang::CFG's basic-block graph -- a deliberate deviation from the original plan. It reaches the same "straight-line + simple branch, no loop fixed-point" semantics the MVP spec calls for, without needing to reassemble held-lock state across CFG block predecessors/successors, and is easier to verify correct at this scope.

  2. Lock identity is a plain string: the qualified name of the underlying FieldDecl/VarDecl (locals are further qualified by their enclosing function's name, to avoid an unrelated same-named local in a different function colliding). This is what makes cross-TU merging work at all -- Clang LibTooling parses each translation unit independently, with no cross-TU declaration merging, so two VarDecls for the same extern std::mutex in different TUs are different C++ objects that happen to produce the same identity string.

  3. Graph + cycle detection (include/sentinel/graph/). A small hand-rolled DirectedGraph plus a recursive Tarjan's SCC implementation. finalize() builds the graph from every fact accumulated across all TUs, and any strongly-connected component with more than one node (or a single node with a self-edge) is a candidate deadlock cycle. findShortestCycle then reconstructs a concrete cycle to report: anchored on the lexicographically smallest node in the SCC, it BFS's a shortest-path tree and picks whichever direct predecessor closes the shortest loop back to the anchor.

Known limitations (all deliberate MVP trims, not oversights): no alias analysis (two different pointers to the same mutex are different lock ids), loop bodies are walked exactly once rather than to a dataflow fixed point, std::unique_lock's defer_lock constructor form is treated as an immediate acquisition, and only lock_guard/unique_lock RAII wrappers are recognized (not scoped_lock).

The generic worklist dataflow engine and use-after-move

include/sentinel/dataflow/WorklistDataflowEngine.h is a small, generic forward "may" dataflow fixed-point solver: runForwardWorklist(nodes, entryIndex, analysis) takes a plain index/adjacency graph (Node -- deliberately not clang::CFGBlock itself, so the algorithm is unit-testable with a hand-built graph and no C++ parse, the same reasoning that keeps graph/SCC.h decoupled from any AST type -- see WorklistDataflowEngineTest.cpp) and an Analysis type supplying a Domain, join/bottom/entry, and a pure per-node transfer. It's the first rule in this codebase that needs a real fixed point rather than a single walk: lock-order-inversion visits each loop body once (a documented trim above), but a moved-from fact can flow into a loop body from the previous iteration via a back edge, so use-after-move needs the fixed point to catch that case correctly (see tests/fixtures/use-after-move/loop.cpp, and WorklistDataflowEngineTest.LoopReachesFixedPointThroughBackEdge for the engine-level version of the same scenario).

UseAfterMoveRule (lib/rules/UseAfterMoveRule.cpp) adapts a clang::CFG (one built per function via CFG::buildCFG) into that engine in two passes:

  1. Fixed-point pass: UseAfterMoveAnalysis::transfer() walks each block's statements with Domain = std::set<const VarDecl*> (the currently moved-from-and-not-reassigned variables), joined by set union at merge points. This must be pure -- called repeatedly during iteration, so it never reports diagnostics itself.
  2. Reporting pass: once IN[] is known correct for every block, the rule re-walks each block's statements exactly once more from its final IN[], this time actually reporting a diagnostic for every read of an already-moved-from variable.

Both passes share one transferStatement() helper (classify a statement's DeclRefExprs into kills/gens/uses via a small RecursiveASTVisitor) so the fixed-point and reporting logic can't drift apart. Known limitations are documented in include/sentinel/rules/UseAfterMoveRule.h; the one worth calling out here since it was caught by dogfooding rather than reasoned out ahead of time: a VarDecl's own declaration has to be treated as a kill of any prior fact tied to that same declaration, or a variable declared fresh inside a loop body (e.g. for (...) { Diagnostic diag; ...; report(std::move(diag)); }) looks moved-from on the next iteration's brand-new instance, once the fixed point carries that fact around the back edge -- a real false positive this rule produced against its own LockOrderInversionRule.cpp and SCC.cpp before the fix (see tests/fixtures/use-after-move/redeclared-each-iteration.cpp).

unbalanced-lock (lib/rules/UnbalancedLockRule.cpp) is the engine's second consumer -- proving out the "generic" in "generic worklist dataflow engine": same Node/Analysis contract, a different Domain (std::set<std::string> of currently-held lock ids, via explicit lock()/unlock() calls only -- RAII is balanced by construction, out of scope). It reuses lock-order-inversion's lock-identity resolution, pulled out into a shared sentinel::resolveLockId (include/sentinel/core/LockIdentity.h) so the two rules can't disagree about what "the same lock" means. Unlike use-after-move, there's no per-statement reporting pass: once the fixed point is reached, the rule just reads the CFG's single Exit block's IN[] directly -- any lock id still in it is held on at least one path that reaches the end of the function, and that's the whole diagnostic.

Reusing the engine for a second rule immediately surfaced a real bug in the first one: UnbalancedLockAnalysis::transfer() reassigns its running domain via in = transferStatement(..., std::move(in), ...) inside a loop -- in is both the move-argument and the assignment target in the same statement. use-after-move's transferStatement() applied kills before gens, so the reassignment (kill) was overwritten by the move (gen), leaving in permanently "moved" and producing a false positive on return in; after the loop. Fixed by applying gens before kills instead, so an assignment always has the last word on its own target within one statement (regression fixture: tests/fixtures/use-after-move/reassigned-from-consuming-call.cpp) -- exactly the kind of gap that only shows up once a second real caller exercises the shared code differently than the first one did.

Two more AST-matcher rules: unchecked-null-result, expensive-copy

Both pure MatchFinder-dispatched rules, no CFG, in the same style as missing-override/redundant-move-return:

  • unchecked-null-result matches dynamic_cast<T*>(x) immediately wrapped in a dereference (memberExpr(isArrow(), hasObjectExpression(...)) or unaryOperator(hasOperatorName("*"), hasUnaryOperand(...))), scoped to only the same-expression case -- storing the result and checking it (or not) later is a dataflow problem like use-after-move, deliberately not attempted here with a matcher-only heuristic that would get it wrong. hasDestinationType(pointerType()) is what excludes dynamic_cast<T&> (throws std::bad_cast on failure instead of returning null).
  • expensive-copy matches every functionDecl(isDefinition()), then walks the body once (ParamUsageVisitor) classifying every direct use of each by-value, non-trivially-copyable parameter as disqualifying (assignment target, address-of, non-const member call) or the std::move(...) "sink" idiom that exempts it entirely. A parameter with no disqualifying use and no move is flagged. Self-scanning found a real gap: a constructor's member-initializer list (Foo(T x) : x_(std::move(x)) {}) isn't part of FunctionDecl::getBody(), so the single most common place a by-value parameter is legitimately consumed was invisible to the walk -- every constructor in this codebase that stores a parameter by move (RuleActionFactory, RuleConsumer, RuleAction, AnalysisDriver, UnbalancedLockAnalysis) was a false positive until CXXConstructorDecl::inits() was walked too (regression fixture: tests/fixtures/expensive-copy/constructor-init-list-negative.cpp).

Header filtering

TranslationUnitContext::report() also checks HeaderFilter (include/sentinel/core/HeaderFilter.h) before a diagnostic reaches the sink: by default, only a diagnostic in the TU's own main file is kept, and anything in a header is dropped unless it matches --header-filter (matching clang-tidy's well-established convention). Without this, any TU that transitively includes gtest/libc++/LLVM/etc. headers reports findings inside that third-party code -- caught concretely while dogfooding cpp-sentinel on its own codebase (583 findings dropped to a handful of real ones once this landed, almost entirely redundant-move-return hits inside standard-library headers). AnalysisDriver::run() threads a HeaderFilter down through RuleFrontendAction to TranslationUnitContext; note this only guards the per-TU AST-rule path, not lock-order-inversion's finalize()-time diagnostics.

Suppression

Two paths reach the same SuppressionState (include/sentinel/core/SuppressionState.h), which parses // cpp-sentinel-ignore <rule> (that line only) and // cpp-sentinel-disable <rule> / // cpp-sentinel-enable <rule> (an inclusive line range, open-ended if never re-enabled) out of a file's raw text:

  • Per-TU AST/CFG rules go through TranslationUnitContext::report(), which lazily reads and caches each file's SuppressionState via Clang's FileManager (so it works against the in-memory VFS runToolOnCodeWithArgs uses in fixture tests, not just real files).
  • lock-order-inversion's finalize() diagnostics apply suppression separately (isSuppressedOnDisk in LockOrderInversionRule.cpp), reading the file straight off disk via llvm::MemoryBuffer, since by the time finalize() runs every per-TU ASTContext/SourceManager is already gone.

Output and fingerprints

output/JsonFormatter.h builds on llvm::json (already a transitive dependency via LLVM) rather than adding a JSON library. Each diagnostic carries a stable fingerprint (core/Fingerprint.h): an FNV-1a hash over rule id + file + line + column. The schema (fingerprint, source location, severity, rule id, optional note/remediation/related locations) was kept deliberately close to SARIF's shape, and output/SarifFormatter.h proved that out: it's the same Diagnostic model serialized differently, plus a basePath prefix-strip so file paths come out repo-relative (GitHub's Security tab expects that, not the absolute paths a real compile_commands.json carries) and a rule catalog (every registered rule, not just ones that fired, per SARIF convention) for tool.driver.rules.

Baseline comparison

core/Baseline.h reuses computeFingerprint() rather than inventing a second identity scheme: a baseline file (--update-baseline) is just a sorted JSON list of {fingerprint, ruleId, file, line, column} records (the non-fingerprint fields exist purely so the file reads sensibly in a git diff, not because they're used for matching). Comparing (--baseline without --update-baseline) loads that file into a std::set<std::string> and calls applyBaseline(), which sets Diagnostic::baselineKnown in place for every diagnostic whose fingerprint is already in the set -- the same flag each output formatter already knows how to render ((baseline) in text, "baselineKnown" in JSON, SARIF's own "baselineState": "unchanged" in SARIF) and that main.cpp's --fail-on check excludes. countFixed() is the mirror query: baseline fingerprints with no matching current diagnostic, i.e. findings that no longer reproduce. Because everything is keyed off the same fingerprint used elsewhere, this cost almost nothing new to build -- it's a diff over a set the tool was already computing.

Differential and property-based testing

tests/unit/DifferentialTest.cpp runs the same tests/projects/multi-rule fixture (a copy of the demo project, seven rules across two translation units, one cross-file lock cycle) through AnalysisDriver under different execution conditions and asserts the resulting diagnostics are identical down to every field (not just the fingerprint -- two runs producing the same fingerprint with a different message would still be a real regression). Differential.OutputIsIdenticalAcrossJobCounts compares --jobs 1/2/4/8; Differential.OutputIsIdenticalWithAndWithoutCache compares no-cache, cold-cache, and warm-cache runs. Both formalize determinism checks that M8/M9 originally validated by hand (diff-ing output across runs) into permanent CI regression tests.

The same file also has PropertyBased.EachRulesFindingsAreIndependentOfWhichOtherRulesAreEnabled: a property checked by enumeration over the fixed, small set of registered rules rather than a generative QuickCheck-style search (there's no sentinel::Rule generator to speak of), but the invariant itself is real -- a rule's findings must not depend on which other rules happen to be enabled alongside it. It runs every rule together, then each rule alone, and asserts each rule's own subset of findings matches exactly.

Fuzzing

tools/fuzz/sentinel-fuzz.cpp is a libFuzzer harness (LLVMFuzzerTestOneInput) that feeds arbitrary bytes to createRuleFrontendAction() with every rule enabled, via the same clang::tooling::runToolOnCodeWithArgs in-memory path the rule unit tests use. Almost every input fails to parse as valid C++; the point isn't reaching deep semantic analysis, it's that a parse failure, however malformed the input, must never crash or hang cpp-sentinel itself. Not part of the default build or CI -- -fsanitize=fuzzer needs a libFuzzer runtime, which Homebrew's LLVM ships but Apple's system clang (the compiler the normal build/ tree uses) doesn't. Build it in a separate directory with the matching compiler:

cmake -S . -B build-fuzz -DSENTINEL_ENABLE_FUZZING=ON -DSENTINEL_BUILD_TESTS=OFF \
    -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm/bin/clang++ \
    -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm/bin/clang
cmake --build build-fuzz -j"$(sysctl -n hw.ncpu)"
./build-fuzz/tools/fuzz/sentinel-fuzz -max_total_time=180 tests/fixtures tests/projects

SENTINEL_ENABLE_FUZZING recompiles the whole project (not just the harness file) under ASan plus libFuzzer's coverage instrumentation, via top-level add_compile_options -- otherwise only the one-file entry point would be instrumented, and bugs in sentinel-lib itself (the actual code under test) wouldn't be caught. A 3-minute local session seeded from tests/fixtures/tests/projects reached 29,060 executions and ~4,800 coverage edges with zero crashes or ASan reports -- a clean baseline data point, not proof of absence of bugs.

Mutation testing

Validates that the test suite would actually notice if the rule/algorithm logic broke, using mull, an LLVM-pass-based mutation-testing tool: it recompiles the code with small, mechanical semantic changes (< to <=, == to !=, ++ to --, ...) one at a time and reruns the tests, expecting each mutant to make some test fail ("killed"). A mutant that "survives" (every test still passes) means either an equivalent no-observable-difference change, or a real gap where that exact behavior isn't actually asserted anywhere.

Getting this working surfaced a real, if narrow, cross-platform bug, independent of mull itself: LockOrderInversionRule.cpp and UnbalancedLockRule.cpp both called CXXMethodDecl::getName() on every CXXMemberCallExpr's callee to check for "lock"/"unlock" -- getName() asserts unless the method has a plain identifier name, and an arbitrary member call walked out of a real <mutex>-family header can resolve to a constructor, destructor, operator, or conversion function, none of which qualify. Invisible in a normal Release build (NDEBUG disables the assert; the wrong codepath just silently fails the name != "lock" check instead of crashing), but a hard abort the moment assertions are compiled in -- which mull's build requires (see below). Reproduced locally with a plain -DCMAKE_BUILD_TYPE=Debug build (no mull needed to hit it) and fixed by guarding with method->getIdentifier() before calling getName() in both files.

Why this needs a separate machine, and why the version has to match exactly: mull instruments code via an LLVM pass plugin (-fpass-plugin=/usr/lib/mull-ir-frontend-N), and a pass plugin's ABI is only compatible with the exact Clang build that loads it -- not just the same version number, but the same linkage. The first attempt, using an official llvm.org release tarball's clang++ (statically links its own copy of LLVM), failed to even load the plugin (CommandLine Error: Option 'sanitizer-early-opt-ep' registered more than once! -- two independently-linked copies of LLVM's global command-line registry colliding in one process). mull's own packages are built against a Linux distro's dynamically-linked libclang-cpp.so/libLLVM.so, so the fix was switching to Ubuntu 26.04's own apt-packaged clang-22 (1:22.1.2-1ubuntu1, exactly matching Mull-22-0.34.0-LLVM-22.1.2's build), which shares that same shared library the plugin also expects. There is no macOS build of mull for LLVM >= 21 at all, and this project has used LLVM 22 (via Homebrew) since M1 -- so this only runs on a throwaway Linux VM, not as part of local dev or CI. SENTINEL_ENABLE_MUTATION_TESTING (lib/CMakeLists.txt) instruments only the sentinel-lib target (not googletest/yaml-cpp/the test binary itself) for exactly this reason: an early attempt applying -fpass-plugin globally instrumented every dependency too, and mull's coverage bookkeeping across that much irrelevant code OOM'd an 8-vCPU/32GB VM outright. mull.yml's includePaths is a second, belt-and-suspenders filter on top of that.

To reproduce (Ubuntu 26.04 amd64, apt install clang-22 libclang-22-dev llvm-22-dev, plus the matching Mull-22-*-ubuntu-amd64-26.04.deb from mull's GitHub releases):

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug \
    -DCMAKE_CXX_COMPILER=/usr/bin/clang++-22 -DCMAKE_C_COMPILER=/usr/bin/clang-22 \
    -DCMAKE_PREFIX_PATH=/usr/lib/llvm-22 \
    -DSENTINEL_ENABLE_MUTATION_TESTING=ON -DMULL_IR_FRONTEND=/usr/lib/mull-ir-frontend-22
ninja -C build
mull-runner-22 build/tests/sentinel-unit-tests --gtest_filter="<fast fixture-test suites>"

The --gtest_filter matters beyond speed: mull reruns the entire selected test binary once per mutant, and the handful of tests that spin up a real ClangTool over real system headers (AnalysisDriverCacheTest, BaselineTest, SuppressionStateTest, DifferentialTest, LockOrderInversionProjectTest) are heavy enough that including them OOM'd the same VM even after scoping instrumentation to sentinel-lib alone. Filtered down to the fast, in-memory rule/algorithm fixture tests (MissingOverrideRule, RedundantMoveReturnRule, UseAfterMoveRule, UnbalancedLockRule, UncheckedNullResultRule, ExpensiveCopyRule, LockOrderInversionRule, SCC, WorklistDataflowEngine), a real run found 69 mutants and killed 27 of them (39%). Of the 42 survivors, 24 are in files whose own dedicated tests were excluded from this run for the memory reason above (AnalysisDriver.cpp: 11, SuppressionState.cpp: 5, SarifFormatter.cpp: 3, Baseline.cpp: 2, plus one each in IncrementalCache.cpp/RuleFrontendAction.cpp/ToolchainDefaults.cpp) -- not a real gap, just this run's scope: none of the 34 selected tests call into those files at all, so no mutant there could ever be killed by this run regardless of how well-tested they are elsewhere. The other 18, in files whose tests were included, are genuine: e.g. LockOrderInversionRule.cpp's cycle-reconstruction loop and its component.size() > 1 cycle/non-cycle boundary (8 survivors total), UseAfterMoveRule.cpp's assignment-operator detection (3), and SCC.cpp's shortest-path tie-breaking (2) -- concrete, file-and-line leads for future fixture work, not chased down further in this session.

This exact result (69/27/42/39%, identical file-by-file breakdown) was independently reproduced on a second, completely fresh VM and build directory, with the --gtest_filter applied from the very first mull invocation rather than after an earlier unfiltered attempt -- ruling out the otherwise-plausible concern that the first run's numbers were an artifact of coverage state left over from the unfiltered --dry-run that OOM'd before the CMake target scoping fix. --debug-coverage didn't produce clean enough output in this environment (missing shared-library warnings even after pointing --ld-search-path at the right directory) to pin down the exact mechanism by which mull decides a line in an excluded-tests file like AnalysisDriver.cpp is mutation-worthy at all; what's verified is that the result is deterministic and reproducible across independent runs, not session-contaminated -- the number is real, even without a fully traced explanation of mull's internal coverage bookkeeping.

Synthetic benchmark corpus

benchmarks/synthetic-corpus/ exists because the spdlog benchmark, while real, is an external dependency at a pinned commit -- reproducing it means cloning something else first. Nothing about the corpus itself is committed: CMakeLists.txt generates SENTINEL_BENCHMARK_CORPUS_SIZE (default 30) unit pairs via configure_file() from three templates, substituting @UNIT@ for a unique integer per unit. Each unit reproduces the same seven-rule bug pattern as tests/projects/multi-rule; units are never linked together (each is only ever an OBJECT library source, no executable), so there's no cross-unit ODR concern and no reason to make each unit's globals more unique than "don't collide within the same pair." This makes the corpus's total finding count exactly predictable (SENTINEL_BENCHMARK_CORPUS_SIZE * 7) -- a correctness check on any benchmark run over it, not just a timing number. See the top-level README's Benchmark section for the cold/warm cache numbers measured against it.

Build

CMake links against a single Homebrew/apt-installed LLVM/Clang (the clang-cpp combined shared library plus the LLVM combined shared library), rather than enumerating individual libClang*/libLLVM* component libraries. sentinel-lib (lib/CMakeLists.txt) is a CMake OBJECT library, not STATIC: rules self-register via a static initializer (RuleRegistration<T>) that nothing else references directly, and a STATIC archive would let the linker drop that "unreferenced" object file -- an OBJECT library's .os are always included whole in whatever links it. GoogleTest and yaml-cpp are pulled in via FetchContent rather than requiring system packages.

CI matrix

.github/workflows/ci.yml runs build-and-test as a 2x2 matrix: llvm_version (21, 22) x sanitizer (none, asan-ubsan). The asan-ubsan leg configures CMAKE_BUILD_TYPE=Debug rather than Release deliberately, not just to enable -fsanitize=address,undefined -- Release defines NDEBUG, which silently disables assert(), and the getName() bug documented under "Mutation testing" above was only ever visible under a Debug build, independent of any sanitizer. Running Debug continuously (not just when someone happens to need it locally, as happened with that bug) catches that whole class of issue automatically going forward. fail-fast: false so one leg's failure doesn't hide results from the other three. clang-format check runs on only one leg (llvm_version == 22 && sanitizer == 'none') since the source doesn't change across the matrix -- running it four times would just burn CI minutes for the same answer.