Add synthetic benchmark corpus + cold/warm cache numbers (M12)

Author: czhao-dev

README.md

@@ -266,10 +266,40 @@ project on a memory-constrained machine. All job counts produced
 identical findings (verified via `diff`), including on the demo project
 run directly on this VM.
 
-This is still closer to "two honest data points" than a real benchmark
-suite -- see [`docs/spec.md`](docs/spec.md) §14 for what a full suite
-(synthetic corpus, cold/warm, scaling *and* memory as first-class
-dimensions, regression gating) would still need to add.
+### Synthetic corpus: cold vs. warm cache
+
+The spdlog and scaling numbers above are real projects, but both are
+external dependencies -- reproducing them means cloning something else
+first. [`benchmarks/synthetic-corpus/`](benchmarks/synthetic-corpus/) is
+a self-contained alternative generated entirely from templates at CMake
+configure time: 30 independent unit pairs (60 TUs), each reproducing the
+same seven-rule bug pattern as `tests/projects/multi-rule`, for an exact,
+predictable 210 total findings (30 per rule) -- a built-in correctness
+check on the run, not just a timing number.
+
+Measured on the same 8-core Apple M3, `--jobs 1` (isolating the cache
+effect from parallelism), average of 3 runs each, fresh `--cache-dir` per
+cold run vs. one warm cache reused across all three warm runs:
+
+| | Wall clock | Peak RSS |
+|---|---|---|
+| Cold (no cache) | 12.68 s | ~171 MiB |
+| Warm (full cache hit, 60/60 TUs) | 0.87 s | ~34 MiB |
+
+~14.6x speedup. All six runs (3 cold + 3 warm) produced byte-for-byte
+identical findings (verified via `diff`), and every run reported exactly
+210 findings, confirming the corpus's own self-check. Cold-run peak RSS
+varied more than expected across repetitions (128-213 MiB) -- likely
+ordinary background load on a shared dev laptop rather than anything
+corpus-specific, included here rather than smoothed over.
+
+This is now three real, reproducible data points (one external project,
+one scaling study, one fully self-contained corpus) rather than "two
+honest data points" -- what's still missing from a complete benchmark
+suite is CI-enforced regression gating (fail a run if wall-clock or peak
+RSS regresses past some threshold), which folds naturally into the CI
+matrix work in [`docs/spec.md`](docs/spec.md) §14's next item rather than
+being its own benchmark-only milestone.
 
 ## Known limitations
 

benchmarks/synthetic-corpus/CMakeLists.txt

@@ -0,0 +1,32 @@
+cmake_minimum_required(VERSION 3.20)
+project(cpp-sentinel-synthetic-corpus LANGUAGES CXX)
+
+set(CMAKE_CXX_STANDARD 20)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
+# Number of independent unit pairs (2 TUs each) to generate. Every unit
+# reproduces the same seven-rule bug pattern as tests/projects/multi-rule
+# (mechanically duplicated via configure_file(), not committed N times),
+# so the corpus's total finding count is always exactly
+# SENTINEL_BENCHMARK_CORPUS_SIZE * 7 -- a built-in sanity check for any
+# benchmark number derived from this corpus.
+set(SENTINEL_BENCHMARK_CORPUS_SIZE 30 CACHE STRING
+  "Number of unit pairs (2 TUs each) to generate for the synthetic corpus")
+
+set(GENERATED_SOURCES "")
+foreach(UNIT RANGE 1 ${SENTINEL_BENCHMARK_CORPUS_SIZE})
+  configure_file(templates/geometry.h.in generated/geometry${UNIT}.h @ONLY)
+  configure_file(templates/shape.cpp.in generated/shape${UNIT}.cpp @ONLY)
+  configure_file(templates/logger.cpp.in generated/logger${UNIT}.cpp @ONLY)
+  list(APPEND GENERATED_SOURCES
+    ${CMAKE_CURRENT_BINARY_DIR}/generated/shape${UNIT}.cpp
+    ${CMAKE_CURRENT_BINARY_DIR}/generated/logger${UNIT}.cpp)
+endforeach()
+
+# OBJECT, not an executable: units are never linked together -- nothing
+# here needs a working binary, only a valid compile_commands.json entry
+# per TU -- so there's no need to give each unit's globals unique names
+# beyond what's already needed to avoid within-pair redefinition.
+add_library(synthetic-corpus OBJECT ${GENERATED_SOURCES})
+target_include_directories(synthetic-corpus PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/generated)

benchmarks/synthetic-corpus/README.md

@@ -0,0 +1,45 @@
+# Synthetic benchmark corpus
+
+The [spdlog benchmark](../../README.md#benchmark) is a real project, but
+it's an external dependency at a pinned commit -- reproducing it means
+cloning spdlog. This corpus is the alternative: a deliberately-sized,
+fully self-contained C++ project generated entirely from templates in
+this directory, so `--jobs`/`--cache-dir` benchmarks are reproducible
+from a clean clone of this repo alone, with a known, exact finding count
+to sanity-check the run against.
+
+## Design
+
+Every unit is an independently-compiled pair of translation units
+reproducing the same seven-rule bug pattern as
+[`tests/projects/multi-rule`](../../tests/projects/multi-rule) (missing
+override, unchecked null dereference, redundant move, use-after-move, an
+unbalanced lock, an expensive by-value copy, and a lock-order-inversion
+cycle across the pair). `SENTINEL_BENCHMARK_CORPUS_SIZE` (default 30)
+units are generated by `configure_file()` at CMake-configure time from
+the three templates in `templates/`, substituting `@UNIT@` for a unique
+integer per unit -- nothing beyond the templates themselves is committed.
+Units never link against each other (each is its own `OBJECT` library
+source, never an executable), so there's no cross-unit ODR concern and no
+reason to give each unit's globals more unique naming than "don't collide
+within the same pair."
+
+This gives an exact, predictable finding count: `SENTINEL_BENCHMARK_CORPUS_SIZE * 7`
+findings total, 30 for each of the seven rules at the default size --
+useful as a built-in correctness check on any benchmark run over this
+corpus, not just a timing number.
+
+## Usage
+
+```bash
+cmake -S benchmarks/synthetic-corpus -B benchmarks/synthetic-corpus/build \
+    -DSENTINEL_BENCHMARK_CORPUS_SIZE=30
+cmake --build benchmarks/synthetic-corpus/build -j"$(sysctl -n hw.ncpu)"
+
+./build/tools/cpp-sentinel/cpp-sentinel analyze \
+    --compile-commands benchmarks/synthetic-corpus/build/compile_commands.json \
+    --header-filter '.*'
+```
+
+See the top-level [README's Benchmark section](../../README.md#benchmark)
+for the cold/warm cache numbers measured against this corpus.

benchmarks/synthetic-corpus/templates/geometry.h.in

@@ -0,0 +1,16 @@
+#pragma once
+#include <mutex>
+#include <string>
+
+// Unit @UNIT@ of the synthetic benchmark corpus (see
+// benchmarks/synthetic-corpus/README.md): an independently-compiled copy
+// of the same seven-bug pattern tests/projects/multi-rule uses,
+// mechanically duplicated by CMake's configure_file() so the corpus size
+// is a single cache variable, not N committed files.
+extern std::mutex shapeMutex@UNIT@;
+extern std::mutex logMutex@UNIT@;
+
+struct Shape@UNIT@ {
+  virtual std::string describe();
+  virtual ~Shape@UNIT@() = default;
+};

benchmarks/synthetic-corpus/templates/logger.cpp.in

@@ -0,0 +1,30 @@
+#include "geometry@UNIT@.h"
+#include <string>
+
+void logEvent@UNIT@(const std::string &message) {
+  std::lock_guard<std::mutex> logLock(logMutex@UNIT@);
+  std::lock_guard<std::mutex> shapeLock(shapeMutex@UNIT@);
+}
+
+std::string buildGreeting@UNIT@() {
+  std::string name = "world";
+  return std::move(name);
+}
+
+void logTwice@UNIT@(std::string message) {
+  std::string archived = std::move(message);
+  // Bug: message was just moved-from above -- triggers use-after-move.
+  logEvent@UNIT@(message);
+}
+
+std::mutex auditMutex@UNIT@;
+
+void auditEvent@UNIT@() {
+  // Bug: locks auditMutex but never unlocks it -- triggers unbalanced-lock.
+  auditMutex@UNIT@.lock();
+}
+
+void summarize@UNIT@(std::string report) {
+  // Bug: report is passed by value but only read -- triggers expensive-copy.
+  logEvent@UNIT@(report);
+}

benchmarks/synthetic-corpus/templates/shape.cpp.in

@@ -0,0 +1,19 @@
+#include "geometry@UNIT@.h"
+
+std::mutex shapeMutex@UNIT@;
+std::mutex logMutex@UNIT@;
+
+struct Circle@UNIT@ : Shape@UNIT@ {
+  // Missing 'override': triggers missing-override.
+  std::string describe() {
+    std::lock_guard<std::mutex> shapeLock(shapeMutex@UNIT@);
+    std::lock_guard<std::mutex> logLock(logMutex@UNIT@);
+    return "circle";
+  }
+};
+
+std::string describeAsCircle@UNIT@(Shape@UNIT@ *shape) {
+  // Bug: dynamic_cast<Circle*> can return nullptr; dereferenced with no
+  // null check -- triggers unchecked-null-result.
+  return dynamic_cast<Circle@UNIT@ *>(shape)->describe();
+}

docs/architecture.md

@@ -558,6 +558,24 @@ across independent runs, not session-contaminated -- the number is real,
 even without a fully traced explanation of mull's internal coverage
 bookkeeping.
 
+## Synthetic benchmark corpus
+
+`benchmarks/synthetic-corpus/` exists because the spdlog benchmark, while
+real, is an external dependency at a pinned commit -- reproducing it
+means cloning something else first. Nothing about the corpus itself is
+committed: `CMakeLists.txt` generates `SENTINEL_BENCHMARK_CORPUS_SIZE`
+(default 30) unit pairs via `configure_file()` from three templates,
+substituting `@UNIT@` for a unique integer per unit. Each unit reproduces
+the same seven-rule bug pattern as `tests/projects/multi-rule`; units are
+never linked together (each is only ever an `OBJECT` library source, no
+executable), so there's no cross-unit ODR concern and no reason to make
+each unit's globals more unique than "don't collide within the same
+pair." This makes the corpus's total finding count exactly predictable
+(`SENTINEL_BENCHMARK_CORPUS_SIZE * 7`) -- a correctness check on any
+benchmark run over it, not just a timing number. See the top-level
+README's Benchmark section for the cold/warm cache numbers measured
+against it.
+
 ## Build
 
 CMake links against a single Homebrew/apt-installed LLVM/Clang (the