Skip to content

chore: cut local cargo test from 6m43s to 20s, fix two loop defects - #1

Merged
Karib0u merged 1 commit into
mainfrom
chore/local-dev-loop-cost
Aug 18, 2026
Merged

chore: cut local cargo test from 6m43s to 20s, fix two loop defects#1
Karib0u merged 1 commit into
mainfrom
chore/local-dev-loop-cost

Conversation

@Karib0u

@Karib0u Karib0u commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

cargo test --workspace took 6 m 43 s at 131% CPU — nearly serial. Three files were 383 s of the 402 s; the other 21 test files ran in ~18 s combined:

Test file Time Share
cross_product_matrix.rs 197 s 49%
aarch64_elf_fuzz.rs 166 s 41%
dotnet_dnfile_fuzz.rs 20 s 5%
all other 21 files ~18 s 4%

All three are acceptance and robustness gates that an ordinary edit does not move — 900 and 2,400 byte-flip mutants asserting "never panics", and a matrix whose four tests each reload the 1,042-rule corpus across 11 shapes. They were paying inner-loop cost for pre-merge value, which is the wrong trade for the check a contributor runs most often.

#[ignore] them; CI opts back in with --include-ignored. Local cargo test --workspace is now 20 s.

The MSRV job deliberately keeps the plain command: it asks whether the workspace compiles and passes on the Rust floor, the ignored tests are still compiled there, and re-running the matrix on a second runner tests nothing about the Rust version. That reasoning is a comment in the workflow so it doesn't read as an oversight.

Two defects found while measuring

scripts/difftest.py had the wrong default binary. --capa-cli defaulted to target/debug/capa, which is not the binary's name — so a run that omitted the flag simply could not find it. Worse, on a tree still holding a stale artifact under the old name, it would difftest that instead; because the harness caches capa-x's side by binary contents, the result would look clean and self-consistent while measuring the wrong build. Default now points at capa-x.

The outer loop could never pass. scripts/corpus-outer.expected.json did not exist, so difftest fell through the baseline branch to if failures: return 1. With 91 known divergences, the pre-merge gate AGENTS.md documents always exited nonzero — usable only as a report read by hand, never as the gate it is described as. Baseline now recorded; the loop exits 0 and reports regressions against known diffs.

Reference behavior

No analysis behavior changes — no crate source is touched, only test attributes, the harness default, docs, and the new baseline file.

The baseline records the existing comparison against pinned Python capa 9.4.0 rather than altering it: 98.55% rule-level agreement, 161/200 identical, 91 divergences, 0 errored. That is the same figure the README and CHANGELOG already publish for 1.0.0, so it pins the released state rather than freezing a regression. I ran and reviewed the corpus before recording it.

Validation

$ cargo fmt --all -- --check
FMT OK

$ cargo clippy --workspace --all-targets -- -D warnings
Finished `dev` profile                    # no warnings

$ cargo test --workspace
26.09s user 2.26s system 140% cpu 20.145 total
25/25 test binaries ok, 6 ignored         # was 6 m 43 s

$ cargo test --workspace -- --include-ignored     # the CI command
all green, 0 ignored, exit 0
  cross_product_matrix   ~197 s
  aarch64_elf_fuzz       ~166 s
  dotnet_dnfile_fuzz      ~20 s

$ scripts/check_env.sh
ok: .venv flare-capa 9.4.0 matches PINNED.md
ok: submodule gitlinks match PINNED.md

Outer corpus (200 samples, --jobs 6, release binary, capa-x side computed from cold):

Rule-level agreement: 98.55% (91 divergence(s) / 6263 reference-matched rules)
Divergences: 61 missing, 30 extra
Samples identical: 161/200 (80.5%)

baseline: scripts/corpus-outer.expected.json
no regressions: 39/200 sample(s) differ, all within the baseline   # exit 0

Baseline sanity: 200 samples, 39 with diffs, 91 total diffs, 0 errored.

Notes for the reviewer

  • --include-ignored is now load-bearing. Dropping it from ci.yml silently removes ~383 s of acceptance coverage behind a green checkmark. There's a comment saying so, but it's the one line here worth guarding — a CI assertion on the ignored-test count would harden it if you want that.
  • Measurements are from a 10-core M1 Max, warm build, clean tree. The loop costs added to the docs distinguish post-rebuild from warm, because the difftest cache keys capa-x's side on binary contents: the outer loop is ~2 min after a rebuild and ~2 s without one.
  • Nothing here changes analysis behavior, so KNOWN_DIVERGENCES.md is untouched.

Checklist

  • The change is focused and includes tests where behavior changed. (No behavior changed; test coverage is identical, verified by the --include-ignored run reporting 0 ignored.)
  • Unknown or unsupported input still fails with context.
  • Untrusted input paths contain no new panic.
  • Output remains deterministic across job counts. (jobs_determinism.rs unchanged and passing; the matrix's jobs cell passes under --include-ignored.)
  • No pinned submodule content was modified.
  • Documentation and changelog entries are updated when needed.
  • New dependencies, if any, are justified and audited in the description. (None added.)

`cargo test --workspace` was 6 m 43 s at 131% CPU. Three files were 383 s of
the 402 s: cross_product_matrix (197 s, four tests each reloading the
1,042-rule corpus over 11 shapes), aarch64_elf_fuzz (166 s, 900 mutants), and
dotnet_dnfile_fuzz (20 s, 2,400 mutants). The other 21 test files ran in ~18 s
combined. All three are acceptance and robustness gates that an ordinary edit
does not move, so they were paying inner-loop cost for pre-merge value.

`#[ignore]` them and have CI opt back in with `--include-ignored`. Local
`cargo test --workspace` is now 20 s; the CI command runs all 6 with 0 ignored
and passes. The MSRV job deliberately keeps the plain command: it asks whether
the workspace compiles and passes on the Rust floor, the ignored tests are
still compiled there, and re-running the matrix on a second runner tests
nothing about the Rust version.

Two defects found while measuring:

difftest.py defaulted `--capa-cli` to `target/debug/capa`, which is not the
binary's name, so a run that omitted the flag could not find it. Worse, on a
tree still holding a stale artifact under the old name it would difftest that
instead -- and since the harness caches capa-x's side by binary contents, the
result would look clean and self-consistent while measuring the wrong build.
Point the default at `capa-x`.

The outer loop had no `corpus-outer.expected.json`, so difftest fell through
the baseline branch to `if failures: return 1`. With 91 known divergences the
pre-merge gate AGENTS.md documents could never pass, only be read by hand.
Record the baseline after reviewing the run: 200 samples, 98.55% rule-level
agreement, 161/200 identical, 91 divergences, 0 errored -- matching the figure
the README and CHANGELOG already publish, so this pins the released state
rather than freezing a regression. The loop now exits 0 and reports
regressions against known diffs.

Docs: split CONTRIBUTING's pre-PR block into "before pushing" and "before
merge or release", and give both loop tables measured costs that distinguish
warm from post-rebuild -- the outer loop is ~2 min after a rebuild but ~2 s
without one, and that gap is the difference between the loops reading as cheap
and reading as punitive.
@Karib0u
Karib0u marked this pull request as ready for review August 2, 2026 20:19
@Karib0u
Karib0u merged commit a3eb2a8 into main Aug 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant