chore: cut local cargo test from 6m43s to 20s, fix two loop defects - #1
Merged
Conversation
`cargo test --workspace` was 6 m 43 s at 131% CPU. Three files were 383 s of the 402 s: cross_product_matrix (197 s, four tests each reloading the 1,042-rule corpus over 11 shapes), aarch64_elf_fuzz (166 s, 900 mutants), and dotnet_dnfile_fuzz (20 s, 2,400 mutants). The other 21 test files ran in ~18 s combined. All three are acceptance and robustness gates that an ordinary edit does not move, so they were paying inner-loop cost for pre-merge value. `#[ignore]` them and have CI opt back in with `--include-ignored`. Local `cargo test --workspace` is now 20 s; the CI command runs all 6 with 0 ignored and passes. The MSRV job deliberately keeps the plain command: it asks whether the workspace compiles and passes on the Rust floor, the ignored tests are still compiled there, and re-running the matrix on a second runner tests nothing about the Rust version. Two defects found while measuring: difftest.py defaulted `--capa-cli` to `target/debug/capa`, which is not the binary's name, so a run that omitted the flag could not find it. Worse, on a tree still holding a stale artifact under the old name it would difftest that instead -- and since the harness caches capa-x's side by binary contents, the result would look clean and self-consistent while measuring the wrong build. Point the default at `capa-x`. The outer loop had no `corpus-outer.expected.json`, so difftest fell through the baseline branch to `if failures: return 1`. With 91 known divergences the pre-merge gate AGENTS.md documents could never pass, only be read by hand. Record the baseline after reviewing the run: 200 samples, 98.55% rule-level agreement, 161/200 identical, 91 divergences, 0 errored -- matching the figure the README and CHANGELOG already publish, so this pins the released state rather than freezing a regression. The loop now exits 0 and reports regressions against known diffs. Docs: split CONTRIBUTING's pre-PR block into "before pushing" and "before merge or release", and give both loop tables measured costs that distinguish warm from post-rebuild -- the outer loop is ~2 min after a rebuild but ~2 s without one, and that gap is the difference between the loops reading as cheap and reading as punitive.
Karib0u
marked this pull request as ready for review
August 2, 2026 20:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cargo test --workspacetook 6 m 43 s at 131% CPU — nearly serial. Three files were 383 s of the 402 s; the other 21 test files ran in ~18 s combined:cross_product_matrix.rsaarch64_elf_fuzz.rsdotnet_dnfile_fuzz.rsAll three are acceptance and robustness gates that an ordinary edit does not move — 900 and 2,400 byte-flip mutants asserting "never panics", and a matrix whose four tests each reload the 1,042-rule corpus across 11 shapes. They were paying inner-loop cost for pre-merge value, which is the wrong trade for the check a contributor runs most often.
#[ignore]them; CI opts back in with--include-ignored. Localcargo test --workspaceis now 20 s.The MSRV job deliberately keeps the plain command: it asks whether the workspace compiles and passes on the Rust floor, the ignored tests are still compiled there, and re-running the matrix on a second runner tests nothing about the Rust version. That reasoning is a comment in the workflow so it doesn't read as an oversight.
Two defects found while measuring
scripts/difftest.pyhad the wrong default binary.--capa-clidefaulted totarget/debug/capa, which is not the binary's name — so a run that omitted the flag simply could not find it. Worse, on a tree still holding a stale artifact under the old name, it would difftest that instead; because the harness caches capa-x's side by binary contents, the result would look clean and self-consistent while measuring the wrong build. Default now points atcapa-x.The outer loop could never pass.
scripts/corpus-outer.expected.jsondid not exist, so difftest fell through the baseline branch toif failures: return 1. With 91 known divergences, the pre-merge gateAGENTS.mddocuments always exited nonzero — usable only as a report read by hand, never as the gate it is described as. Baseline now recorded; the loop exits 0 and reports regressions against known diffs.Reference behavior
No analysis behavior changes — no crate source is touched, only test attributes, the harness default, docs, and the new baseline file.
The baseline records the existing comparison against pinned Python capa 9.4.0 rather than altering it: 98.55% rule-level agreement, 161/200 identical, 91 divergences, 0 errored. That is the same figure the README and CHANGELOG already publish for 1.0.0, so it pins the released state rather than freezing a regression. I ran and reviewed the corpus before recording it.
Validation
Outer corpus (200 samples,
--jobs 6, release binary, capa-x side computed from cold):Baseline sanity: 200 samples, 39 with diffs, 91 total diffs, 0 errored.
Notes for the reviewer
--include-ignoredis now load-bearing. Dropping it fromci.ymlsilently removes ~383 s of acceptance coverage behind a green checkmark. There's a comment saying so, but it's the one line here worth guarding — a CI assertion on the ignored-test count would harden it if you want that.KNOWN_DIVERGENCES.mdis untouched.Checklist
--include-ignoredrun reporting 0 ignored.)jobs_determinism.rsunchanged and passing; the matrix's jobs cell passes under--include-ignored.)