Skip to content

The reproduce command and the verdict vocabulary - #108

Merged
tamnd merged 3 commits into
mainfrom
b3-22-reproduce-verdicts
Sep 6, 2026
Merged

tamnd merged 3 commits into
mainfrom
b3-22-reproduce-verdicts

Conversation

@tamnd

@tamnd tamnd commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Closes #22.

A claim this project intends to make is now a committed file rather than a paragraph, and iris-bench reproduce C0002 reads that file, runs the instrument it names, and prints one of five words. The registrations live in crates/bench-store/claims/, one TOML file per claim, carrying the identifier, the purpose, the day and place it was registered, the threshold, and the settings the instrument is to be run with. That last part matters more than it looks: a registration that reads its numbers from whatever the tool currently defaults to is a registration whose threshold can be moved without editing it.

The vocabulary is REPRODUCED, REPRODUCED-WITH-CAVEAT, NOT-REPRODUCED, NOT-ATTEMPTABLE and PENDING, and there is no sixth. The set being closed is the only reason it is worth having, because a vocabulary somebody can extend grows a gentler word every time a result comes out badly and the gentler word is the one that gets quoted.

The three that follow from a measurement are not chosen by a person. Bar::decide is the only thing that produces them and it takes three numbers, the reading's distance from the target, the tolerance, and what the instrument can resolve. A caveat can turn a pass into a pass with the condition next to it and it can never turn a loss into anything softer, which is the exact move the fixed set exists to prevent.

NOT-ATTEMPTABLE is reached two ways and both are facts about our circumstances rather than criticism of anybody's work. One is an artifact that cannot be obtained or run here, which has to carry a citation saying where it was looked for, since an unsupported assertion that something was unavailable is not better than no entry. The other is a bar narrower than what the instrument can resolve, and that is why reproduce runs the control first and grades against its interval. A harness that cannot put two copies of one buffer closer than nine percent has no business reporting that two different things are three percent apart. The control comes first rather than after, because running it afterwards would mean grading the reading against a resolution measured on a machine that had since changed.

Run against C0002 the arithmetic lands exactly where the prose in docs/CLAIMS.md already had it, NOT-REPRODUCED at the four mebibyte span iris ships and NOT-ATTEMPTABLE at a span that holds the file. That was the strongest check available that the rule matches what a person reading the same numbers concluded.

Three things are errors rather than verdicts. A claim nobody registered, a reading taken before the day its own threshold was written down, and a claim whose instrument nobody has built yet. The last one is deliberate: giving unwritten work a word would put a finished looking row on a page for something that was never attempted. Everything else exits zero whichever word it lands on, because a failed reproduction is a result and not a broken build.

resident gained a measure and report split so that reproduce can take a reading without printing a gate report at the terminal. That also drops report from eleven arguments to two and removes the too_many_arguments allow that was sitting on it.

The four CI fixes in the first commit

These are unrelated to the above and are what CI on main has been failing on since before #105. The local gate does not run cargo deny, does not cross compile for Windows, and does not resolve action tags, so all four passed locally and failed on push.

fit in the ClickBench tests always produced a value and wrapped it in an Option, which pedantic flags. It now returns the record and the four call sites wrap it, which also reads closer to what the tests are saying.

bench-run called libc::geteuid guarded by a runtime cfg! rather than a cfg attribute, so it had to compile on Windows where that function does not exist. It is now two attributed functions and the non unix one returns the same answer every non root process gets.

libbz2-rs-sys arrives under DataFusion's default features through async-compression and carries the bzip2 licence, which cargo deny rejected. It gets a scoped exception with the reasoning written next to it rather than a feature change, because reducing DataFusion's features to dodge a licence check would change the system under test.

OpenSSF publishes no floating major tag for scorecard-action, only exact versions, so v2 resolves to nothing and the job fails when it tries to download the action. It is pinned to v2.4.4 with a comment saying why the major version rule cannot apply there.

Checks

The full local gate is green: fmt, clippy at -D warnings across all targets and features, both discipline checks, the workspace tests, the doc tests, rustdoc at -D warnings, and cargo check --locked. bench-store gained sixteen tests and a doc test, bench-cli five more.

None of these are related to the verdict work in the next commit, they are
what CI has been red on since before #105 and the local gate does not cover
any of them.

Clippy at pedantic flags fit in the ClickBench tests for wrapping a value it
always produces, so it returns the record and the call sites wrap it.

bench-run called libc::geteuid behind a runtime cfg! rather than a compile
time one, which compiles nowhere without geteuid, so the Windows job failed
on the lib test. It is now two cfg attributed functions.

libbz2-rs-sys arrives under DataFusion's default features and carries the
bzip2 licence, which cargo deny rejected. It gets a scoped exception rather
than a feature change, since reducing DataFusion's features to dodge a
licence check would change the system under test.

OpenSSF publishes no floating major tag for scorecard-action, so v2 resolves
to nothing. It is pinned to the exact version with a comment saying why the
major version rule cannot apply there.
Closes the first half of B3. A claim is registered as a committed TOML file
under crates/bench-store/claims, and iris-bench reproduce runs the instrument
that registration names and prints one of five words.

The five are REPRODUCED, REPRODUCED-WITH-CAVEAT, NOT-REPRODUCED,
NOT-ATTEMPTABLE and PENDING, and the set is closed. The three that follow
from a measurement come out of Bar::decide, which takes the reading, the
threshold and what the instrument can resolve and returns the word. Nobody
picks it after seeing the number, which is the whole point of fixing the
vocabulary.

A caveat can turn a pass into a pass with a condition next to it and it can
never turn a loss into anything softer. NOT-ATTEMPTABLE is reached two ways,
by an artifact that cannot be obtained, which has to carry a citation, and by
a bar narrower than the instrument can resolve. That second road is why
reproduce runs the control first and grades against its interval: a harness
that cannot put two copies of one buffer closer than nine percent has no
business reporting a three percent difference.

Run against C0002 the arithmetic lands where the prose in docs/CLAIMS.md
already had it, NOT-REPRODUCED at the span iris ships and NOT-ATTEMPTABLE at
a span that holds the file, which is the strongest check available that the
rule matches the intent.

A claim nobody registered, a reading taken before its own threshold was
written down, and a claim whose instrument nobody has built are errors rather
than verdicts. Everything else exits zero, because a failed reproduction is a
result and not a broken build.

resident gained a measure and report split so that reproduce can take a
reading without printing a gate report, which also drops report from eleven
arguments to two.
@tamnd tamnd added type: chore Build, tooling, dependencies and repository upkeep area: report Rendering, the site and the publication rules labels Sep 6, 2026
The macOS test that says a control run is not refused for a span it never
opens was asking gate, which also puts the result against a bar. On a shared
runner two pairs can bootstrap an interval wider than any bar, so the test
failed on a question it was not asking. It now asks measure, which is the
part that decides whether the run is refused.

DuckDB was autoloading the Parquet extension the first time a query touched a
Parquet file. That puts a download inside the machine under test and lands on
whatever version the server had rather than the one the lockfile names, and
where the extension directory is not writable it fails outright, which is how
the Windows job found it. It is linked in now.
@tamnd
tamnd merged commit c3975c3 into main Sep 6, 2026
17 of 19 checks passed
@tamnd
tamnd deleted the b3-22-reproduce-verdicts branch September 6, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: report Rendering, the site and the publication rules type: chore Build, tooling, dependencies and repository upkeep

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The reproduce command and the verdict vocabulary

1 participant