Repository navigation
The reproduce command and the verdict vocabulary - #108
Merged
Merged
Conversation
None of these are related to the verdict work in the next commit, they are what CI has been red on since before #105 and the local gate does not cover any of them. Clippy at pedantic flags fit in the ClickBench tests for wrapping a value it always produces, so it returns the record and the call sites wrap it. bench-run called libc::geteuid behind a runtime cfg! rather than a compile time one, which compiles nowhere without geteuid, so the Windows job failed on the lib test. It is now two cfg attributed functions. libbz2-rs-sys arrives under DataFusion's default features and carries the bzip2 licence, which cargo deny rejected. It gets a scoped exception rather than a feature change, since reducing DataFusion's features to dodge a licence check would change the system under test. OpenSSF publishes no floating major tag for scorecard-action, so v2 resolves to nothing. It is pinned to the exact version with a comment saying why the major version rule cannot apply there.
Closes the first half of B3. A claim is registered as a committed TOML file under crates/bench-store/claims, and iris-bench reproduce runs the instrument that registration names and prints one of five words. The five are REPRODUCED, REPRODUCED-WITH-CAVEAT, NOT-REPRODUCED, NOT-ATTEMPTABLE and PENDING, and the set is closed. The three that follow from a measurement come out of Bar::decide, which takes the reading, the threshold and what the instrument can resolve and returns the word. Nobody picks it after seeing the number, which is the whole point of fixing the vocabulary. A caveat can turn a pass into a pass with a condition next to it and it can never turn a loss into anything softer. NOT-ATTEMPTABLE is reached two ways, by an artifact that cannot be obtained, which has to carry a citation, and by a bar narrower than the instrument can resolve. That second road is why reproduce runs the control first and grades against its interval: a harness that cannot put two copies of one buffer closer than nine percent has no business reporting a three percent difference. Run against C0002 the arithmetic lands where the prose in docs/CLAIMS.md already had it, NOT-REPRODUCED at the span iris ships and NOT-ATTEMPTABLE at a span that holds the file, which is the strongest check available that the rule matches the intent. A claim nobody registered, a reading taken before its own threshold was written down, and a claim whose instrument nobody has built are errors rather than verdicts. Everything else exits zero, because a failed reproduction is a result and not a broken build. resident gained a measure and report split so that reproduce can take a reading without printing a gate report, which also drops report from eleven arguments to two.
The macOS test that says a control run is not refused for a span it never opens was asking gate, which also puts the result against a bar. On a shared runner two pairs can bootstrap an interval wider than any bar, so the test failed on a question it was not asking. It now asks measure, which is the part that decides whether the run is refused. DuckDB was autoloading the Parquet extension the first time a query touched a Parquet file. That puts a download inside the machine under test and lands on whatever version the server had rather than the one the lockfile names, and where the extension directory is not writable it fails outright, which is how the Windows job found it. It is linked in now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22.
A claim this project intends to make is now a committed file rather than a paragraph, and
iris-bench reproduce C0002reads that file, runs the instrument it names, and prints one of five words. The registrations live incrates/bench-store/claims/, one TOML file per claim, carrying the identifier, the purpose, the day and place it was registered, the threshold, and the settings the instrument is to be run with. That last part matters more than it looks: a registration that reads its numbers from whatever the tool currently defaults to is a registration whose threshold can be moved without editing it.The vocabulary is REPRODUCED, REPRODUCED-WITH-CAVEAT, NOT-REPRODUCED, NOT-ATTEMPTABLE and PENDING, and there is no sixth. The set being closed is the only reason it is worth having, because a vocabulary somebody can extend grows a gentler word every time a result comes out badly and the gentler word is the one that gets quoted.
The three that follow from a measurement are not chosen by a person.
Bar::decideis the only thing that produces them and it takes three numbers, the reading's distance from the target, the tolerance, and what the instrument can resolve. A caveat can turn a pass into a pass with the condition next to it and it can never turn a loss into anything softer, which is the exact move the fixed set exists to prevent.NOT-ATTEMPTABLE is reached two ways and both are facts about our circumstances rather than criticism of anybody's work. One is an artifact that cannot be obtained or run here, which has to carry a citation saying where it was looked for, since an unsupported assertion that something was unavailable is not better than no entry. The other is a bar narrower than what the instrument can resolve, and that is why
reproduceruns the control first and grades against its interval. A harness that cannot put two copies of one buffer closer than nine percent has no business reporting that two different things are three percent apart. The control comes first rather than after, because running it afterwards would mean grading the reading against a resolution measured on a machine that had since changed.Run against C0002 the arithmetic lands exactly where the prose in
docs/CLAIMS.mdalready had it, NOT-REPRODUCED at the four mebibyte span iris ships and NOT-ATTEMPTABLE at a span that holds the file. That was the strongest check available that the rule matches what a person reading the same numbers concluded.Three things are errors rather than verdicts. A claim nobody registered, a reading taken before the day its own threshold was written down, and a claim whose instrument nobody has built yet. The last one is deliberate: giving unwritten work a word would put a finished looking row on a page for something that was never attempted. Everything else exits zero whichever word it lands on, because a failed reproduction is a result and not a broken build.
residentgained ameasureandreportsplit so thatreproducecan take a reading without printing a gate report at the terminal. That also dropsreportfrom eleven arguments to two and removes thetoo_many_argumentsallow that was sitting on it.The four CI fixes in the first commit
These are unrelated to the above and are what CI on
mainhas been failing on since before #105. The local gate does not runcargo deny, does not cross compile for Windows, and does not resolve action tags, so all four passed locally and failed on push.fitin the ClickBench tests always produced a value and wrapped it in anOption, which pedantic flags. It now returns the record and the four call sites wrap it, which also reads closer to what the tests are saying.bench-runcalledlibc::geteuidguarded by a runtimecfg!rather than acfgattribute, so it had to compile on Windows where that function does not exist. It is now two attributed functions and the non unix one returns the same answer every non root process gets.libbz2-rs-sysarrives under DataFusion's default features throughasync-compressionand carries the bzip2 licence, whichcargo denyrejected. It gets a scoped exception with the reasoning written next to it rather than a feature change, because reducing DataFusion's features to dodge a licence check would change the system under test.OpenSSF publishes no floating major tag for
scorecard-action, only exact versions, sov2resolves to nothing and the job fails when it tries to download the action. It is pinned tov2.4.4with a comment saying why the major version rule cannot apply there.Checks
The full local gate is green: fmt, clippy at
-D warningsacross all targets and features, both discipline checks, the workspace tests, the doc tests, rustdoc at-D warnings, andcargo check --locked.bench-storegained sixteen tests and a doc test,bench-clifive more.