Repository navigation
Put the conditions a run was taken under into its record - #107
Merged
Merged
Conversation
The eligibility gates have an override for measuring on an unfit machine deliberately, and the override left no trace in the record. A line on the terminal saying the machine was busy is gone a week later and the number is still there. So a record now carries whether the gates passed, what was overridden when they did not, and the threads and memory the system was given, and calibrate refuses to grade a run that does not say it passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Towards #19.
clickbench runrefuses to measure on a machine that failed its eligibility gates, and--anywayoverrides that on purpose, because measuring a busy machine deliberately is a reasonable thing to do. What was missing is that the override left no trace. It printed a line on the terminal and then wrote a record that looked exactly like one taken on an idle machine. A week later the line is gone and the number is still there, which is the wrong way round and is the failure mode this repository's first rule exists to prevent.calibrateis where that bites hardest. It reads a set of records and pronounces a driver misconfigured, and until now it would do that from records taken under any conditions at all. That verdict is not safe to draw from a busy machine, and not because the machine is slow. A slower machine is one factor and the arithmetic already divides it out. Another tenant competing for memory bandwidth is not one factor: it slows the queries that stream and leaves the ones that do not alone, which is precisely the shape a misconfigured driver makes. Grading that would be reporting a busy machine as a broken engine.What changed
A record carries a
conditionsobject: what the gates refused and why, when they refused, and the thread count and memory budget the system was given. The refusal is carried whole, as the gate's own text, so a reader gets which gate rather than the bare fact that one failed.calibraterefuses to grade unless every record says it passed.--anywayoncalibrategrades it and prints that nothing below is a finding about any engine and none of it belongs in a table.A record with no
conditionsat all, which is every record written before this, is treated the same as one that failed. That is the only safe reading: a record that does not claim it passed its gates has not passed them. It is a deliberate choice rather than a migration oversight, and the doc comment on the field says so.calibratealso refuses records that were given different thread counts or different memory budgets, in the same spirit as the existing check that they share an environment hash. A machine factor shared across two drivers means something only if both drivers ran on the same machine.Checked against the real records
The three from the first complete three driver run predate the field, so they are now refused by name:
With
--anywaythe same three print the table under a line saying it is not a finding. That is the right outcome for those records: they were taken with the gates overridden at a load average of 25.28 against a limit of 6.40, and the calibration they produce is not evidence about either engine.Four unit tests cover it: an unfit record is refused and grades under
--anyway, a record that says nothing is refused the same way, records given different machines are refused even when both are fit, and a fit pair grades without complaint.Full local gate green: fmt, clippy with pedantic on all targets and all features, discipline and its tests, the workspace tests, the doc tests, rustdoc with warnings denied, and a locked check.