Skip to content

Put the conditions a run was taken under into its record - #107

Merged
tamnd merged 1 commit into
mainfrom
record-the-conditions
Sep 6, 2026
Merged

tamnd merged 1 commit into
mainfrom
record-the-conditions

Conversation

@tamnd

@tamnd tamnd commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Towards #19.

clickbench run refuses to measure on a machine that failed its eligibility gates, and --anyway overrides that on purpose, because measuring a busy machine deliberately is a reasonable thing to do. What was missing is that the override left no trace. It printed a line on the terminal and then wrote a record that looked exactly like one taken on an idle machine. A week later the line is gone and the number is still there, which is the wrong way round and is the failure mode this repository's first rule exists to prevent.

calibrate is where that bites hardest. It reads a set of records and pronounces a driver misconfigured, and until now it would do that from records taken under any conditions at all. That verdict is not safe to draw from a busy machine, and not because the machine is slow. A slower machine is one factor and the arithmetic already divides it out. Another tenant competing for memory bandwidth is not one factor: it slows the queries that stream and leaves the ones that do not alone, which is precisely the shape a misconfigured driver makes. Grading that would be reporting a busy machine as a broken engine.

What changed

A record carries a conditions object: what the gates refused and why, when they refused, and the thread count and memory budget the system was given. The refusal is carried whole, as the gate's own text, so a reader gets which gate rather than the bare fact that one failed.

calibrate refuses to grade unless every record says it passed. --anyway on calibrate grades it and prints that nothing below is a finding about any engine and none of it belongs in a table.

A record with no conditions at all, which is every record written before this, is treated the same as one that failed. That is the only safe reading: a record that does not claim it passed its gates has not passed them. It is a deliberate choice rather than a migration oversight, and the doc comment on the field says so.

calibrate also refuses records that were given different thread counts or different memory budgets, in the same spirit as the existing check that they share an environment hash. A machine factor shared across two drivers means something only if both drivers ran on the same machine.

Checked against the real records

The three from the first complete three driver run predate the field, so they are now refused by name:

/tmp/gpcruns/duckdb.json does not say what it was taken under
/tmp/gpcruns/datafusion.json does not say what it was taken under
/tmp/gpcruns/arrow-parquet.json does not say what it was taken under
Error: a run taken on a machine that failed its gates cannot be graded against the leaderboard, because the failure is not a factor this arithmetic divides out; pass --anyway to print the table with that said on it

With --anyway the same three print the table under a line saying it is not a finding. That is the right outcome for those records: they were taken with the gates overridden at a load average of 25.28 against a limit of 6.40, and the calibration they produce is not evidence about either engine.

Four unit tests cover it: an unfit record is refused and grades under --anyway, a record that says nothing is refused the same way, records given different machines are refused even when both are fit, and a fit pair grades without complaint.

Full local gate green: fmt, clippy with pedantic on all targets and all features, discipline and its tests, the workspace tests, the doc tests, rustdoc with warnings denied, and a locked check.

The eligibility gates have an override for measuring on an unfit machine
deliberately, and the override left no trace in the record. A line on the
terminal saying the machine was busy is gone a week later and the number is
still there. So a record now carries whether the gates passed, what was
overridden when they did not, and the threads and memory the system was
given, and calibrate refuses to grade a run that does not say it passed.
@tamnd tamnd added type: gate A checkable exit gate for a milestone area: runner Process isolation, ordering, tiers and cache control benchmark: clickbench ClickBench labels Sep 6, 2026
@tamnd
tamnd merged commit f234b8c into main Sep 6, 2026
14 of 17 checks passed
@tamnd
tamnd deleted the record-the-conditions branch September 6, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: runner Process isolation, ordering, tiers and cache control benchmark: clickbench ClickBench type: gate A checkable exit gate for a milestone

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant