Repository navigation
Our ClickBench geomean lands within 25% of the public leaderboard #19
Description
This is the calibration gate for the entire repository. If a system we configured is much slower here than the same system on the public leaderboard, we configured it wrong, and every comparison we publish afterwards is measuring our configuration mistake.
Different hardware moves the absolute numbers, which is why the gate is on the shape of the result rather than on the value.
Done when the comparison is done for all three drivers, and any driver outside the band gets a misconfiguration issue rather than a footnote.
Activity
tamnd commentedon Sep 6, 2026
The calibration has now been run against a real three driver record set, on the class B Intel Core i9-13900K workstation, so there are numbers here rather than a plan.
DuckDB's hot geometric mean over the forty three is 0.0628s against 0.2287s for DuckDB on c6a.4xlarge, which is 0.27x. DataFusion's is 0.1872s against 0.3558s for DataFusion in its single Parquet file entry, which is 0.53x. The reference reader answers none of the forty three so it has no published counterpart and is reported as having nothing to compare against rather than as passing.
The shared machine factor is 0.38x, and the band is 25 percent either side of it. DuckDB is 28 percent under and DataFusion is 38 percent over, so both are outside and the command exits non zero, which is what it is for.
The interesting part is the direction. A slower or faster machine moves every system the same way, and that is the whole reason the gate is on the shape rather than on the value. These two moved in opposite directions, so at least one of them is not the machine. The per query breakdown puts most of it in a handful of heavy queries: q32 and q18 are the top two on both sides, DataFusion is 5.5x the published number on q32 and 2.8x on q18, and DuckDB is 4.7x on q32 and faster than published almost everywhere else.
Two things have to be said about this record before any of that is treated as a finding about either engine.
The run carries --anyway, and the reason is in the record. The machine failed the core pinning gate for the DuckDB run and the load average gate for the other two, reporting a one minute load average of 25.28 against a limit of 6.40. It is a workstation that is doing other things. Absolute durations from it are not publishable and the record says so.
It was also given --memory 6, because the machine had about ten gibibytes available at the time and twelve had DuckDB killed. Six gibibytes is not what the leaderboard entries are taken with, and a memory limit that tight is a plausible cause of exactly the pattern above, since it hurts the queries that build large hash tables and leaves the cheap ones alone. q32 groups by WatchID and ClientIP, which on this corpus is close to one group per row, and it is the worst query on both engines.
So the number to take from this is that the calibration works and produces a verdict, and the verdict on this record is that neither engine is in band. What it is not yet is evidence that either is misconfigured, because the most likely misconfiguration is the one in the command line. The next run needs a machine that passes its gates and enough memory to give the engines what the published entries had, and if the same split shows up there then it is a finding and gets its own issue as the exit condition requires.
tamnd commentedon Sep 6, 2026
The comparison exists and runs on all three drivers, so the first half of the exit condition is met. The second half is not, and after checking the fleet today the reason is that no machine here can currently produce a run this gate is entitled to read.
The four machines, checked within a few minutes of each other:
The 4 vCPU class A guest was at a load average of 47.93 with none of its 5 GiB of memory available. The 6 vCPU guest was at 5.34 with 7 GiB of 11 available. The 8 vCPU guest was at 30.70 with 5 GiB of 23 available. Class A publishes no durations anyway, so those three are only context.
The one that matters is class B, the Intel Core i9-13900K workstation, which is the only machine in the fleet whose noise floor is under the bar and the only one that can carry a duration. It had been rebooted, so the first look showed a load average of 0.45 and 12 GiB of 23 available. Ten seconds later it was down to 4.4 GiB available, and the resident set was a JVM at 8.9 GiB, a drive daemon at 5.4 GiB, a memcached at 3.3 GiB, and a Postgres, a MinIO, a MongoDB and a memtier load generator underneath those. The machine is hosting a working set on the order of twenty gibibytes and a load generator is driving it.
That explains the run behind the current records rather than excusing it. It needed
--anywayat a load average of 25.28 against a limit of 6.40, and it needed--memory 6because--memory 12had DuckDB killed. Six gibibytes is not what the published c6a.4xlarge entries used, and a budget that far below the reference changes which queries spill, which is not a uniform factor. So the most likely misconfiguration in that calibration is still the one in the command line.I have not touched anything on that machine. Those are somebody's services and stopping them is not mine to decide.
What went in rather than a footnote
#107 makes the tooling unable to repeat this quietly. A record now carries whether the gates passed, the gate's own text when they did not, and the threads and memory the system was given.
calibraterefuses to grade a run that does not say it passed, and refuses records that were given different budgets, because a shared machine factor across two drivers means something only if both ran on the same machine. A record with no conditions at all, which is every record written before that change, is treated as one that failed, since a record that does not claim it passed its gates has not passed them.The three records from the first complete run are now refused by name. Grading them requires
--anyway, and that prints a line saying nothing below is a finding about any engine.What this needs
One run of all three drivers on the class B machine with its resident services stopped, or at least quiet enough to pass
load-averageandmemory-headroom, and with a memory budget in the region of what the published entries had rather than a quarter of it. Until then this stays open, and the calibration numbers taken so far are not a finding about DuckDB or DataFusion and should not be quoted as one.