[tamnd]

Closes #20.

The exit condition for #20 is that digests agree across all three drivers on all 43 queries and that every disagreement is resolved rather than tolerated. Thirty three agree. This resolves the other ten, and the resolution is that ten of ClickBench's forty three do not have one answer to agree about.

Two systems returning different rows for the same query is normally one correct system and one that is fast because it did less. These ten are not that. Their published text does not pick out a single result, so two correct systems return different rows and both are right.

Nine of them are ties under a LIMIT. The window a LIMIT returns is only determined when the ordering key tells the rows inside it apart from the rows just outside, and in these nine it does not. q17 has no ORDER BY at all and takes ten of many million groups. q23 and q24 order by EventTime, which is a second and not a row identity. q21, q31 and q32 order by a count over a grouping close to one row per group. q38, q39, q40 and q41 order by a count and then take ten rows at an offset of a hundred, a thousand or ten thousand, which lands the window deep in a tail where every remaining group has the same tiny count.

That reasoning is not the evidence. Each of the ten was run on the real corpus, on both engines, with the new clickbench answer command, and the rows were read. For all six of the grouped ones the sequence of ordering key values is identical between the two, and the values are tied across the whole window: q31 and q41 return ten counts of one, q38 ten of two, q39 ten of fifteen, q32 four twos then six ones, and q40 has every value in it repeated at least twice so even its boundary sits inside a tie. Two engines returned the same counts and a different arbitrary ten of the rows holding them.

The tenth is q23 and it is a different thing wearing the same clothes. It is SELECT *, and the two published setups do not produce the same table out of the same corpus. The DuckDB load uses SELECT * REPLACE, which converts EventDate where it stands. The DataFusion view uses SELECT * EXCEPT with the converted column appended, which moves EventDate from the fifth column to the hundred and fifth. DuckDB also converts the three timestamps and DataFusion leaves them as raw seconds. On the real corpus the two returned the same ten rows, and a hundred and two of the hundred and five columns matched exactly: the three that did not are EventTime, ClientEventTime and LocalEventTime, holding the same instants written two ways. The rows agree and the rendering cannot, and that follows from carrying each entry's own setup rather than from anything either engine did.

Why a list and not a rule

Whether an ordering key separates the rows a LIMIT cuts between depends on the data, not on the SQL. A syntactic rule would either excuse every query with a LIMIT, which is most of the workload, or claim a precision it does not have. So this is a list of ten, in bench_workload::clickbench::undetermined, one entry at a time with the reason attached.

Getting on the list is an edit somebody makes on purpose and defends, not something a query earns by disagreeing. A test asserts the list is exactly these ten so it cannot grow while nobody is reading, a second test asserts only the query that selects every column blames the setup, and a third asserts every listed query does in fact cut a window short. The other thirty three still have to agree and a disagreement there is still a hard failure that stops the timings being published.

What changed

bench_workload::clickbench gains Undetermined and undetermined(), and the module doc carries the evidence above query by query. Comparison::clean() keeps the strict reading and its doc now says so, and says that a workload with undetermined queries sorts its own disagreements against a list, so the generic comparison crate does not learn about one benchmark's quirks. clickbench check partitions both the disagreements and the instabilities against that list and fails only on the determined ones.

Instability gets the same treatment for the same reason. A system that answers one query two different ways across its own three runs is a fault on a query with one answer and it fails. On one of the ten it is the same tie seen from one system instead of two, and it is reported as that rather than counted as a second finding. q21 disagreed between the engines on one run and agreed on a re-run, which is what that looks like from the outside.

Checked

Against the three records from the first complete three driver run:

33 agreed
q38 disagreed, and the LIMIT cuts a tie, so which rows come back is not decided by the query
    duckdb           10 rows f0a5ae7de7fa0ec7
    datafusion       10 rows b141588c9d30e5ce
...
every query with one answer got the same answer, and 10 of the ones without a single answer came back differently, which is what they do

Exit zero. Thirty three plus ten is the whole workload, and every instability in that record set is on the list too.

Full local gate green: fmt, clippy with pedantic on all targets and all features, discipline and its tests, the workspace tests, the doc tests, rustdoc with warnings denied, and a locked check.