Skip to content

Give the four google queries something to filter - #106

Merged
tamnd merged 1 commit into
mainfrom
google-blind-spot
Sep 6, 2026
Merged

tamnd merged 1 commit into
mainfrom
google-blind-spot

Conversation

@tamnd

@tamnd tamnd commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Four of the forty three ClickBench queries filter on the word google. q20 counts URL LIKE '%google%', q21 groups by search phrase under the same filter, q22 wants Title LIKE '%Google%' with URL NOT LIKE '%.google.%' beside it, and q23 is SELECT * under the first filter again.

The fixture's made up text was http://www.example{group}.org/{tail} and Title-{tail}. Neither contains the word, so all four queries returned no rows on both engines, and two empty results have the same digest. The test that says the engines agree on all forty three was agreeing about those four for a reason that had nothing to do with either engine, which is a test that cannot fail.

Now group zero is www.google.org, which matches %google% and also matches %.google.%, so it is the row q22 has to drop. Group one is www.google1.org, which matches the first and not the second, so it is the row q22 has to keep. Group two has no google in its address and is what the other three leave behind. Every title carries the word, so q22 is deciding on the address rather than on nothing. A new test asserts the row counts: one row from q20, two from q21, two from q22 and six from q23.

The address lengths stay different from each other, twenty three, twenty five and twenty seven, because the workload orders by AVG(length(URL)) in two places and equal lengths there are the tie the whole fixture is shaped to avoid.

q23 now disagrees on the fixture, which is correct

With rows to return, q23 does on this fixture exactly what it does on the real corpus. It asks for every column, DuckDB's published load converts EventDate where it stands and DataFusion's published view appends it, so the same rows come back with that column in a different place. It is on the list of queries this benchmark does not determine, added in #104, and the tests now say so by name rather than by counting.

That also means the fixture reproduces one entry on that list rather than none. The other nine are ties in the real data, and no arrangement of a tie free fixture shows those, which the doc comment on the fixture's shape now says instead of pointing at a milestone that has since finished.

The test that proves the comparison can find something no longer asserts there is exactly one disagreement. It asserts that the lie it planted is the one disagreement nothing excuses, which is the check a caller actually makes before deciding whether to publish timings.

Full local gate green: fmt, clippy with pedantic on all targets and all features, discipline and its tests, the workspace tests, the doc tests, rustdoc with warnings denied, and a locked check.

Four of the forty three filter on the word google and the fixture's made up
text never contained it, so all four returned no rows on both engines and
agreed with each other whatever either had done. The addresses and titles
carry the word now, placed so that each clause of the fourth query has
something to do, and a test asserts all four come back with rows. q23 then
disagrees on the fixture for the same setup reason it disagrees on the real
corpus, which the tests now expect and name.
@tamnd tamnd added type: bug Something behaves incorrectly area: runner Process isolation, ordering, tiers and cache control benchmark: clickbench ClickBench labels Sep 6, 2026
@tamnd
tamnd merged commit 29799fe into main Sep 6, 2026
13 of 17 checks passed
@tamnd
tamnd deleted the google-blind-spot branch September 6, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: runner Process isolation, ordering, tiers and cache control benchmark: clickbench ClickBench type: bug Something behaves incorrectly

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant