Skip to content

Add the ClickBench driver - #19

Merged
tamnd merged 3 commits into
mainfrom
clickbench
Oct 7, 2026
Merged

tamnd merged 3 commits into
mainfrom
clickbench

Conversation

@tamnd

@tamnd tamnd commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Part of #1.

What changes

  • Add rupg-bench clickbench, the ClickBench driver of spec/20 section 20.5. It runs the scripts of one system directory of the ClickBench pin in the order of bench_main: ./start, ./load and a sync, the tries of each query with the cold cycle of the pin, ./data-size, and the concurrent test bench_concurrent_qps of the pin.
  • The settings come from the export lines of benchmark.sh, with the defaults of lib/benchmark-common.sh. The environment overrides a setting of the form ${NAME:-default}.
  • With --unit or --attach the driver measures the server cgroup with the idle base. With neither, each try runs in a new cgroup.
  • The answers come from a second pass that is not timed: through the PostgreSQL protocol for a server, or with duckdb on hits.db. --save-answers keeps them and --answers checks them. --result-file writes the ClickBench result from template.json.
  • Add a check that knows what a query fixes. A query with no LIMIT fixes all its rows. With LIMIT and ORDER BY keys in the output, the rows fix only up to the ties at the limit: the multiset of the keys must match, and the rows that do not tie with the last row must match. With LIMIT and no ORDER BY, or a key that is not in the output (q17, q23, q24), only the row and column counts must match. The report shows the rule in the column answer_rule.
  • q27 and q28 are not compared. The PostgreSQL queries use length, which counts characters. The DuckDB queries use STRLEN, which counts bytes. The report says "not compared" with the reason.
  • machines/install/clickbench.sh fetches the pin at e6bda4e with a sparse checkout of lib/ and the system directories.
  • machines/install/postgresql.sh grants pg_read_all_data to bench. The pin scripts load the database test as postgres, and the harness reads the answers as bench.
  • The README lists ClickBench in the status and describes the driver with two examples.
  • Two smoke reports in reports/2026-10-07/. They are smoke runs and not baselines.

Test

  • srvtest on server3 at c935d99, 2026-10-07: fmt, clippy, cargo deny and 66 tests passed. The new tests are what_a_query_determines and ties_at_the_limit.
  • PostgreSQL smoke run on server3, 2026-10-07 13:42 to 14:42 UTC, load average about 70, at commit 2b1cb0b8 (the same driver before the rebase on main): PGVERSION=19 BENCH_CONCURRENT_DURATION=60 rupg-bench clickbench --dir postgresql --engine postgresql --unit postgresql@19-main --source hits1m.tsv --save-answers ans-pg --smoke. hits1m.tsv is the first 1,000,000 rows of hits.tsv. Load 189.5 s, data size 2614124609 bytes, hot total 427.439 s, cold total 841.971 s, 0 failed, concurrent 0.183 queries per second with error ratio 0.000.
  • DuckDB smoke run on server3, 2026-10-07 14:59 to 15:08 UTC, at c935d991: BENCH_CONCURRENT_DURATION=60 rupg-bench clickbench --dir duckdb --engine duckdb --source hits_0.parquet --duckdb-bin duckdb --answers ans-pg --smoke. hits_0.parquet has 1,000,000 rows. Load 28.6 s, data size 149434368 bytes, hot total 9.755 s, cold total 18.405 s, 0 failed, 0 of 41 checked answers wrong, q27 and q28 not compared.
  • Before the check knew about ties, the same DuckDB run had 12 of 43 answers wrong (q11, q17, q18, q22, q24, q27, q28, q31, q32, q38, q39, q40). I read each one. They were rows that tie at the LIMIT, queries with no ORDER BY, and the two STRLEN queries. None was a real difference in the data.

Notes

  • The DuckDB concurrent test of the pin has an error ratio of 0.934. The pin's duckdb/query opens hits.db read-write, so the parallel processes fail with "Could not set lock on file ... Conflicting lock is held". I reproduced the error by hand on server3. This is how the pin behaves, not a fault of the harness, so the DuckDB concurrent number of the pin mostly counts lock errors.
  • The PostgreSQL scripts of the pin default to PGVERSION=17. The harness needs PGVERSION=19 in the environment.
  • The PostgreSQL data-size of the pin is du of the whole cluster directory, so it includes the other databases of the cluster.
  • spec/20 section 20.5 says each result must equal the result of PostgreSQL 19. That cannot hold for the ClickBench queries that tie at the LIMIT or have no ORDER BY, and for q27 and q28. The check above is the strongest rule that the queries allow.

@tamnd tamnd added kind/feature It does not do something that it must do. area/clickbench ClickBench, run to the official rules. labels Oct 7, 2026
@tamnd tamnd self-assigned this Oct 7, 2026
@tamnd tamnd added kind/feature It does not do something that it must do. area/clickbench ClickBench, run to the official rules. labels Oct 7, 2026
@tamnd
tamnd merged commit 24d6158 into main Oct 7, 2026
@tamnd
tamnd deleted the clickbench branch October 7, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/clickbench ClickBench, run to the official rules. kind/feature It does not do something that it must do.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant