Repository navigation
Add the ClickBench driver - #19
Merged
Merged
Conversation
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1.
What changes
rupg-bench clickbench, the ClickBench driver of spec/20 section 20.5. It runs the scripts of one system directory of the ClickBench pin in the order ofbench_main:./start,./loadand async, the tries of each query with the cold cycle of the pin,./data-size, and the concurrent testbench_concurrent_qpsof the pin.exportlines ofbenchmark.sh, with the defaults oflib/benchmark-common.sh. The environment overrides a setting of the form${NAME:-default}.--unitor--attachthe driver measures the server cgroup with the idle base. With neither, each try runs in a new cgroup.duckdbonhits.db.--save-answerskeeps them and--answerschecks them.--result-filewrites the ClickBench result fromtemplate.json.answer_rule.length, which counts characters. The DuckDB queries useSTRLEN, which counts bytes. The report says "not compared" with the reason.machines/install/clickbench.shfetches the pin ate6bda4ewith a sparse checkout oflib/and the system directories.machines/install/postgresql.shgrantspg_read_all_datatobench. The pin scripts load the databasetestaspostgres, and the harness reads the answers asbench.reports/2026-10-07/. They are smoke runs and not baselines.Test
srvteston server3 at c935d99, 2026-10-07: fmt, clippy, cargo deny and 66 tests passed. The new tests arewhat_a_query_determinesandties_at_the_limit.PGVERSION=19 BENCH_CONCURRENT_DURATION=60 rupg-bench clickbench --dir postgresql --engine postgresql --unit postgresql@19-main --source hits1m.tsv --save-answers ans-pg --smoke.hits1m.tsvis the first 1,000,000 rows ofhits.tsv. Load 189.5 s, data size 2614124609 bytes, hot total 427.439 s, cold total 841.971 s, 0 failed, concurrent 0.183 queries per second with error ratio 0.000.BENCH_CONCURRENT_DURATION=60 rupg-bench clickbench --dir duckdb --engine duckdb --source hits_0.parquet --duckdb-bin duckdb --answers ans-pg --smoke.hits_0.parquethas 1,000,000 rows. Load 28.6 s, data size 149434368 bytes, hot total 9.755 s, cold total 18.405 s, 0 failed, 0 of 41 checked answers wrong, q27 and q28 not compared.Notes
duckdb/queryopenshits.dbread-write, so the parallel processes fail with "Could not set lock on file ... Conflicting lock is held". I reproduced the error by hand on server3. This is how the pin behaves, not a fault of the harness, so the DuckDB concurrent number of the pin mostly counts lock errors.PGVERSION=17. The harness needsPGVERSION=19in the environment.data-sizeof the pin isduof the whole cluster directory, so it includes the other databases of the cluster.