Repository navigation
Add the instruction count runner - #15
Merged
Merged
Conversation
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1.
What changes
src/instructions.rsis the instruction count runner of spec/21 §21.14. It runs each query of a set underperf stat -e instructions -x ,, the counter that the spec names. Each query runs 3 times and the runner keeps the smallest count.duckdbprocess, withSET threads = 1. For a server, perf counts the cgroup of the server unit on all CPUs with-G, and the client ispsql. Background processes of the server add to that count, so the runner also countsSELECT 13 times and subtracts the smallest of those counts from each query.rupg-bench instructions --queries PATHreads the set from a file with one query on each line, as in ClickBench, or from a directory of.sqlfiles, as for TPC-H.--setnames the set.--enginenames the engine and is part of the suite name, for exampleinstructions-clickbench-postgresql.--ratchet ratchet.tomlcompares the counts with the table of that engine and set. The command fails when one query is more than 3 percent over its row or the total is more than 1 percent over.--savewrites the counts as the new table.ratchet.tomlis new. It has no table yet. The counts must come from a dedicated runner (spec/21 §21.14), and server3 is a shared machine.reports/2026-10-07/6005c2b1-server3-instructions-clickbench-{postgresql,duckdb}-smoke.{md,json}are the reports of the two smoke runs below. Their names and their first line mark them as smoke runs.Test
srvtestpassed on server3: clippy is clean and 43 tests pass. The tests cover the parser of the perf output, the query files and the budgets.Smoke runs on server3, 7 October 2026. These are not baselines and not the ratchet. server3 is a shared machine with 8 cores, and its load average was 66 to 85 that day. The data is the first 1,000,000 rows of ClickBench (
hits_0.parquet). DuckDB 1.5.6 read it intohits1m.db, and PostgreSQL REL_19_STABLE at7d3d2db7read it with\copyinto the tablehitsof the databasebench. The query files arequeries.sqlof the ClickBench pine6bda4efor each system.PostgreSQL, 10:05:06 to 10:32:42 UTC:
DuckDB, 10:32:42 to 10:46:12 UTC:
SELECT 1, smallest of 3The spread is (largest run minus smallest run) divided by the smallest run. On this shared machine the DuckDB spread is above the 3 percent budget. That is why the ratchet table stays empty until a dedicated runner exists.
The
instructionsline of #1 stays open. The fixed set also needs the 22 TPC-H queries at SF0.1, 1,000 YCSB point reads and 100 TPC-C new-order transactions, and those drivers come in the next PRs.