Skip to content

Add the instruction count runner - #15

Merged
tamnd merged 3 commits into
mainfrom
instructions
Oct 7, 2026
Merged

tamnd merged 3 commits into
mainfrom
instructions

Conversation

@tamnd

@tamnd tamnd commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Part of #1.

What changes

  • src/instructions.rs is the instruction count runner of spec/21 §21.14. It runs each query of a set under perf stat -e instructions -x ,, the counter that the spec names. Each query runs 3 times and the runner keeps the smallest count.
  • For DuckDB, perf counts the duckdb process, with SET threads = 1. For a server, perf counts the cgroup of the server unit on all CPUs with -G, and the client is psql. Background processes of the server add to that count, so the runner also counts SELECT 1 3 times and subtracts the smallest of those counts from each query.
  • rupg-bench instructions --queries PATH reads the set from a file with one query on each line, as in ClickBench, or from a directory of .sql files, as for TPC-H. --set names the set. --engine names the engine and is part of the suite name, for example instructions-clickbench-postgresql.
  • --ratchet ratchet.toml compares the counts with the table of that engine and set. The command fails when one query is more than 3 percent over its row or the total is more than 1 percent over. --save writes the counts as the new table.
  • ratchet.toml is new. It has no table yet. The counts must come from a dedicated runner (spec/21 §21.14), and server3 is a shared machine.
  • reports/2026-10-07/6005c2b1-server3-instructions-clickbench-{postgresql,duckdb}-smoke.{md,json} are the reports of the two smoke runs below. Their names and their first line mark them as smoke runs.
  • The README explains the command.

Test

srvtest passed on server3: clippy is clean and 43 tests pass. The tests cover the parser of the perf output, the query files and the budgets.

Smoke runs on server3, 7 October 2026. These are not baselines and not the ratchet. server3 is a shared machine with 8 cores, and its load average was 66 to 85 that day. The data is the first 1,000,000 rows of ClickBench (hits_0.parquet). DuckDB 1.5.6 read it into hits1m.db, and PostgreSQL REL_19_STABLE at 7d3d2db7 read it with \copy into the table hits of the database bench. The query files are queries.sql of the ClickBench pin e6bda4e for each system.

PostgreSQL, 10:05:06 to 10:32:42 UTC:

RUPG_BENCH_COMMIT=6005c2b1 rupg-bench instructions --set clickbench --queries pg-queries.sql --engine postgresql --unit postgresql@19-main --smoke --report /root/work/reports --machine server3

DuckDB, 10:32:42 to 10:46:12 UTC:

RUPG_BENCH_COMMIT=6005c2b1 rupg-bench instructions --set clickbench --queries duckdb-queries.sql --duckdb hits1m.db --duckdb-bin /root/work/rupg-bench-opt/bin/duckdb --smoke --report /root/work/reports --machine server3
Number PostgreSQL DuckDB
base, SELECT 1, smallest of 3 12,103,536 78,040,265
total of the 43 queries, net 228,920,014,998 18,064,983,237
largest query q28, 54,758,339,550 q28, 8,580,483,958
median spread of the 3 runs of a query 0.62 percent 4.45 percent
largest spread q0, 77 percent q6, 63 percent

The spread is (largest run minus smallest run) divided by the smallest run. On this shared machine the DuckDB spread is above the 3 percent budget. That is why the ratchet table stays empty until a dedicated runner exists.

The instructions line of #1 stays open. The fixed set also needs the 22 TPC-H queries at SF0.1, 1,000 YCSB point reads and 100 TPC-C new-order transactions, and those drivers come in the next PRs.

@tamnd tamnd added kind/feature It does not do something that it must do. area/resources cgroup v2 memory, CPU and disk measurement. labels Oct 7, 2026
@tamnd tamnd self-assigned this Oct 7, 2026
@tamnd
tamnd merged commit 6e0d4c1 into main Oct 7, 2026
2 checks passed
@tamnd
tamnd deleted the instructions branch October 7, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/resources cgroup v2 memory, CPU and disk measurement. kind/feature It does not do something that it must do.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant