Skip to content

Add the instruction count runner #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tamnd merged 3 commits into from
Oct 7, 2026
Merged

Add the instruction count runner #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
13c225f
Add the instruction count runner
tamnd Oct 7, 2026
6005c2b
Name the engine in the report
tamnd Oct 7, 2026
d3389d3
Add the instruction count smoke reports for ClickBench
tamnd Oct 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The procedure is [`spec/20-benchmarks.md`](https://github.com/tamnd/rupg/blob/ma

## Status

Early. The crate builds and CI is green, but no suite runs yet. `rupg-bench gates` prints the twelve gates of spec/02 section 2.10, `rupg-bench pins` prints the pinned versions, `rupg-bench measure` runs the cgroup v2 runner, `rupg-bench load` runs the load cases, `rupg-bench answers` checks answer sets, `rupg-bench report` writes the report files, and `rupg-bench pgbench` runs the pgbench smoke run with its check. The first milestone, M0, builds the harness and measures every baseline that a later gate uses. Every command below is the planned interface from the spec.
Early. The crate builds and CI is green, but no suite runs yet. `rupg-bench gates` prints the twelve gates of spec/02 section 2.10, `rupg-bench pins` prints the pinned versions, `rupg-bench measure` runs the cgroup v2 runner, `rupg-bench load` runs the load cases, `rupg-bench answers` checks answer sets, `rupg-bench report` writes the report files, `rupg-bench pgbench` runs the pgbench smoke run with its check, and `rupg-bench instructions` counts the instructions of each query. The first milestone, M0, builds the harness and measures every baseline that a later gate uses. Every command below is the planned interface from the spec.

## The gates

Expand Down Expand Up @@ -123,6 +123,13 @@ rupg-bench report --result idle.json --suite idle-base --machine server3 --smoke
rupg-bench pgbench --unit postgresql@19-main --scale 1 --clients 4 --time 30 --smoke --report reports --machine server3
```

`rupg-bench instructions` is the instruction count runner of spec/21 section 21.14. It runs each query of a set under `perf stat -e instructions`. For DuckDB, perf counts the `duckdb` process with one thread. For a server, perf counts the cgroup of the server on all CPUs while `psql` sends the query, so the count is the work of the server and not the work of the client. Each query runs three times and the minimum counts. The count of `SELECT 1` in the same way is subtracted, so the net count is the work of the query. A set is a file with one query on each line, named `q0` to `q42` for ClickBench, or a directory of `<name>.sql` files. A server needs `--engine NAME`, and the report name has the set and the engine. `--ratchet ratchet.toml` fails when one query rises by more than 3 percent or the total by more than 1 percent. `--save` writes the table for the ratchet. The ratchet is filled only on the dedicated runner, so [`ratchet.toml`](ratchet.toml) has no table yet.

```sh
rupg-bench instructions --set clickbench --queries queries.sql --duckdb hits.db --duckdb-bin duckdb
rupg-bench instructions --set clickbench --queries queries.sql --engine postgresql --unit postgresql@19-main --ratchet ratchet.toml
```

## The reporting rules

1. A claim has five parts: one named machine, one metric, a baseline that we measured, the same transport on both sides, and the ratio. A claim without all five is not made.
Expand Down
11 changes: 11 additions & 0 deletions ratchet.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# The instruction count ratchet of spec/21 section 21.14.
#
# The dedicated Linux x86-64 runner writes one table for each set with
# `rupg-bench instructions --set NAME --save FILE`, for example
# [instructions.clickbench]. Each key is a query and its value is the net
# instruction count of that query. `total` is the sum of the net counts.
# `rupg-bench instructions --ratchet ratchet.toml` fails when one query rises by
# more than 3 percent or the total by more than 1 percent.
#
# The file has no table yet. Counts from a shared machine such as server3 are
# smoke runs and do not belong here.
246 changes: 246 additions & 0 deletions reports/2026-10-07/6005c2b1-server3-instructions-clickbench-duckdb-smoke.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,246 @@
{
"suite": "instructions-clickbench-duckdb",
"machine": "server3",
"date": "2026-10-07",
"commit": "6005c2b1",
"smoke": true,
"note": "smoke run: it shows that the runner works, it is not the ratchet",
"set": "clickbench",
"engine_name": "duckdb",
"queries_from": "duckdb-queries.sql",
"repeat": 3,
"kernel": "6.8.0-106-generic",
"engine": {
"mode": "in process",
"client": [
"/root/work/rupg-bench-opt/bin/duckdb",
"-readonly",
"hits1m.db",
"-cmd",
"SET threads = 1",
"-c"
]
},
"counts": {
"base_runs": [80916556, 78040265, 78382784],
"base": 78040265,
"total_net": 18064983237
},
"queries": [
{
"query": "q0",
"net": 2658966,
"runs": [81782230, 81926639, 80699231]
},
{
"query": "q1",
"net": 9794033,
"runs": [87834298, 88071044, 98749864]
},
{
"query": "q2",
"net": 21379579,
"runs": [116705717, 100057642, 99419844]
},
{
"query": "q3",
"net": 30056892,
"runs": [109395308, 108097157, 110024982]
},
{
"query": "q4",
"net": 96061769,
"runs": [174102034, 176684910, 174479860]
},
{
"query": "q5",
"net": 142365074,
"runs": [229474235, 220405339, 221811745]
},
{
"query": "q6",
"net": 6391527,
"runs": [84431792, 91324585, 138004129]
},
{
"query": "q7",
"net": 12274205,
"runs": [90314470, 98506520, 90877270]
},
{
"query": "q8",
"net": 141955649,
"runs": [229414983, 220329937, 219995914]
},
{
"query": "q9",
"net": 260171263,
"runs": [338211528, 374255484, 338243517]
},
{
"query": "q10",
"net": 79267130,
"runs": [158599180, 157307395, 158798206]
},
{
"query": "q11",
"net": 85726959,
"runs": [165710997, 163767224, 164860287]
},
{
"query": "q12",
"net": 104967374,
"runs": [184804364, 184240163, 183007639]
},
{
"query": "q13",
"net": 126302995,
"runs": [206476167, 205229570, 204343260]
},
{
"query": "q14",
"net": 116660750,
"runs": [208069401, 194701015, 195197372]
},
{
"query": "q15",
"net": 114154184,
"runs": [193169086, 192194449, 193594877]
},
{
"query": "q16",
"net": 238392237,
"runs": [316985871, 380977329, 316432502]
},
{
"query": "q17",
"net": 232148198,
"runs": [324993525, 310188463, 314307544]
},
{
"query": "q18",
"net": 490274337,
"runs": [577124723, 568314602, 588599687]
},
{
"query": "q19",
"net": 7260510,
"runs": [108010818, 86214489, 85300775]
},
{
"query": "q20",
"net": 474816252,
"runs": [602058081, 557769957, 552856517]
},
{
"query": "q21",
"net": 443819841,
"runs": [524793576, 521860106, 535583770]
},
{
"query": "q22",
"net": 311919457,
"runs": [433448618, 389959722, 411982077]
},
{
"query": "q23",
"net": 780471101,
"runs": [910784254, 858511366, 874619848]
},
{
"query": "q24",
"net": 124617635,
"runs": [202796815, 222973585, 202657900]
},
{
"query": "q25",
"net": 97594025,
"runs": [175634290, 192633348, 176087162]
},
{
"query": "q26",
"net": 131454751,
"runs": [209495016, 218807746, 210925889]
},
{
"query": "q27",
"net": 429523741,
"runs": [507564006, 517167810, 508441273]
},
{
"query": "q28",
"net": 8580483958,
"runs": [8675722436, 8668755425, 8658524223]
},
{
"query": "q29",
"net": 69401430,
"runs": [148583472, 150163834, 147441695]
},
{
"query": "q30",
"net": 136512966,
"runs": [216352389, 229079759, 214553231]
},
{
"query": "q31",
"net": 169098914,
"runs": [256408322, 258144163, 247139179]
},
{
"query": "q32",
"net": 552858320,
"runs": [636146474, 630898585, 635693870]
},
{
"query": "q33",
"net": 635693402,
"runs": [768820987, 714799322, 713733667]
},
{
"query": "q34",
"net": 697613263,
"runs": [779471590, 775653528, 782402582]
},
{
"query": "q35",
"net": 215031035,
"runs": [307080719, 293071300, 337445189]
},
{
"query": "q36",
"net": 429593955,
"runs": [514337619, 509244140, 507634220]
},
{
"query": "q37",
"net": 88197302,
"runs": [180336420, 166237567, 173174879]
},
{
"query": "q38",
"net": 187898092,
"runs": [267594687, 265938357, 269324026]
},
{
"query": "q39",
"net": 908767319,
"runs": [1023851467, 986807584, 1001875938]
},
{
"query": "q40",
"net": 68201989,
"runs": [154972951, 146242254, 147902136]
},
{
"query": "q41",
"net": 62543633,
"runs": [145026872, 140583898, 148454573]
},
{
"query": "q42",
"net": 150607225,
"runs": [231513245, 232694981, 228647490]
}
]
}
81 changes: 81 additions & 0 deletions reports/2026-10-07/6005c2b1-server3-instructions-clickbench-duckdb-smoke.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# instructions-clickbench-duckdb on server3, 2026-10-07

**This is a smoke run.** It shows that the driver works. It ran on a small scale or on a shared machine, so its numbers are not baselines and no claim may use them.

| Field | Value |
|---|---|
| suite | instructions-clickbench-duckdb |
| machine | server3 |
| date | 2026-10-07 |
| commit | 6005c2b1 |
| smoke | true |
| note | smoke run: it shows that the runner works, it is not the ratchet |
| set | clickbench |
| engine_name | duckdb |
| queries_from | duckdb-queries.sql |
| repeat | 3 |
| kernel | 6.8.0-106-generic |

## engine

| Field | Value |
|---|---|
| mode | in process |
| client | /root/work/rupg-bench-opt/bin/duckdb, -readonly, hits1m.db, -cmd, SET threads = 1, -c |

## counts

| Field | Value |
|---|---|
| base_runs | 80916556, 78040265, 78382784 |
| base | 78040265 |
| total_net | 18064983237 |

## queries

| query | net | runs |
|---|---|---|
| q0 | 2658966 | 81782230, 81926639, 80699231 |
| q1 | 9794033 | 87834298, 88071044, 98749864 |
| q2 | 21379579 | 116705717, 100057642, 99419844 |
| q3 | 30056892 | 109395308, 108097157, 110024982 |
| q4 | 96061769 | 174102034, 176684910, 174479860 |
| q5 | 142365074 | 229474235, 220405339, 221811745 |
| q6 | 6391527 | 84431792, 91324585, 138004129 |
| q7 | 12274205 | 90314470, 98506520, 90877270 |
| q8 | 141955649 | 229414983, 220329937, 219995914 |
| q9 | 260171263 | 338211528, 374255484, 338243517 |
| q10 | 79267130 | 158599180, 157307395, 158798206 |
| q11 | 85726959 | 165710997, 163767224, 164860287 |
| q12 | 104967374 | 184804364, 184240163, 183007639 |
| q13 | 126302995 | 206476167, 205229570, 204343260 |
| q14 | 116660750 | 208069401, 194701015, 195197372 |
| q15 | 114154184 | 193169086, 192194449, 193594877 |
| q16 | 238392237 | 316985871, 380977329, 316432502 |
| q17 | 232148198 | 324993525, 310188463, 314307544 |
| q18 | 490274337 | 577124723, 568314602, 588599687 |
| q19 | 7260510 | 108010818, 86214489, 85300775 |
| q20 | 474816252 | 602058081, 557769957, 552856517 |
| q21 | 443819841 | 524793576, 521860106, 535583770 |
| q22 | 311919457 | 433448618, 389959722, 411982077 |
| q23 | 780471101 | 910784254, 858511366, 874619848 |
| q24 | 124617635 | 202796815, 222973585, 202657900 |
| q25 | 97594025 | 175634290, 192633348, 176087162 |
| q26 | 131454751 | 209495016, 218807746, 210925889 |
| q27 | 429523741 | 507564006, 517167810, 508441273 |
| q28 | 8580483958 | 8675722436, 8668755425, 8658524223 |
| q29 | 69401430 | 148583472, 150163834, 147441695 |
| q30 | 136512966 | 216352389, 229079759, 214553231 |
| q31 | 169098914 | 256408322, 258144163, 247139179 |
| q32 | 552858320 | 636146474, 630898585, 635693870 |
| q33 | 635693402 | 768820987, 714799322, 713733667 |
| q34 | 697613263 | 779471590, 775653528, 782402582 |
| q35 | 215031035 | 307080719, 293071300, 337445189 |
| q36 | 429593955 | 514337619, 509244140, 507634220 |
| q37 | 88197302 | 180336420, 166237567, 173174879 |
| q38 | 187898092 | 267594687, 265938357, 269324026 |
| q39 | 908767319 | 1023851467, 986807584, 1001875938 |
| q40 | 68201989 | 154972951, 146242254, 147902136 |
| q41 | 62543633 | 145026872, 140583898, 148454573 |
| q42 | 150607225 | 231513245, 232694981, 228647490 |

Loading
Loading