M0: Baselines on the reference machines

[tamnd]

This milestone builds the benchmark harness and turns every published reference into a baseline that we measured. It runs PostgreSQL 19, ClickHouse and DuckDB on ClickBench, TPC-H, TPC-C and YCSB on the machines of the spec, with cgroup resource numbers for each run. At the end, each baseline number in spec/03 has a remeasured number in reports/.

Why

Every ratio in a rupg report has both numbers measured by us, on one machine, on the same day, with the same harness. A published number is a reference and never one side of a ratio. So no gate from M2 to M11 can be decided until this milestone has measured its baseline.

Some gates have no reference at all. ClickBench does not publish memory or CPU time, so G8 has no baseline until M0 measures one, and M4 needs it. The oltp machine and the warehouse counts of TPC-C are also chosen here, because a TPC-C run on a device with slow sync measures the device and not the engine.

The work

• Create the crate with the suite drivers, the cgroup runner, the answer sets, the report generator and the machine scripts (spec/20 §20.12)
• Write pins.toml with ClickBench at e6bda4e, PostgreSQL REL_19_STABLE at 7d3d2db7, rudb at cd9f9676, and the versions of ClickHouse, DuckDB, Umbra or CedarDB and SQLite (spec/20 §20.1, spec/03 §3.14)
• Write the cgroup v2 runner that reads memory.peak, file in memory.stat, Pss, usage_usec, io.stat and du --apparent-size --bytes, and the idle base over 10 s (spec/20 §20.4)
• Write the load case steps for case A and case B with drop_caches, fincore and the timed cat of the source file (spec/20 §20.3.1)
• Write the machine scripts that start a 4xl, metal or oltp instance, install the pinned versions, run a suite and stop the instance (spec/20 §20.12)
• Choose the oltp machine: at least 32 cores and local NVMe with power loss protection, with an fdatasync p50 below 100 µs measured with fio (spec/20 §20.2)
• Choose the tpch machine for SF100: at least 64 vCPU, 256 GiB and local NVMe (spec/20 §20.2)
• Run ClickBench on 4xl and metal, hot, cold and load, for PostgreSQL 19, PostgreSQL 19 tuned, ClickHouse, DuckDB and Umbra or CedarDB where the license permits (spec/03 §3.15)
• Compute the PostgreSQL 19 answer sets for ClickBench and TPC-H once on metal and store them with the harness (spec/20 §20.5.2)
• Run TPC-H at SF1, SF10 and SF100 for DuckDB and PostgreSQL 19, hot and cold, with dbgen 3.0.1 and qgen -d (spec/03 §3.15, spec/20 §20.7)
• Choose the two TPC-C warehouse counts and the virtual users from the memory of oltp, and run HammerDB for PostgreSQL 19 with procedures and with statements (spec/20 §20.8)
• Run YCSB A, B, C and F for PostgreSQL 19 with and without pipelining, and choose the record count (spec/20 §20.9)
• Measure the commit curve and the fdatasync latency of the gate machine (spec/03 §3.15)
• Measure the idle memory of PostgreSQL 19 at 1,000 connections with the same shared_buffers as rupg (spec/03 §3.15)
• Add the pgbench smoke run with its consistency check (spec/21 §21.4.6)
• Write the instruction count runner with the fixed set of spec/21 §21.14 (spec/21 §21.14)
• Record each difference above 10 percent from the published number in the M0 report with its cause (spec/03 §3.15)

Exit criterion

1. Each baseline number in spec/03 has a remeasured number in reports/, with the exact version, the configuration file, the machine and the date.
2. Each remeasured number that differs from the published one by more than 10 percent has its cause in the M0 report.
3. The memory and CPU time of each ClickBench baseline on 4xl are measured, so G8 has a reference before M4 starts.
4. The oltp machine has a measured fdatasync p50 below 100 µs, and the TPC-C warehouse counts, the virtual users and the YCSB record count are fixed in the report.
5. No gate of spec/02 §2.10 is decided here.

Engine milestone

• M0: The harness and the baselines rupg#1

Depends on

Nothing.

Spec

• spec/02 §2.2 How a 10x claim is made
• spec/03 §3.14 Versions
• spec/03 §3.15 The M0 remeasurement list
• spec/03 §3.16 What the baselines decide
• spec/20 §20.1 The rules
• spec/20 §20.2 The machines
• spec/20 §20.4 The resource measurements
• spec/20 §20.12 The harness
• spec/23 §23.3 M0: the harness and the baselines

[tamnd]

PR #10 merged. It adds pins.toml with ClickBench at e6bda4e, PostgreSQL REL_19_STABLE at 7d3d2db7, rudb at cd9f9676, ClickHouse 26.9.12.8, DuckDB 1.5.6, Umbra 26.09, CedarDB v2026-09-29, SQLite 3.53.4, HammerDB 6.0 and the TPC-H tools 3.0.1. Each version was looked up on 7 October 2026. rupg-bench pins prints them.

Two pins differ from spec/03 §3.14: DuckDB 2.0.0 is not released yet, so the pin is 1.5.6, and the newest SQLite is 3.53.4, not 3.53.1. The pins move to DuckDB 2.0.0 and PostgreSQL 19.0 when they exist.

Next: the cgroup v2 runner of spec/20 §20.4.

[tamnd]

#11 adds the cgroup v2 runner of spec/20 §20.4 as rupg-bench measure. It reads memory.peak, the file line of memory.stat, the summed Pss, usage_usec, io.stat and du --apparent-size --bytes, and it measures the idle base over 10 s with --idle. I tested it on server3 on 7 October 2026 with a DuckDB 1.5.6 load and query in a new cgroup, and with the idle base of a systemd unit. The CPU times agree with /usr/bin/time -v. The PR has the commands and the numbers.

server3 runs Linux 6.8, which cannot reset memory.peak. The result marks the peak as "since the cgroup was created" there, and the drivers start each system in a new cgroup for each suite. On Linux 6.12 and later the runner resets the peak at the start of the interval.

The PR also adds machines/lib.sh and the install scripts for DuckDB and for PostgreSQL at the pinned commit.

[tamnd]

#12 adds rupg-bench load with the steps of spec/20 §20.3.1. Case A drops the caches, checks with fincore that the source has 0 resident pages, times the cat of the source as R, drops and checks again, and times the load L. Case B reads the source twice and needs 99 percent of the pages resident, or the run becomes case A. I ran both cases on server3 on 7 October 2026 with a DuckDB 1.5.6 load of hits_0.parquet. These were smoke runs on a shared machine, not baselines. The PR has the commands and the output.

[tamnd]

#13 adds the answer set format, the answer check and the report generator. This is part of the line "Create the crate with the suite drivers, ...". That line stays open until the drivers and the machine scripts exist.

The check of #13 found a fact for the TPC-H driver. The built-in generator of DuckDB 1.5.6 does not make the data of dbgen 3.0.1. At SF1, 5 of the 22 answers differ from the official answer set (Q2, Q10, Q13, Q15 and Q20, in address strings and order comments). So the TPC-H driver must load the data of the pinned dbgen.

[tamnd]

#14 adds the pgbench driver and the consistency check of spec/21 §21.4.6. I ticked "Add the pgbench smoke run".

The run was on server3 on 7 October 2026, against PostgreSQL REL_19_STABLE at 7d3d2db7 built from source. The command was rupg-bench pgbench --unit postgresql@19-main --scale 1 --clients 4 --time 30 --smoke. pgbench processed 588 transactions with 0 failures. The three balance sums were 141710, and pgbench_history had 588 rows, so the check passed. This was a smoke run on a shared machine with a load average near 80, so its numbers are not baselines. The report is reports/2026-10-07/de0a7ce2-server3-pgbench-smoke.md.

[tamnd]

#15 adds the instruction count runner of spec/21 §21.14, rupg-bench instructions. It counts with perf stat -e instructions, keeps the smallest of 3 runs, and compares with ratchet.toml at 3 percent for one query and 1 percent for the total.

It ran on server3 on 7 October 2026 for the 43 ClickBench queries on 1,000,000 rows, as smoke runs. PostgreSQL REL_19_STABLE at 7d3d2db7 used 228,920,014,998 net instructions and DuckDB 1.5.6 used 18,064,983,237. The reports are in reports/2026-10-07/. The DuckDB runs spread by 4.45 percent at the median on this shared machine, which is above the 3 percent budget, so ratchet.toml has no table yet.

The line stays open. The fixed set also needs TPC-H at SF0.1, YCSB and TPC-C, and those drivers come next.

[tamnd]

#16 adds the TPC-H driver of spec/20 §20.7, rupg-bench tpch. It runs dbgen and qgen -d 3.0.1, loads the same .tbl files into DuckDB and PostgreSQL with the keys of the specification, runs each query once cold and twice hot, and checks the answers with the rules of clause 2.1.3.5.

Smoke runs on server3, 7 October 2026, with the reports in reports/2026-10-07/68cad8fc-server3-*. These are not baselines. DuckDB 1.5.6 at SF1 matched all 22 answers of the kit, with a hot total of 12.723 s. At SF0.1, DuckDB matched all 22 answers of PostgreSQL REL_19_STABLE at 7d3d2db7. The PostgreSQL hot total at SF0.1 was 720.1 s, mostly Q20 and Q17 with only the keys of the specification. The instruction runner also counted the 22 queries at SF0.1 on both engines, so the ClickBench and TPC-H parts of the fixed set of spec/21 §21.14 run now.

The TPC-H baseline line stays open because it needs the tpch machine. The instruction line stays open until the YCSB and TPC-C sets run.

[tamnd]

#17 adds the YCSB driver of spec/20 §20.9, rupg-bench ycsb. It runs the core workloads A, B, C and F with the scrambled zipfian distribution (0.99), at 1 client, 16 clients and 16 clients with a pipeline depth of 64. Each row has the server CPU for each operation, memory.peak and the latency quantiles. After each row the driver checks that every updated field holds an update that no later acknowledged update replaced.

A smoke run on server3 on 7 October 2026 with 100,000 records and 10 s for each row checked 12,000 fields with 0 wrong. The report is reports/2026-10-07/c7a1474-server3-ycsb-postgresql-smoke.md, and it is not a baseline. A trigger that drops 10 percent of the updates made the check fail, as it must.

The YCSB baseline line stays open because it needs the oltp machine.