Skip to content

Add the machine scripts and the install scripts of the other engines - #20

Merged
tamnd merged 4 commits into
mainfrom
machines
Oct 7, 2026
Merged

tamnd merged 4 commits into
mainfrom
machines

Conversation

@tamnd

@tamnd tamnd commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Part of #1.

What changes

  • Add machines/instance.sh NAME SUITE [KEY=VALUE...]. It starts the AWS machine 4xl (c6a.4xlarge), metal (c6a.metal), tpch (c6a.4xlarge, or INSTANCE_TYPE for SF100) or oltp (INSTANCE_TYPE or HOST, because spec/20 §20.2 has not chosen it). It uses Ubuntu 24.04 and a 500 GB gp2 root volume, logs the hardware, copies the repository, runs run-suite.sh, copies the reports back and terminates the machine on exit. HOST runs on a machine that exists, and DRY_RUN=1 prints the AWS command.
  • Add machines/run-suite.sh SUITE [MACHINE] for pgbench, tpch, tpcc, ycsb, clickbench and fdatasync. It builds the harness, installs what the suite needs from the pins and runs the driver. SMOKE=1 runs a tiny scale and marks every report as a smoke run. A full run of tpcc stops when WAREHOUSES or VU is not set, and a full run of ycsb stops when RECORDS is not set, because spec/20 leaves these sizes to M0. SERVER_CPUS and CLIENT_CPUS set the cpuset of spec/20 §20.2.
  • Add machines/fdatasync.sh DIR [SIZE]. It runs fio with 4 KiB writes and an fdatasync after each write, and prints the p50, p95 and p99.
  • Add the install scripts clickhouse.sh, sqlite.sh, umbra.sh, cedardb.sh and harness.sh in machines/install/. Each one checks the hash of the download against the pin and checks the version that the program reports.
  • Add digest to [umbra] in pins.toml, because the tag 26.09 can move. umbra.sh stops when the pulled digest is not the pin. src/pins.rs requires the digest.
  • The report of a YCSB run with --sync has the setting in its name, and the report of a TPC-C run has the warehouse count, so the runs of one suite do not write the same file.
  • The README has a section on the machine scripts.
  • Smoke reports in reports/2026-10-07/. They are smoke runs and not baselines.

Test

All runs were on server3 (8 cores, shared, load average about 60 to 70) on 2026-10-07, with HOST=server3 MACHINE=server3. Times are UTC.

  • shellcheck -x passes for every script in machines/. srvtest passed on server3 at the head of this branch: fmt, clippy, cargo deny and 66 tests.
  • On 2026-10-07 at 15:53, DRY_RUN=1 machines/instance.sh 4xl clickbench and the same for metal, tpch and oltp printed aws ec2 run-instances with the instance types c6a.4xlarge, c6a.metal, c6a.4xlarge and the test value of INSTANCE_TYPE. oltp with no INSTANCE_TYPE and no HOST stops with an error, as it must. I could not run the AWS part, because we have no AWS account.
  • clickhouse.sh installed 26.9.12.8 at 14:19 in 63.8 s. sqlite.sh installed 3.53.4 at 14:20. umbra.sh pulled umbradb/umbra:26.09 at 14:20 to 14:23 and the digest matched the pin. sqlite.sh with a wrong hash in the pin stopped with exit 1, as it must. I did not run cedardb.sh on server3, because it installs a systemd unit on a shared machine.
  • instance.sh 4xl fdatasync SMOKE=1 at 14:27 to 14:33: 4095 syncs of 4 KiB in /var/lib/postgresql, p50 6324.2 us, p95 66846.7 us, p99 133693.4 us. server3 has no local NVMe, so these numbers say nothing about a reference machine.
  • instance.sh 4xl pgbench SMOKE=1 at 14:44 to 14:46: 1045 transactions, 34.98 tps, and the consistency check passed.
  • instance.sh 4xl ycsb SMOKE=1 at 14:47 to 14:51: every row had 0 wrong reads, with synchronous_commit on and off.
  • instance.sh 4xl tpch SMOKE=1 at 14:51 to 15:02, SF0.01: PostgreSQL hot 3.602 s and cold 19.776 s. DuckDB hot 0.935 s and cold 4.820 s, with 0 of 22 answers wrong against the PostgreSQL answers.
  • These four ran at commit 2b41dd83, before the rebase on Add the ClickBench driver #19, so the reports have that commit in their names.
  • instance.sh 4xl clickbench SMOKE=1 HITS_DIR=/root/work/data at 15:20 to 15:50, at fb76dcf, on the first 100,000 rows. PostgreSQL: load 34.3 s, hot total 19.269 s, cold total 76.904 s, 0 failed, concurrent 2.733 queries per second with error ratio 0.000. DuckDB: load 3.9 s, hot total 1.594 s, cold total 9.132 s, 0 failed, concurrent 3.500 queries per second with error ratio 0.870. A smoke run does not compare the DuckDB answers, because the parquet sample is not known to hold the same rows as the TSV sample.

Notes

  • The PostgreSQL data size of the ClickBench pin is du of the whole cluster, so the smoke report shows 1664993329 bytes for 100,000 rows. It counts the other databases of the cluster on server3.
  • The DuckDB concurrent error ratio comes from the file lock of the pin's duckdb/query, as described in Add the ClickBench driver #19.
  • The clickbench suite runs PostgreSQL and DuckDB only. The driver does not run ClickHouse, Umbra, CedarDB or SQLite yet, so the install scripts for them are ready but no suite uses them.
  • The machine line of M0: Baselines on the reference machines #1 stays open. The scripts are written and the install and run parts work on server3, but no AWS machine was started.

@tamnd tamnd added kind/feature It does not do something that it must do. area/machines Machine records, CPU pinning and noise control. labels Oct 7, 2026
@tamnd tamnd self-assigned this Oct 7, 2026
@tamnd
tamnd merged commit f39a2c4 into main Oct 7, 2026
1 of 2 checks passed
@tamnd
tamnd deleted the machines branch October 7, 2026 15:54
@tamnd tamnd mentioned this pull request Oct 7, 2026
5 of 17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/machines Machine records, CPU pinning and noise control. kind/feature It does not do something that it must do.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant