Repository navigation
Conversation
5 of 17 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1.
What changes
machines/instance.sh NAME SUITE [KEY=VALUE...]. It starts the AWS machine4xl(c6a.4xlarge),metal(c6a.metal),tpch(c6a.4xlarge, orINSTANCE_TYPEfor SF100) oroltp(INSTANCE_TYPEorHOST, because spec/20 §20.2 has not chosen it). It uses Ubuntu 24.04 and a 500 GB gp2 root volume, logs the hardware, copies the repository, runsrun-suite.sh, copies the reports back and terminates the machine on exit.HOSTruns on a machine that exists, andDRY_RUN=1prints the AWS command.machines/run-suite.sh SUITE [MACHINE]forpgbench,tpch,tpcc,ycsb,clickbenchandfdatasync. It builds the harness, installs what the suite needs from the pins and runs the driver.SMOKE=1runs a tiny scale and marks every report as a smoke run. A full run oftpccstops whenWAREHOUSESorVUis not set, and a full run ofycsbstops whenRECORDSis not set, because spec/20 leaves these sizes to M0.SERVER_CPUSandCLIENT_CPUSset the cpuset of spec/20 §20.2.machines/fdatasync.sh DIR [SIZE]. It runsfiowith 4 KiB writes and anfdatasyncafter each write, and prints the p50, p95 and p99.clickhouse.sh,sqlite.sh,umbra.sh,cedardb.shandharness.shinmachines/install/. Each one checks the hash of the download against the pin and checks the version that the program reports.digestto[umbra]inpins.toml, because the tag26.09can move.umbra.shstops when the pulled digest is not the pin.src/pins.rsrequires the digest.--synchas the setting in its name, and the report of a TPC-C run has the warehouse count, so the runs of one suite do not write the same file.reports/2026-10-07/. They are smoke runs and not baselines.Test
All runs were on server3 (8 cores, shared, load average about 60 to 70) on 2026-10-07, with
HOST=server3 MACHINE=server3. Times are UTC.shellcheck -xpasses for every script inmachines/.srvtestpassed on server3 at the head of this branch: fmt, clippy, cargo deny and 66 tests.DRY_RUN=1 machines/instance.sh 4xl clickbenchand the same formetal,tpchandoltpprintedaws ec2 run-instanceswith the instance types c6a.4xlarge, c6a.metal, c6a.4xlarge and the test value ofINSTANCE_TYPE.oltpwith noINSTANCE_TYPEand noHOSTstops with an error, as it must. I could not run the AWS part, because we have no AWS account.clickhouse.shinstalled 26.9.12.8 at 14:19 in 63.8 s.sqlite.shinstalled 3.53.4 at 14:20.umbra.shpulledumbradb/umbra:26.09at 14:20 to 14:23 and the digest matched the pin.sqlite.shwith a wrong hash in the pin stopped with exit 1, as it must. I did not runcedardb.shon server3, because it installs a systemd unit on a shared machine.instance.sh 4xl fdatasync SMOKE=1at 14:27 to 14:33: 4095 syncs of 4 KiB in/var/lib/postgresql, p50 6324.2 us, p95 66846.7 us, p99 133693.4 us. server3 has no local NVMe, so these numbers say nothing about a reference machine.instance.sh 4xl pgbench SMOKE=1at 14:44 to 14:46: 1045 transactions, 34.98 tps, and the consistency check passed.instance.sh 4xl ycsb SMOKE=1at 14:47 to 14:51: every row had 0 wrong reads, withsynchronous_commiton and off.instance.sh 4xl tpch SMOKE=1at 14:51 to 15:02, SF0.01: PostgreSQL hot 3.602 s and cold 19.776 s. DuckDB hot 0.935 s and cold 4.820 s, with 0 of 22 answers wrong against the PostgreSQL answers.instance.sh 4xl clickbench SMOKE=1 HITS_DIR=/root/work/dataat 15:20 to 15:50, at fb76dcf, on the first 100,000 rows. PostgreSQL: load 34.3 s, hot total 19.269 s, cold total 76.904 s, 0 failed, concurrent 2.733 queries per second with error ratio 0.000. DuckDB: load 3.9 s, hot total 1.594 s, cold total 9.132 s, 0 failed, concurrent 3.500 queries per second with error ratio 0.870. A smoke run does not compare the DuckDB answers, because the parquet sample is not known to hold the same rows as the TSV sample.Notes
duof the whole cluster, so the smoke report shows 1664993329 bytes for 100,000 rows. It counts the other databases of the cluster on server3.duckdb/query, as described in Add the ClickBench driver #19.clickbenchsuite runs PostgreSQL and DuckDB only. The driver does not run ClickHouse, Umbra, CedarDB or SQLite yet, so the install scripts for them are ready but no suite uses them.