Skip to content

Commit 76bd5f6

Browse files
SouravRoy-ETLSouravRoy-ETL
committed
docs: feature ClickBench-43 result (33 of 43 vs DuckDB)
Adds the ClickBench benchmark as the headline result on the README and the slothdb.org landing page: SlothDB is ahead of DuckDB on 33 of 43 queries over the 100M-row hits dataset, measured on a 6-core laptop. - README.md: new ClickBench section after the hero, with the outcome donut and per-query speedup charts. - docs/index.html: ClickBench block at the top of the benchmarks section; meta, og, twitter, and JSON-LD descriptions updated. - docs/assets/benchmarks/: two charts generated by the new bench/clickbench/make_charts.py from the verify run. The 8 losses and 2 timeouts are shown, not hidden. Copy states the hardware and that this is not an official ClickBench submission.
1 parent 825aeee commit 76bd5f6

5 files changed

Lines changed: 190 additions & 6 deletions

File tree

‎README.md‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,41 @@
3131

3232
---
3333

34+
## ClickBench: 33 of 43 queries beat DuckDB
35+
36+
A from-scratch C++ embedded database, ahead of DuckDB on the majority of
37+
the industry-standard analytical benchmark.
38+
39+
<div align="center">
40+
<img src="docs/assets/benchmarks/clickbench_outcomes.png" alt="ClickBench-43: SlothDB vs DuckDB" width="60%">
41+
</div>
42+
43+
[ClickBench](https://github.com/ClickHouse/ClickBench) is the standard
44+
analytical-database benchmark: 43 queries over the 100M-row, 14 GB `hits`
45+
Parquet dataset. SlothDB runs all 43 head to head against DuckDB and comes
46+
out ahead on **33 of 43** on a 6-core laptop. Of those 33, SlothDB is
47+
genuinely faster on 25; the other 8 are queries DuckDB rejects outright on
48+
a type-strictness difference and SlothDB runs. Standout margins: Q24 about
49+
10x, Q18 about 8x, Q7 about 5x, Q1 about 3x faster than DuckDB.
50+
51+
<div align="center">
52+
<img src="docs/assets/benchmarks/clickbench_speedup.png" alt="ClickBench-43 per-query speedup vs DuckDB" width="100%">
53+
</div>
54+
55+
The 8 queries SlothDB loses are shown in red, not hidden: high-cardinality
56+
two-column `GROUP BY` (Q31 at 0.29x, Q32 at 0.12x) is the current weak
57+
spot, and two more (Q29, Q33) run past 30s. The whole suite reproduces
58+
from `bench/clickbench/`, which holds the 43 queries verbatim from the
59+
ClickBench repo plus the runner. Warm-cache, min-of-3, on a Ryzen 5 5600U
60+
laptop. This is not an official ClickBench submission; ClickBench timings
61+
are hardware-specific, so run it on yours:
62+
63+
```bash
64+
python bench/clickbench/verify_all.py
65+
```
66+
67+
---
68+
3469
## Ask in any language. Get SQL.
3570

3671
Type `.ask` at the `slothdb>` prompt. A rules parser handles catalog questions and common English shapes in under 10 ms with no model. Anything else falls through to a local Qwen2.5-Coder (0.5B for simple, 1.5B for analytic; lazy-downloaded on first use under `-DSLOTHDB_ASK_MODEL=ON`), which speaks 29 natural languages: English, Chinese, Spanish, French, German, Japanese, Korean, Russian, Arabic, Portuguese, Italian, Hindi, and more. Every generated statement is shown before it runs. Nothing leaves the machine. Set `SLOTHDB_ASK_CONFIRM=1` to add a `[Y/n]` prompt before each run.
@@ -293,7 +328,7 @@ Median speedup: 1.70×. Range: 1.04× - 5.43×.
293328

294329
</details>
295330

296-
Caveats worth knowing: Parquet aggregates are within ~20 % of DuckDB on most queries - both engines saturate the columnar fast path there, so don't expect 3× on Parquet. The big gaps come from SlothDB's native decoders (Avro, CSV `COUNT(*)`) and the 0.1.6 JOIN hot path. We have not submitted to ClickBench yet - on the roadmap.
331+
Caveats worth knowing: Parquet aggregates are within ~20 % of DuckDB on most queries - both engines saturate the columnar fast path there, so don't expect 3× on Parquet. The big gaps come from SlothDB's native decoders (Avro, CSV `COUNT(*)`) and the 0.1.6 JOIN hot path. Full ClickBench-43 results (33 of 43 vs DuckDB) are in the [ClickBench section](#clickbench-33-of-43-queries-beat-duckdb) near the top.
297332

298333
The architectural decisions behind the numbers (typed columnar decode, per-worker buffer reuse, fused scan+aggregate, zero-copy VARCHAR, vectorized filter, parallel CSV aggregate, typed int64 JOIN hash path) are in [CHANGELOG.md](CHANGELOG.md) with a commit per optimization.
299334

‎bench/clickbench/make_charts.py‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
"""Generate ClickBench-43 benchmark charts from a verify run.
2+
3+
Reads the table written by verify_all.py and renders two PNGs into
4+
docs/assets/benchmarks/:
5+
clickbench_outcomes.png - outcome breakdown across the 43 queries
6+
clickbench_speedup.png - per-query speedup vs DuckDB
7+
8+
Numbers are exactly what the verify run measured. Queries SlothDB loses
9+
are shown, not hidden.
10+
"""
11+
import re
12+
import sys
13+
from pathlib import Path
14+
15+
import matplotlib
16+
matplotlib.use("Agg")
17+
import matplotlib.pyplot as plt
18+
19+
ROOT = Path(__file__).resolve().parents[2]
20+
SRC = ROOT / "_private/orchestrator/phase2_clickbench43_verify.md"
21+
OUT = ROOT / "docs/assets/benchmarks"
22+
OUT.mkdir(parents=True, exist_ok=True)
23+
24+
GREEN = "#2e9e5b"
25+
TEAL = "#4bb5a0"
26+
RED = "#d4543f"
27+
GREY = "#9aa0a6"
28+
INK = "#1d2021"
29+
HW = "ClickBench hits.parquet, 100M rows . 6-core laptop . warm cache, min-of-3"
30+
31+
32+
def parse(md_path):
33+
rows = []
34+
for line in md_path.read_text(encoding="utf-8").splitlines():
35+
if not line.startswith("|"):
36+
continue
37+
cells = [c.strip() for c in line.strip().strip("|").split("|")]
38+
if len(cells) < 5 or not cells[0].isdigit():
39+
continue
40+
num = int(cells[0])
41+
status = cells[1]
42+
sloth = re.sub(r"[^\d.]", "", cells[2]) or None
43+
duck = re.sub(r"[^\d.]", "", cells[3]) or None
44+
ratio = re.sub(r"[^\d.]", "", cells[4]) or None
45+
rows.append({
46+
"q": num,
47+
"status": status,
48+
"sloth": float(sloth) if sloth else None,
49+
"duck": float(duck) if duck else None,
50+
"ratio": float(ratio) if ratio else None,
51+
})
52+
return rows
53+
54+
55+
def outcomes_chart(rows):
56+
win = sum(r["status"] == "WIN" for r in rows)
57+
win_df = sum(r["status"] == "WIN_DF" for r in rows)
58+
loss = sum(r["status"] == "LOSS" for r in rows)
59+
timeout = sum(r["status"] in ("TIMEOUT", "ERROR", "RUNTIME", "PARSE_ERROR")
60+
for r in rows)
61+
62+
labels = [
63+
f"SlothDB faster ({win})",
64+
f"DuckDB rejects the query, SlothDB runs it ({win_df})",
65+
f"DuckDB faster ({loss})",
66+
f"SlothDB over 30s ({timeout})",
67+
]
68+
sizes = [win, win_df, loss, timeout]
69+
colors = [GREEN, TEAL, RED, GREY]
70+
71+
fig, ax = plt.subplots(figsize=(8.4, 5.4))
72+
wedges, _ = ax.pie(sizes, colors=colors, startangle=90,
73+
counterclock=False,
74+
wedgeprops=dict(width=0.42, edgecolor="white",
75+
linewidth=2))
76+
ax.text(0, 0.08, f"{win + win_df}/43", ha="center", va="center",
77+
fontsize=34, fontweight="bold", color=INK)
78+
ax.text(0, -0.16, "SlothDB ahead", ha="center", va="center",
79+
fontsize=12, color=GREY)
80+
ax.legend(wedges, labels, loc="center", bbox_to_anchor=(0.5, -0.13),
81+
ncol=2, frameon=False, fontsize=9.5)
82+
ax.set_title("ClickBench-43: SlothDB vs DuckDB",
83+
fontsize=15, fontweight="bold", color=INK, pad=14)
84+
fig.text(0.5, 0.045, HW, ha="center", fontsize=8, color=GREY)
85+
fig.savefig(OUT / "clickbench_outcomes.png", dpi=150,
86+
bbox_inches="tight", facecolor="white")
87+
plt.close(fig)
88+
return win, win_df, loss, timeout
89+
90+
91+
def speedup_chart(rows):
92+
comp = [r for r in rows if r["status"] in ("WIN", "LOSS") and r["ratio"]]
93+
comp.sort(key=lambda r: r["ratio"], reverse=True)
94+
labels = [f"Q{r['q']}" for r in comp]
95+
ratios = [r["ratio"] for r in comp]
96+
colors = [GREEN if x >= 1.0 else RED for x in ratios]
97+
98+
fig, ax = plt.subplots(figsize=(13, 5))
99+
ax.bar(range(len(ratios)), ratios, color=colors, width=0.78)
100+
ax.axhline(1.0, color=INK, linewidth=1, linestyle="--")
101+
ax.set_yscale("log")
102+
ax.set_yticks([0.25, 0.5, 1, 2, 4, 8])
103+
ax.set_yticklabels(["0.25x", "0.5x", "1x", "2x", "4x", "8x"])
104+
ax.set_xticks(range(len(labels)))
105+
ax.set_xticklabels(labels, rotation=90, fontsize=8)
106+
ax.set_ylabel("Speedup vs DuckDB (higher is better)", fontsize=10)
107+
ax.set_xlim(-0.7, len(ratios) - 0.3)
108+
nwin = sum(x >= 1.0 for x in ratios)
109+
ax.set_title(f"Per-query speedup, {len(ratios)} head-to-head queries: "
110+
f"SlothDB faster on {nwin}, DuckDB faster on "
111+
f"{len(ratios) - nwin}",
112+
fontsize=13, fontweight="bold", color=INK, pad=10)
113+
for s in ("top", "right"):
114+
ax.spines[s].set_visible(False)
115+
fig.text(0.5, -0.02, HW, ha="center", fontsize=8, color=GREY)
116+
fig.savefig(OUT / "clickbench_speedup.png", dpi=150,
117+
bbox_inches="tight", facecolor="white")
118+
plt.close(fig)
119+
120+
121+
def main():
122+
if not SRC.exists():
123+
sys.exit(f"verify table not found: {SRC}")
124+
rows = parse(SRC)
125+
if len(rows) != 43:
126+
print(f"warning: parsed {len(rows)} rows, expected 43")
127+
win, win_df, loss, timeout = outcomes_chart(rows)
128+
speedup_chart(rows)
129+
print(f"parsed {len(rows)} queries")
130+
print(f" SlothDB faster {win}")
131+
print(f" DuckDB rejects, run {win_df}")
132+
print(f" DuckDB faster {loss}")
133+
print(f" timeout {timeout}")
134+
print(f"wrote {OUT}/clickbench_outcomes.png")
135+
print(f"wrote {OUT}/clickbench_speedup.png")
136+
137+
138+
if __name__ == "__main__":
139+
main()

‎docs/assets/benchmarks/clickbench_outcomes.png‎

73.6 KB
Loading

‎docs/assets/benchmarks/clickbench_speedup.png‎

65.2 KB
Loading

‎docs/index.html‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<meta charset="UTF-8">
55
<meta name="viewport" content="width=device-width, initial-scale=1.0">
66
<title>SlothDB · Run analytics faster.</title>
7-
<meta name="description" content="SlothDB is an embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. Built from scratch. Up to 5x faster where it counts. 138 ms vs 540 ms (3.9x) on the warm 5-query batch, 5.43x on Avro SUM. MIT.">
7+
<meta name="description" content="SlothDB is an embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. Built from scratch in C++. Ahead of DuckDB on 33 of 43 ClickBench queries on a 6-core laptop. 138 ms vs 540 ms (3.9x) on the warm 5-query batch, 5.43x on Avro SUM. MIT.">
88
<meta name="keywords" content="DuckDB alternative, in-process SQL database, embedded analytical database, Parquet query, CSV SQL, DuckDB replacement, SQLite OLAP, analytical database C++, fast Parquet reader, query Parquet with SQL">
99
<meta name="author" content="SlothDB">
1010
<meta name="robots" content="index, follow">
@@ -13,7 +13,7 @@
1313
<meta property="og:type" content="website">
1414
<meta property="og:url" content="https://slothdb.org/">
1515
<meta property="og:title" content="SlothDB - Run analytics faster.">
16-
<meta property="og:description" content="An embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. Built from scratch. Up to 5x faster where it counts. 138 ms vs DuckDB 1.1.3's 540 ms (3.9x) on the 5-query warm batch; 5.43x on Avro SUM. MIT.">
16+
<meta property="og:description" content="An embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. Built from scratch in C++. Ahead of DuckDB on 33 of 43 ClickBench queries on a 6-core laptop. 138 ms vs DuckDB 1.1.3's 540 ms (3.9x) on the 5-query warm batch; 5.43x on Avro SUM. MIT.">
1717
<meta property="og:image" content="https://slothdb.org/assets/benchmarks/speedup.png">
1818
<meta property="og:image:width" content="1000">
1919
<meta property="og:image:height" content="520">
@@ -22,7 +22,7 @@
2222
<meta name="twitter:card" content="summary_large_image">
2323
<meta name="twitter:url" content="https://slothdb.org/">
2424
<meta name="twitter:title" content="SlothDB - Run analytics faster.">
25-
<meta name="twitter:description" content="An embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. Built from scratch. Up to 5x faster. 138 ms vs 540 ms on the 5-query warm batch; 5.43x on Avro SUM. MIT.">
25+
<meta name="twitter:description" content="An embedded SQL database that runs everywhere: on your laptop, on a server, and in the browser. Built from scratch in C++. Ahead of DuckDB on 33 of 43 ClickBench queries on a 6-core laptop. 138 ms vs 540 ms on the 5-query warm batch; 5.43x on Avro SUM. MIT.">
2626
<meta name="twitter:image" content="https://slothdb.org/assets/benchmarks/speedup.png">
2727

2828
<link rel="canonical" href="https://slothdb.org/">
@@ -77,7 +77,7 @@
7777
"@type": "SoftwareApplication",
7878
"name": "SlothDB",
7979
"alternateName": ["slothdb"],
80-
"description": "Embedded in-process SQL OLAP database in C++20, built from scratch as a faster alternative to DuckDB. CREATE LIVE VIEW with incremental CSV append, sub-1 MB edge WASM build, seven file formats (Parquet, CSV, JSON, Avro, Excel, Arrow, SQLite) in the core binary vs DuckDB's three, stable C ABI. 5-query warm JOIN batch: 138 ms vs DuckDB 1.1.3's 540 ms (3.9x); 5.43x peak on Avro SUM; 16-query suite median 1.70x, range 1.04x-5.43x. Also ships optional .ask natural-language sub-REPL.",
80+
"description": "Embedded in-process SQL OLAP database in C++20, built from scratch as an alternative to DuckDB. Ahead of DuckDB on 33 of 43 ClickBench queries on a 6-core laptop. CREATE LIVE VIEW with incremental CSV append, sub-1 MB edge WASM build, seven file formats (Parquet, CSV, JSON, Avro, Excel, Arrow, SQLite) in the core binary vs DuckDB's three, stable C ABI. 5-query warm JOIN batch: 138 ms vs DuckDB 1.1.3's 540 ms (3.9x); 5.43x peak on Avro SUM; 16-query suite median 1.70x, range 1.04x-5.43x. Also ships optional .ask natural-language sub-REPL.",
8181
"applicationCategory": "DatabaseApplication",
8282
"operatingSystem": "Windows, macOS, Linux",
8383
"url": "https://slothdb.org/",
@@ -516,6 +516,16 @@ <h2>Try it in 60 seconds</h2>
516516
</section>
517517

518518
<section class="page" id="benchmarks">
519+
<h2>33 of 43 ClickBench queries beat DuckDB</h2>
520+
<p class="section-lede">SlothDB is a from-scratch C++ embedded database, ahead of DuckDB on the majority of the industry-standard analytical benchmark. ClickBench runs 43 queries over the 100M-row, 14 GB <code>hits</code> Parquet dataset; SlothDB wins 33 of them on a 6-core laptop. Of those 33, it is genuinely faster on 25, and runs 8 more that DuckDB rejects outright on a type-strictness difference. Standout margins: Q24 about 10x, Q18 about 8x, Q7 about 5x, Q1 about 3x faster.</p>
521+
<div style="margin: 26px auto 8px; max-width: 1060px; text-align:center;">
522+
<img src="assets/benchmarks/clickbench_outcomes.png" alt="ClickBench-43: SlothDB vs DuckDB outcomes" style="max-width: 440px; width: 100%;">
523+
<img src="assets/benchmarks/clickbench_speedup.png" alt="ClickBench-43 per-query speedup vs DuckDB" style="width: 100%; margin-top: 14px;">
524+
</div>
525+
<p style="max-width:760px; margin: 12px auto 56px; font-size:0.9em; opacity:0.75;">
526+
The 8 queries SlothDB loses are shown in red, not hidden: high-cardinality two-column GROUP BY (Q31 at 0.29x, Q32 at 0.12x) is the current weak spot, and two more (Q29, Q33) run past 30s. Reproduce the whole suite with <code>python bench/clickbench/verify_all.py</code> from the repo. Warm-cache, min-of-3, Ryzen 5 5600U laptop. Not an official ClickBench submission; ClickBench timings are hardware-specific.
527+
</p>
528+
519529
<h2>Three proof points, not sixteen</h2>
520530
<p class="section-lede">1M-row datasets · warm cache · 5-run median · <a href="https://github.com/SouravRoy-ETL/slothdb#performance--three-stories-not-sixteen">reproducible locally</a> via <code>pip install slothdb &amp;&amp; python -c "import slothdb; slothdb.demo()"</code>.</p>
521531

@@ -569,7 +579,7 @@ <h3>Avro - native decode</h3>
569579
</details>
570580

571581
<p style="max-width:740px; margin:28px auto 0; font-size:0.9em; opacity:0.75;">
572-
Tested on 1 M-row datasets on a single workstation. Parquet aggregates are within ~20% of DuckDB on most queries - both engines saturate the columnar fast path there. The big gaps (Avro, CSV <code>COUNT(*)</code>, the JOIN batch) come from SlothDB's native decoders and the 0.1.6 JOIN hot path. We haven't submitted to ClickBench yet - on the roadmap.
582+
Tested on 1 M-row datasets on a single workstation. Parquet aggregates are within ~20% of DuckDB on most queries - both engines saturate the columnar fast path there. The big gaps (Avro, CSV <code>COUNT(*)</code>, the JOIN batch) come from SlothDB's native decoders and the 0.1.6 JOIN hot path. Full ClickBench-43 results are shown at the top of this section.
573583
</p>
574584
</section>
575585

0 commit comments

Comments
 (0)