rupg

A PostgreSQL 19 compatible database in Rust. It runs as a server and as an embedded library, on one file.

A client that connects to PostgreSQL 19 connects to rupg with no change except the host, the port and the credentials. The same engine also runs inside the application process, with no server. A database is one .rupg file. Cold data is in compressed columns. Hot data is in row pages. The file format, the timestamps and the log are designed for shards and replicas from the start.

Status: M1. The file, the log, recovery and a Rust key-value API over typed tables work. The engine does not answer SQL yet. The milestones that build the engine are issues with the label kind/milestone.

The goal

The goal has five axes. Each axis has a number, a machine, a baseline and a gate. spec/02-the-goal.md gives the arithmetic for each number and says what must be true for it to hold.

Axis	Target	Spec
Compatibility	Pass the PostgreSQL 19 regression suite, the isolation suite and the suites of 48 clients at the rate of a real PostgreSQL 19 server. Shims emulate versions 14 to 18.	05
Analytic speed	ClickBench hot sum at most 1.753 s on c6a.4xlarge. That is one tenth of ClickHouse (17.53 s). TPC-H SF100 at 10x DuckDB.	02
Transactional speed	TPC-C and YCSB at one tenth of the server CPU per transaction of PostgreSQL, on the same machine.	02
Resources	One tenth of the peak memory and CPU time of each baseline. Disk below 7.67 GB on ClickBench.	19
Scale out	TPC-C on 2 to 16 nodes at 0.8 N or more of one node.	18

A 10x claim has five parts: one machine, one metric, a baseline that we measured on that machine, the same transport for both systems, and the ratio. A claim without all five parts is not made. The measurements are in rupg-bench. The compatibility levels are in rupg-compat.

Gates

Each gate is a measured number that decides if a milestone is done. When a gate fails, we publish the number, the gap and the cause. We do not change the target in the same change.

Gate	Suite	Number	Milestone
G1	L1 and L2	7 clients connect and show the schema: psql, pgAdmin 4, DBeaver, psycopg 3, node-postgres, pgx, asyncpg	M2
G2	ClickBench, c6a.4xlarge	hot sum below ClickHouse (17.53 s)	M4
G3	ClickBench, c6a.4xlarge	disk below 9.45 GB	M4
G4	YCSB A, B, C, F	3x server CPU per operation, no pipelining	M3
G5	TPC-C, statements	3x NOPM per core	M5
G6	TPC-C, procedures	10x NOPM per core	M7
G7	ClickBench, c6a.4xlarge	hot sum at most 1.753 s, cold sum at most 11.37 s	M8
G8	ClickBench, c6a.4xlarge	10x peak memory and CPU time, disk below 7.67 GB	M8
G9	TPC-H SF100	10x DuckDB hot total	M8
G10	ClickBench, c6a.metal	hot sum at most 0.440 s	M8
G11	TPC-C, 2 to 16 nodes	at least 0.8 N	M11
G12	L1 to L5, PostgreSQL 19 and shims 14 to 18	100 percent	M12

The design on one page

• One engine, four ways in. The server speaks protocol 3.0 and 3.2. The library opens a file in process from Rust, C (with libpq exports) or WebAssembly. A replica follows the log. The importer reads PostgreSQL data directories, pg_dump archives and logical replication. All four use the same parser, catalog, executor and file.
• One file. A .rupg file has no sidecar. Hot rows are in a B+ tree of PAX pages. Cold rows are in column segments of up to 262,144 rows with global dictionaries and per-segment summaries. Writes go out of place, so a torn page cannot damage committed data. (spec/08, spec/09, spec/10)
• Transactions. MVCC with undo, hybrid logical clock timestamps (48 bits of milliseconds and 16 logical bits), SSI, and one log ring per worker. A row id has a 16-bit shard and 48 local bits, so a later split does not rewrite rows. (spec/11)
• Execution. Thread per core with io_uring on Linux. Vectors of 1024 values. Operators run on the encoded data. Three compile tiers: an interpreter, a direct x86-64 tier and Cranelift. (spec/14)
• Indexes. All PostgreSQL access methods, plus USING link for graph adjacency and USING vector for DiskANN with RaBitQ, which hnsw and ivfflat map to. (spec/12, spec/13)
• Extensions. Built-in extensions are Rust code. There is no C extension ABI. (spec/16)
• Cluster. One Raft group per shard. A table is split into shards by a hash of its distribution key, as in PostgreSQL hash partitioning. An unchanged schema runs on shard 0. Distributed transactions commit at an HLC timestamp. (spec/18)

Building

You need Rust 1.98.0. The rust-toolchain.toml file selects it. The MSRV is 1.88.0.

cargo build --release
./target/release/rupg --version
./target/release/rupg --print-config
cargo test --workspace

rupg check app.rupg checks a database file and does not change it. It prints the counts of pages, tables, rows, log blocks and log bytes, or the first error with its SQLSTATE.

Run the same checks as CI before you push:

cargo fmt --all --check
cargo xtask layers
cargo xtask style
cargo xtask vendor --check
cargo xtask grammar
cargo clippy --workspace --all-targets --all-features

Repository layout

Path	Contents
crates/	The 41 crates of spec/22. Most are stubs until their milestone.
xtask/	The check tasks. layers.toml has the rank of each crate. A crate may depend only on a lower rank.
vendor/	Files from PostgreSQL, the time zone data and Unicode, at their pins. Each directory has a PIN file with the hashes.
spec/	The technical specification, 25 documents.
.github/	CI, the issue templates and Dependabot.

The specification

The design is written before it is built. Start with spec/00-README.md. The milestones are in spec/23-milestones.md. The open questions are in spec/24-open-questions.md. A path that starts with ../2140/ names a file in the rudb specification.

Sibling repositories

Repository	Purpose
tamnd/rupg-compat	The PostgreSQL oracles from 14 to 19, the runner and the suites that decide the compatibility levels L1 to L5.
tamnd/rupg-bench	The benchmark harness, the baselines and the reports that decide the performance gates.
tamnd/rudb	The analytical engine that rupg lifts kernels and encodings from.

Contributing

Read CONTRIBUTING.md. To report a wrong answer, use the wrong answer template. A wrong answer is the most serious kind of bug. To report a security problem, read SECURITY.md.

License

Apache-2.0. See LICENSE-APACHE. rupg is a clean room implementation. Do not read or copy code under the AGPL or another copyleft license into this repository.