M3: SQL and the point path

The fourth milestone of rupg, specified in spec/23-milestones.md section 23.6. M3 runs the SQL that applications run on rows: the full analyzer, INSERT, UPDATE, DELETE, MERGE and RETURNING, MVCC at READ COMMITTED and REPEATABLE READ, prepared statements and the plan cache. It also delivers the point path, the first optimizer, and version 1 of the vectorized executor on the hot store. Gate G4 closes here.

Why

The point path is at M3, not at M8. Gate G4 measures the CPU per operation. A slow protocol layer is hard to fix late, because its cost is spread over every message. The point path sets the cost floor while the code is still small.

M3 also completes the first SQL path that M2 shipped for the catalog queries of G1. The regression suite starts to count here, as a tracked pass count with a ratchet.

The work

Taken from spec/23 §23.6. The reference in brackets names the spec section that owns the detail. Change the list in the spec and copy it again. Do not edit it here.

Exit criterion

G4 (spec/02 §2.10): YCSB workloads A, B, C and F at 3x less server CPU per operation than PostgreSQL, without pipelining. spec/20 §20.13 runs it on the oltp machine with 16 clients against PostgreSQL 19, and the ratio must hold on every workload.
The report publishes the pass count of the 239 regression tests, at or above its ratchet.
sql_parse runs 48 CPU hours with no open find.
1 million more crash files with DML and DDL run with no lost commit.
The close report answers each line of the checklist in spec/23 §23.18 that applies from M3.

Open questions

These questions in spec/24 close at M3.

Q12 (spec/24 §24.3): max_parallel_workers_per_gather, an effective default that differs from the reported value or a gate run that sets it. Q12: What does max_parallel_workers_per_gather report? #25 Closed by Close Q12, Q13, Q21 and Q25 with the M3 evidence #1068: rupg reports 2, and a gate run sets the value and prints it.
Q13 (spec/24 §24.4): float aggregates in a different order, an exclusion for the last bits or PostgreSQL's serial order. Q13: Do float aggregates in a different order match the oracle? #26 Closed by Close Q12, Q13, Q21 and Q25 with the M3 evidence #1068: document 05 item 17 accepts the last bits only for a parallel or segment sum.
Q18 (spec/24 §24.4): the collation order of ICU4X against ICU4C, decided by a corpus test. Q18: Does ICU4X order strings as ICU4C does? #31
Q20 (spec/24 §24.4): math functions on other platforms, and correctly rounded functions in Rust. Q20: Do math functions give the same answers on other platforms? #33 Closed by Call cbrt and tgamma of the C library #1067: rupg calls the C math library of the platform, as PostgreSQL does.
Q21 (spec/24 §24.4): hash values that users can see, with two sets of hashes in rupg-kernels. Q21: Do hash values that users can see match PostgreSQL? #34 Closed by Close Q12, Q13, Q21 and Q25 with the M3 evidence #1068: the hashes that users see match PostgreSQL 19.
Q25 (spec/24 §24.4): relpages, bytes divided by 8,192 or a count that keeps external cost formulas as on PostgreSQL. Q25: What does relpages report? #38 Closed by Close Q12, Q13, Q21 and Q25 with the M3 evidence #1068: the page count of PostgreSQL for the same rows.
Q72 (spec/24 §24.9): facts not verified in two cited sources. Q72: Which facts are not verified? #85 Closed by Close Q72 with the release of self-join elimination #1060: PostgreSQL 18 added self-join elimination.

Harness work

Compatibility: M3: L3 tracked and the generated corpus rupg-compat#3
Benchmarks: M3: YCSB and the point path rupg-bench#3

Depends on

#3.

Spec

#391 is merged. It is the first step of the item for INSERT, UPDATE, DELETE, MERGE and RETURNING, and of the item for MVCC.

Tables hold rows now. INSERT with VALUES, with a query and with DEFAULT VALUES stores them, and SELECT reads them.
The defaults of the columns and of the domains, the not-null and check constraints, and the errors of transformInsertStmt match PostgreSQL 19 in connection/insert.test.
The rows are in memory with transaction IDs and snapshots. READ COMMITTED takes a snapshot for each statement, and REPEATABLE READ keeps one for the transaction. A rollback removes the rows.

The two items stay open. The next steps are the unique checks, UPDATE, DELETE, RETURNING, and the move of the rows to rupg-txn and the hot store.

#392 is merged. It adds UPDATE and DELETE on the heap tables. Each row version now has xmin, cmin, xmax and cmax. A change of a row that a running transaction changed waits with the wait event Lock/transactionid, and it has the deadlock check and lock_timeout. After a commit of the other transaction, read committed runs the statement again and repeatable read gives 40001. The output of a single-session script and of a two-session script is the same as the output of PostgreSQL 19, with the exception of the transaction IDs.

Known differences: the second run uses a new snapshot for the full statement and not EvalPlanQual. A table with a unique index, a foreign key, a trigger or a rule, RETURNING and WITH still give 0A000.

Next: the unique and primary key checks on the heap, then the foreign key checks and RETURNING.

#395 is merged. INSERT and UPDATE now check the unique indexes and the primary keys. A duplicate key gives 23505 with the detail Key (...)=(...) already exists., and a key that a transaction in progress holds makes the write wait, with the context while inserting index tuple. The heap keeps a hash map for each unique key, so a run of single-row inserts does not scan the full table. The output of a script of unique keys and of a two-session script is the same as the output of PostgreSQL 19, with the exception of the transaction IDs.

Known differences: a unique index on an expression, a partial unique index and a deferrable unique constraint still give 0A000.

Release 0.1.42 is out with INSERT, UPDATE and DELETE (#391 and #392).

Next: the foreign key checks, then RETURNING.

#396 adds RETURNING to INSERT, UPDATE and DELETE. The list can use *, expressions, subqueries, and the columns of FROM and USING. It also has the names old and new of PostgreSQL 18 and the option RETURNING WITH (OLD AS o, NEW AS n). The rows go to the client in the simple protocol, the extended protocol and EXECUTE, and Describe gives RowDescription for a statement with RETURNING. The test script was recorded from PostgreSQL 19 and gives the same output.

The checklist item for DML stays open until MERGE and ON CONFLICT are done.

Next: the foreign key checks, then ON CONFLICT.

#397 is merged. INSERT, UPDATE and DELETE now check the foreign keys after the statement, as the internal triggers of ri_triggers.c do. The checks support MATCH SIMPLE, MATCH FULL, NO ACTION and RESTRICT, a table that references itself, and keys of two types. The errors and their details are the same as in PostgreSQL. The session reads the other table one time for each foreign key and keeps its keys in a hash map.

The actions CASCADE, SET NULL and SET DEFAULT and the constraints that are INITIALLY DEFERRED still give 0A000. The row locks of FOR KEY SHARE are not made yet.

The checklist item for DML stays open. Next: the actions of the foreign keys, then ON CONFLICT.

Merged #1183. Deparse and EXPLAIN VERBOSE now name the temporary schema of the session pg_temp, as get_namespace_name_or_temp does. A WindowAgg below another WindowAgg now gives its input columns before its window calls, as create_one_window_path does. The explain regression diff goes from 423 to 386 lines, and the window, temp, create_view and triggers diffs do not change.

Next: HAVING false as a Result node with Replaces: Aggregate, and the temporary schema made on first use for CREATE FUNCTION pg_temp.f.

statement_timeout now stops a statement, and scans, joins, aggregates and the projection check for a cancel after each 1024 rows. (#1184)

Merged #1185. CREATE FUNCTION, CREATE DOMAIN, CREATE TYPE and CREATE AGGREGATE with the schema pg_temp now make the temporary schema of the session when it does not exist yet, as LookupCreationNamespace does. Before, they failed with schema "pg_temp" does not exist until the session made a temporary table. The diffs of 9 related regression tests do not change.

Note: catalog_statistics in rupg-session fails on main, also before #1183. It compares the pg_stats values of sql_parts. That area is in the ANALYZE work.

Next: HAVING false as a Result node with Replaces: Aggregate in EXPLAIN.

Merged #1186. A query with HAVING or grouping sets, but with no GROUP BY item and no aggregate, now plans as create_degenerate_grouping_paths does in PostgreSQL 19: a Result with Replaces: Aggregate and the HAVING conditions as its one-time filter, or an Append of one Result for each grouping set. The plan does not read FROM. The explain regression diff goes from 386 to 366 lines and the join diff from 4750 to 4625 lines. Other related tests do not change.

Next: the Query Identifier in EXPLAIN VERBOSE with compute_query_id, then the top "not supported yet" groups (ALTER, CREATE, XMLTABLE).

Merged #1187. EXPLAIN VERBOSE now shows Query Identifier after the plan when compute_query_id is on, as in PostgreSQL 19. With auto and regress it shows no line, as in PostgreSQL. The identifier is a hash of the tokens of the statement with each constant as the same token, so statements that differ only in their constants share an identifier. The numbers are not the numbers of PostgreSQL, which hashes its own query tree. The explain regression diff goes from 363 to 334 lines.

Next: pg_stat_activity.query_id with the same function, then the top "not supported yet" groups (ALTER, CREATE, XMLTABLE).

Merged #1188. pg_stat_activity.query_id now shows the query identifier of the running statement when compute_query_id is on or regress. It uses the same hash as EXPLAIN VERBOSE, so the two values match for the same statement. With the default auto the column stays null and rupg computes no hash, so the default path has no extra cost.

Next: the top "not supported yet" groups in the regression run (ALTER, CREATE, XMLTABLE), one category for each PR.

Merged #1189. A full parallel_schedule run on the current main gives 141 tests that pass and 98 that fail, and it found one crash. EXPLAIN ANALYZE of a nested loop that scans a Sort of a removed subquery again stopped the server process with an index out of bounds (select_parallel, with enable_material off). The row count of the subquery now starts again at each new scan, and the output is the same as PostgreSQL 19. The PR also fixes the rustfmt format of #1188.

The top "not supported yet" groups in this run: ALTER (112), pg_restore_*_stats (155, the other session's area), CREATE (46), XMLTABLE (36), test_enc_conversion (24), ICU collations (other session), pg_column_compression (22), xpath (20), brin_summarize_range (20), XMLSERIALIZE INDENT (20), DROP (20), TABLESPACE (17). Next: the ALTER, CREATE and DROP groups, split by statement type.

Aggregates and function calls use less time for each row: avg of integer keeps a 128-bit sum, and the transition and operator calls need no vectors. (#1190)

Merged #1191: the TABLESPACE clause of CREATE TABLE, PARTITION OF, CREATE TABLE AS, CREATE MATERIALIZED VIEW, CREATE INDEX and constraint indexes, with default_tablespace and pg_class.reltablespace.

The rules follow DefineRelation, DefineIndex and GetDefaultTablespace of PostgreSQL 19: the inherited tablespace of a partition, the check of CREATE on the tablespace, the pg_global error and the error for a partitioned relation in the default tablespace.
SET default_tablespace rejects a tablespace that does not exist.
DROP TABLESPACE gives "some objects depend on it" for a partitioned relation and "is not empty" for other relations.
The tablespace regression diff goes from 633 to 268 lines. The full schedule stays at 141 ok.

Next in the tablespace category: ALTER TABLE/INDEX/MATERIALIZED VIEW ... SET TABLESPACE, ALL IN TABLESPACE and REINDEX (TABLESPACE), then ALTER TABLESPACE.

A scan of some of the columns of a table finds the positions of the columns once, and not again for each row. (#1192)

Released 0.4.14 (#1193), with the changes from #1169 to #1192.

Merged #1194: ALTER TABLE/INDEX/MATERIALIZED VIEW ... SET TABLESPACE, ALTER ... ALL IN TABLESPACE and REINDEX (TABLESPACE).

SET TABLESPACE follows ATPrepSetTableSpace and ATExecSetTableSpace. The TOAST table and its index move with the table and get a new relfilenode. A second SET TABLESPACE in one statement is an error.
ALL IN TABLESPACE follows AlterTableMoveAll, with OWNED BY, the owner check, the pg_global error and the "no matching relations" notice.
REINDEX (TABLESPACE) moves the rebuilt indexes. System and TOAST relations give "cannot move system relation". pg_global gives the errors of PostgreSQL 19. REINDEX SCHEMA, SYSTEM and DATABASE skip the catalogs with one warning.
The tablespace regression diff goes from 268 to 63 lines. The full schedule stays at 141 ok.

Next in the tablespace category: ALTER TABLESPACE ... SET/RESET/RENAME/OWNER and GRANT ON TABLESPACE.

pg_current_wal_lsn, pg_current_wal_insert_lsn and pg_current_wal_flush_lsn now work, which fixes the diff of the sequence regress test. (#1195)

Merged #1196: ALTER TABLESPACE ... SET/RESET, RENAME TO, OWNER TO, and GRANT/REVOKE on a tablespace.

The changed rows of pg_tablespace now live in the transaction, as the rows of pg_database do. The commit moves them to the store, and a rollback discards them.
SET/RESET follows AlterTableSpaceOptions. RENAME TO follows RenameTableSpace, with the checks of the owner, the pg_ prefix, a newline and a duplicate name. OWNER TO follows AlterObjectOwner_internal and changes the owner in spcacl.
GRANT and REVOKE on a tablespace use the shared ACL code, with CREATE as the only privilege.
The tablespace regression test now passes. The full schedule goes from 141 ok to 142 ok, with no test worse.

The tablespace category is done, except changes of pg_default and pg_global, which give 0A000 for now.

Merged #1197: REFRESH MATERIALIZED VIEW now gives the same errors as PostgreSQL 19 when the new data does not agree with a unique index. A refresh gives the error of the index build for a duplicate key. A concurrent refresh gives the error for duplicate rows without nulls, and checks again for a usable unique index after its query runs. The matview regression test now passes.

Navigation Menu

Repository navigation

M3: SQL and the point path #4

Why

The work

Exit criterion

Open questions

Harness work

Depends on

Spec

647 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

M3: SQL and the point path #4

Description

Why

The work

Exit criterion

Open questions

Harness work

Depends on

Spec

Activity

tamnd commented on Oct 17, 2026

tamnd commented on Oct 18, 2026

tamnd commented on Oct 18, 2026

tamnd commented on Oct 18, 2026

tamnd commented on Oct 19, 2026

647 remaining items

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

tamnd commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions