M2: Connect and introspect

[tamnd]

The third milestone of rupg, specified in spec/23-milestones.md section 23.5. M2 lets a real client connect, authenticate, read the catalog and see the schema. It delivers protocols 3.0 and 3.2, TLS, SCRAM-SHA-256, the virtual catalog, and the text and binary forms of the types. Gate G1 closes here.

Why

Connect before query. A client can test nothing until it connects. Many clients read the catalog before they send any user query, so the catalog comes with the connection. This is "follow the client" and "catalog before dialect" of the compatibility work of rudb.

G1 needs clients to read the catalog with SQL. So the SQL front, a first analyzer and a simple executor ship at M2, and M3 completes them (spec/24 §24.2, item C9). The parser is the full vendored grammar from the start, because the grammar is one file.

The work

Taken from spec/23 §23.5. The reference in brackets names the spec section that owns the detail. Change the list in the spec and copy it again. Do not edit it here.

• rupg-wire and rupg-server: startup, NegotiateProtocolVersion, SSL negotiation, SCRAM, the simple and extended query flows, cancel, and ParameterStatus (spec/06 §6.5, spec/06 §6.6, spec/06 §6.7, spec/06 §6.9.1, spec/06 §6.12)
• rupg-session with the 438 settings, SET, SHOW and RESET (spec/06 §6.13)
• rupg-catalog and rupg-pgcatalog: the built-in objects with their OIDs, the 64 catalogs, the 86 system views and the 65 information_schema views, as virtual tables (spec/07 §7.12, spec/07 §7.13, spec/07 §7.14)
• The types with text and binary forms equal to the oracle (spec/07 §7.9)
• The parser from the full vendored grammar (spec/07 §7.2)
• The analyzer, the plan and the executor for SELECT with joins, subqueries, CASE, casts to regclass and regtype, and the functions that the 48 recorded traces call on the catalog, for example format_type, pg_get_expr, pg_get_indexdef and pg_get_constraintdef (spec/23 §23.5, spec/07 §7.16, spec/07 §7.17)
• DDL that creates tables, columns, constraints and index definitions (spec/23 §23.5, spec/07 §7.11)
• The L1 traces of the 48 clients, with the share of identical messages published for each (spec/21 §21.15)
• The L2 comparison of every catalog row for the gate schema (spec/05 §5.2)
• The 9 libpq_pipeline tests (spec/21 §21.4.1)
• The pg_dump round trip (spec/21 §21.4.1)
• 24 CPU hours each on wire_frontend, wire_startup, type_input and type_recv with no open find (spec/21 §21.7)
• The idle session test against the 64 KiB limit (spec/21 §21.13, spec/06 §6.4)
• The connection rate with trust over a Unix socket and with SCRAM over TLS, against the oracle. The target is at least 10x the oracle (spec/06 §6.18)
• Crates that first ship: rupg-wire, rupg-catalog, rupg-pgcatalog, rupg-sql, rupg-analyze, rupg-plan, rupg-func, rupg-exec, rupg-session and rupg-server (spec/23 §23.2)

Exit criterion

1. G1 (spec/02 §2.10): 100 percent of 7 clients connect and show the schema. The seven are psql, pgAdmin 4, DBeaver, psycopg 3, node-postgres, pgx and asyncpg. "Show the schema" means that the client's own object browser or describe command lists the tables, columns, types, indexes and constraints of the gate schema, equal to the list that the same client shows against the oracle (spec/21 §21.4.2).
2. The report publishes the share of identical L1 messages for each of the 48 clients, and the L2 comparison of every catalog row for the gate schema.
3. An idle session costs at most 64 KiB (spec/06 §6.4).
4. wire_frontend, wire_startup, type_input and type_recv each run 24 CPU hours with no open find.
5. The close report answers each line of the checklist in spec/23 §23.18 that applies from M2.

Open questions

These questions in spec/24 close at M2.

• Q46 (spec/24 §24.6): whether the TLS state fits in the 24 KiB of the idle budget, or needs kTLS on Linux. Q46: Does the TLS state fit in the idle budget? #59
• Q71 (spec/24 §24.9): dates. A forecast is possible after M2, from the time per milestone and the lines against the budgets. Q71: When can the project give dates? #84

Harness work

• Compatibility: M2: L1 and L2 for the seven gate clients rupg-compat#2

Depends on

#2.

Spec

• spec/23-milestones.md §23.5 M2: connect and introspect
• spec/23-milestones.md §23.16 Why the order is what it is
• spec/02-the-goal.md §2.3 Axis 1: compatibility
• spec/02-the-goal.md §2.10 When a gate fails
• spec/05-compatibility.md §5.2 What is compared
• spec/06-wire-and-sessions.md
• spec/07-sql-types-and-catalog.md §7.2 The vendored grammar
• spec/07-sql-types-and-catalog.md §7.9 Types and their input and output
• spec/07-sql-types-and-catalog.md §7.13 The virtual pg_catalog
• spec/07-sql-types-and-catalog.md §7.14 information_schema and the system views
• spec/21-testing.md §21.4 The suites
• spec/21-testing.md §21.13 Resource tests
• spec/24-open-questions.md §24.6 Transactions and the cluster
• spec/24-open-questions.md §24.9 The project

[tamnd]

M2 starts with #132. It adds rupg-wire, the codec of protocols 3.0 and 3.2, copied from rudb-pgwire at f5f7065a as spec/04 section 4.9 says. A new test checks the table of command tags against the vendored cmdtaglist.h.

The fuzz targets wire_startup and wire_frontend are in fuzz/, and the nightly job runs them. The 24 CPU hour runs start when rupg-server adds the SCRAM exchange to wire_startup, because the failure that the spec names for this target is a session that starts without the correct password.

No line of the checklist closes yet. The first line needs rupg-server. The next PR adds SASLprep to rupg-wire, which rudb does not have.

[tamnd]

#133 adds SASLprep to rupg-wire, as pg_saslprep does it, and verify_password now uses it. The NFKC step uses the new rupg_types::unicode, which has the four normalization forms and passes every line of NormalizationTest.txt of Unicode 17.0.0. cargo xtask unicode makes the tables from the vendored files, and CI checks that they are up to date.

The next PR copies the parser from rudb-pgparse into rupg-sql.

[tamnd]

#134 makes the full SQLSTATE list in rupg-common from the vendored errcodes.txt, with cargo xtask errcodes and a CI check. The parser, the types and the server need these codes. The parser PR and the types PR are in progress.

[tamnd]

#135 adds the text and binary forms of the types to rupg-types, lifted from rudb-pgtypes. It covers the numbers, the strings, bytea, uuid, json, jsonb, numeric, the date and time types, arrays and the OID alias types, and a test checks 8753 results of the server at the pin. cargo xtask pgtype makes the type OIDs from the vendored pg_type.dat, and CI checks them. The types line stays open until the ranges, the geometric, network, bit string, text search and xml types are in.

[tamnd]

#137 adds the configuration parameters to rupg-session, lifted from the guc module of rudb-common. cargo xtask guc makes the table from the vendored files with the boot values of the Linux oracle, and a test finds the same 419 rows as pg_settings of the oracle. TimeZone now takes the 598 names of the vendored tzdata, which are the names of pg_timezone_names. The session line stays open until SET, SHOW and RESET run as SQL. The spec gives 438 parameters and 432 rows, but the oracle build has 425 parameters and shows 419, so the PR follows the oracle and records the difference.