From 27bed253f5804541c60b9f81c17024f04ccd2339 Mon Sep 17 00:00:00 2001 From: sebi Date: Sat, 10 Oct 2026 14:13:18 -0500 Subject: [PATCH 1/2] fix(entity): an arrayOf(uuid()) column reads on a real Postgres; every column kind round-trips on both drivers Bun.SQL returns uuid[] (and arrays of types it has no parser for) as the Postgres array literal; decodeRow handed that string to arrayOf's parser, which refused it (X_INVARIANT_VIOLATED column.array). PGlite parses them, so only a real Postgres showed it: a row with a uuid[] could be written and never read back. - decodeRow reads a declared array column through arrayFromDriver: an Array passes, a typed array becomes one, a string goes through the array-literal grammar and each element through its declared kind. A non-array column is never inspected. - One array-literal reader (parsePgArray), shared with @ultimat3/realtime. - column-matrix: every column kind and an array of every accepted element kind, insert/read/update on Bun.SQL against Postgres (gate live) and on PGlite (unit). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 +- docs/history/entity.md | 56 +++ packages/entity/CLAUDE.md | 11 +- packages/entity/README.md | 2 +- packages/entity/src/column-matrix-fixture.ts | 353 ++++++++++++++++++ .../entity/src/column-matrix-parity.test.ts | 29 ++ .../entity/src/column-matrix.live.test.ts | 39 ++ packages/entity/src/index.ts | 6 +- packages/entity/src/pg-array-decode.test.ts | 180 +++++++++ packages/entity/src/pg-array-decode.ts | 82 ++++ packages/entity/src/pg-array-literal.test.ts | 90 +++++ packages/entity/src/pg-array-literal.ts | 106 ++++++ packages/entity/src/pg-row.ts | 26 +- packages/realtime/src/pg-array.ts | 110 +----- packages/realtime/src/pg-values.test.ts | 3 +- packages/realtime/src/pg-values.ts | 3 +- wiki/Entities-And-Migrations.md | 2 +- 17 files changed, 988 insertions(+), 118 deletions(-) create mode 100644 packages/entity/src/column-matrix-fixture.ts create mode 100644 packages/entity/src/column-matrix-parity.test.ts create mode 100644 packages/entity/src/column-matrix.live.test.ts create mode 100644 packages/entity/src/pg-array-decode.test.ts create mode 100644 packages/entity/src/pg-array-decode.ts create mode 100644 packages/entity/src/pg-array-literal.test.ts create mode 100644 packages/entity/src/pg-array-literal.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 586a7e2f4..5351ce2ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,13 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major ## [Unreleased] -Nothing yet. +### Fixed + +Tier 2 — entity. Tier 3 — realtime. + +- **An `arrayOf(uuid())` column reads on a real Postgres.** `Bun.SQL` hands back the array types it has no parser for as their TEXT literal (`'{01a1…,01a2…}'` for `uuid[]`, the empty `'{}'` included), and `decodeRow` passed that string to `arrayOf`'s parser, which refuses anything that is not an `Array`: every read of a row holding one, the `returning` of the insert that wrote it included, was `X_INVARIANT_VIOLATED` ("column.array: expected an array, got a string of 38 characters", a 422 on the HTTP surface). PGlite parses `uuid[]`, so `x dev`, the `unit` step and an app's own tests never saw it; only a server did. An `arrayOf()` cell is now read by its declaration: an `Array` as before, a typed array (`Bun.SQL` answers `int4[]` with an `Int32Array` on a pool that prepares statements) as a plain array, and a string through the Postgres array literal grammar, with quoting, backslash escapes, NULL members and `{}`, each element then parsed by the element column. A cell that is none of these is still `X_INVARIANT_VIOLATED`, now naming the entity and the property; a string in a non-array column is never inspected. Writes, and `contains` / `overlaps` with a bound array, were measured for every element type and were already correct. Nothing to change in an app; rows written before this release read back as written. +- **Every column kind is round-tripped on both drivers.** One entity declares every column kind and an array of every element kind `arrayOf()` accepts (nullable elements included), and seven rows go through insert, a keyed read, an unfiltered read and an update on PGlite (`unit` step) and on `Bun.SQL` against Postgres (`live` step), each compared with `memoryDriver()`. A kind added to `COLUMN_KINDS` without a column in that entity fails the suite. The only live array test before this used `arrayOf(text())`, the one element type both drivers parse. +- **The array literal grammar has one reader.** `parsePgArray` moved from `@ultimat3/realtime` into `@ultimat3/entity` (exported from the barrel), and the WAL decoder imports it, so a live row and a repository row cannot disagree about a quoted `"NULL"`. ## 27.6.0 - 2026-10-10 diff --git a/docs/history/entity.md b/docs/history/entity.md index 3484ed3a6..522bd73a4 100644 --- a/docs/history/entity.md +++ b/docs/history/entity.md @@ -1324,6 +1324,7 @@ string (a `lookup` column's is equal for equal values). | `jit-preload.ts` | a page's foreign key values → one `in` statement for the whole `for … of` loop | | `preload.ts` | the relation `preload()` names → one related-rows statement → attached to the page | | `pg-sql.ts` / `pg-row.ts` | plan → parameterised SQL; physical row ⇄ entity row (money is three columns) | +| `pg-array-literal.ts` / `pg-array-decode.ts` | the Postgres array literal grammar; an `arrayOf()` cell in any shape a driver hands back → a JS array | | `row-observer.ts` | `setRowObserver` — committed row changes, above the driver, for a change feed that has no log to read. A change made inside a keyed request carries `write` (`currentWriteOrigin()`) | | `write-tag.ts` | a keyed request's write names itself in the WAL: `pg_logical_emit_message(true, WRITE_ORIGIN_WAL_PREFIX, digest)` opens its transaction, once per transaction; a write outside one gets a transaction of its own; a role that may not execute it is probed once and its writes go out untagged; a pinned repository is never wrapped | | `registry.ts` | duplicate detection, `describeEntities()` for the manifest, `references()` per entry | @@ -1363,3 +1364,58 @@ the same function, which reads either side as plain digits. So `where({ total: 1 `bigint()` column matches the row holding `'10'` in the memory driver, as it always did in Postgres. `compare-parity.test.ts` runs the rows against both drivers; emitted DDL did not move (`ddl-pin.test.ts`). + +## 2026-10-10 — an array cell is read by its declaration, not by the driver's type table + +`arrayOf(uuid())` could be written and never read on a real server: `decodeRow` handed the driver's +cell to `arrayOf`'s `$parse`, and `Bun.SQL` answers `uuid[]` with the array's TEXT literal +(`'{01a1…}'`), which is not an `Array`. Every read of a row holding one was `X_INVARIANT_VIOLATED` +("expected an array, got a string of 38 characters"), including the `returning *` of the insert +that wrote it. PGlite parses `uuid[]`, and PGlite is what `x dev`, the `unit` step and an app's own +tests run on, so nothing in this tree or in an app's gate could see it; a built image against +Postgres was the first thing that did. The one live array test (`columns-data.live.test.ts`) used +`arrayOf(text())`, the one element type both drivers parse. + +Measured on Bun 1.4.2 against Postgres 17 (the framework's pool, `prepare: false`) and PGlite, for +`select ` of each type: + +| Postgres type | `Bun.SQL`, `prepare: false` (what `postgresClient` opens) | `Bun.SQL`, prepared (binary results) | PGlite | +|---|---|---|---| +| `uuid[]` (and `'{}'::uuid[]`) | **`string`, the literal** | **`string`, the literal** | `Array` | +| `text[]`, `varchar[]` | `Array` | `Array` | `Array` | +| `int4[]` | `Array` of `number` | **`Int32Array`**; with a NULL member **`Failed to read data`** and the connection is dropped | `Array` | +| `float4[]` (no column kind) | `Array` | **`Float32Array`**; NULL member fails as above | `Array` | +| `int8[]` | `Array` of `string` | same | `Array` of `number` / `bigint` | +| `numeric[]` | `Array` of `string` | same | same | +| `bool[]` | `Array` of `boolean` | same | same | +| `timestamptz[]`, `date[]` | `Array` of `Date` | same | same | +| `jsonb[]`, `bytea[]` (refused by `arrayOf`) | `Array` | same | `Array` | +| an `enum` type's array (no column kind) | `string`, the literal | same | `string`, the literal | +| every scalar kind | as `columns-data.ts` records | same | `int8` is a `bigint` | + +So the unparsed cases are exactly the array types the driver has no entry for, and a user-defined +type's oid is per-database: a table keyed by oid cannot be complete. `Bun.SQL` also has no +type-parser registration to hang one on. The declared column is the only thing that knows the cell +is an array, so `decodeRow` asks `arrayFromDriver(meta, value)` (`pg-array-decode.ts`) for an array +column and for no other: an `Array` passes through, a typed array becomes a plain one, a string is +read by `parsePgArray` with each element's text turned into the value a parsing driver hands over +(`number`, `boolean`, `Date`, or the text itself), and anything else is `X_INVARIANT_VIOLATED` +naming the entity and the property. A string in a `text()` column is never inspected. + +`parsePgArray` moved here from `@ultimat3/realtime`'s `pg-array.ts`, which read the same literal +off the WAL; realtime imports it, so the grammar has one reader. `pg-array-literal.test.ts` holds +it to the writer (`arrayLiteral` in `pg-row.ts`). + +The write direction was measured in the same pass and was already right: an array is bound as its +quoted literal (`bindable`), an untyped parameter takes the column's type, and `@>` / `&&` with a +bound operand match on every element type. + +`column-matrix-fixture.ts` is the guard: one entity declaring every `COLUMN_KINDS` member and an +array of every element kind `arrayOf()` accepts (a kind added to the list without a column there is +red), seven adversarial rows, each through insert `returning`, a keyed read, an unfiltered read and +update `returning`, compared with `memoryDriver()`. `column-matrix-parity.test.ts` runs it on +PGlite in the `unit` step and `column-matrix.live.test.ts` on `Bun.SQL` in the `live` step. + +Not fixed, because the framework never opens such a pool: on a PREPARING `Bun.SQL` connection an +`int4[]` with a NULL member fails inside the driver. `prepare: false` is `bun-sql.ts`'s; a change +to it has to answer the third column of the table above first. diff --git a/packages/entity/CLAUDE.md b/packages/entity/CLAUDE.md index f76d567bf..9f0f24d57 100644 --- a/packages/entity/CLAUDE.md +++ b/packages/entity/CLAUDE.md @@ -48,7 +48,7 @@ Columns + invariants; the row type is derived from the columns. Tier 2. list or nothing, and a NULL in it emits `(col in (…) or col is null)`; a column the row never NAMED is NULL for `eq`/`neq`/`in`. - **The Postgres driver is proved against a real Postgres** (`pg-driver.live.test.ts`, skipped without - `TEST_DATABASE_URL`). A new operator, column kind or write path is not done until it round-trips. + `TEST_DATABASE_URL`). A new operator, kind or write path is not done until it round-trips. - **A repository call rejects, never throws synchronously** (`tableFor`'s writes are `async`). - **`defaultDriver()`** is the process default; `Driver.reset?()` is optional (memory only), resets repositories in place. Test seam only. @@ -102,7 +102,7 @@ Columns + invariants; the row type is derived from the columns. Tier 2. `seekFrom(entity, plan)`; **`assertSeekable` runs in `planFor`**, before a statement exists. A `timestamptz` sort key is refused when its `$US` alias would pass 63 bytes. A cursor is bound to `planScope(plan)` (entity, filters, sort); a bad one is `X_CURSOR_INVALID`, never "from the top". -- **A NULLABLE sort key orders** `asc nulls last` / `desc nulls first` (written down); the cursor tags +- **A NULLABLE sort key orders** `asc nulls last` / `desc nulls first`; the cursor tags `~` for NULL and `!` for present; the seek reaches NULLs; a nullable key has no row comparison. Only a nullable PRIMARY-KEY column is refused. `pg-null-order.live.test.ts`. - **A `timestamp` cursor carries MICROSECONDS**: `seekPrecision` projects `(col at time zone @@ -145,8 +145,7 @@ Columns + invariants; the row type is derived from the columns. Tier 2. NAME and the TABLE are different things** — cache tags and policies key on the name; index names are the table's. - **Invariants run twice and only ONE side is rendered here**: app-side `assertInvariants`, and the - CHECK/UNIQUE DDL is `@ultimat3/db`'s `invariant-ddl.ts` reading `$describe()` (`$migration()` and the - local renderers are deleted). An untranslatable predicate is `kind: 'assert'`, `sql: null`. + CHECK/UNIQUE DDL is `@ultimat3/db`'s `invariant-ddl.ts` reading `$describe()`. An untranslatable predicate is `kind: 'assert'`, `sql: null`. **`InvariantDescription.columns` is projected** (`describe-invariant.test.ts`). - **The two halves must AGREE, term by term** (`expr.ts`): `matches(/…/i)` is `~*`, other flags are refused (`matchOperator`); `minLength` counts code points; `trimmed()` strips U+0020 only (`btrim`); @@ -175,8 +174,8 @@ Columns + invariants; the row type is derived from the columns. Tier 2. `scale: null` means a two-column amount (`decodeRow` branches on `$meta.kind === 'money'`). - **Timestamps are `timestamptz`.** A naive timestamp stays inexpressible. - **A `jsonb` value is bound `::text::jsonb`** (load-bearing); an array is written as a quoted `{…}` - literal. **Wide column types are normalised in `$parse`** to one row type per driver (`bigint()` and - `decimal()` are strings). + literal, read by its DECLARATION (`pg-array-decode.ts`); `column-matrix-fixture.ts` is every kind + on both drivers. **Wide types are normalised in `$parse`** (`bigint()`, `decimal()`: strings). - **Every framework member on an entity is `$`-prefixed.** Row types are derived, never re-declared. **A branded id survives to the signature** (`IdOf`). `type-pins.ts` enforces all of it. - **`$parse` tells absence from `null`** (`raw === undefined ? defaultValue(...) : raw`). diff --git a/packages/entity/README.md b/packages/entity/README.md index 0bab8190c..fa15c62d5 100644 --- a/packages/entity/README.md +++ b/packages/entity/README.md @@ -240,7 +240,7 @@ for a table it was going to create; these are the shapes a table already has, `A | `date()` | `date` | `@ultimat3/time`'s `PlainDate` — a calendar date, no time, no zone. `effective_on` is the date a rate applies, and as a `timestamptz` it is a different date on either side of midnight for half the planet | | `bigint()` | `bigint` | a decimal `string`. A JS `bigint` is what `JSON.stringify` throws on and a `number` loses digits past 2^53 — which is exactly where a legacy `int8` key lives. Both driver spellings (a string from Bun's `sql`, a `bigint` from PGlite) arrive as one. Bounded at `int8` (−2^63 … 2^63−1): one past either end is refused where Postgres answers `22003` | | `bytes()` | `bytea` | a plain `Uint8Array`, normalised: Bun's `sql` returns a `Buffer` and PGlite a `Uint8Array`, and the two do not serialise alike | -| `arrayOf(column)` | `[]` | `readonly T[]`, each member parsed by the element column it was given. Money and nested arrays are refused — an element is one scalar column | +| `arrayOf(column)` | `[]` | `readonly T[]`, each member parsed by the element column it was given. Money and nested arrays are refused — an element is one scalar column. Read by the declaration, not by the driver: `Bun.SQL` hands `uuid[]` back as its text literal (`'{…}'`) and PGlite as an `Array`, and both decode to the same row | ## Adopting an existing table diff --git a/packages/entity/src/column-matrix-fixture.ts b/packages/entity/src/column-matrix-fixture.ts new file mode 100644 index 000000000..82dc538a0 --- /dev/null +++ b/packages/entity/src/column-matrix-fixture.ts @@ -0,0 +1,353 @@ +// Every column kind an entity can declare, and every element kind `arrayOf()` accepts, in ONE +// entity with a corpus of adversarial rows — the matrix the two driver suites walk. Shared so the +// PGlite run (`column-matrix-parity.test.ts`, the `unit` step) and the `Bun.SQL` run +// (`column-matrix.live.test.ts`, the `live` step) answer the same question about the same rows. + +import { expect, test } from 'bun:test'; +import { type DbClient, generateMigration, raw, statementsOf } from '@ultimat3/db'; +import { t } from '@ultimat3/schema'; +import type { PlainDate } from '@ultimat3/time'; +import { isRefusedElement } from './array-element'; +import { boolean, integer, money, text, timestamp, url, uuid } from './columns'; +import { arrayOf, bigint, bytes, date, decimal, json } from './columns-data'; +import { type Driver, database, memoryDriver } from './database'; +import { entity } from './entity'; +import { enumerated } from './enum-column'; +import { COLUMN_KINDS } from './types'; + +const MOODS = ['sad', 'ok', 'happy'] as const; + +/** + * `s` is `arrayOf()` and `Gaps` is the same array with nullable ELEMENTS — a SQL + * NULL inside a row value, the one shape a driver's binary array reader has to carry a null bitmap + * for. Every array column is nullable so one row can hold `null` for the whole column. + */ +export const matrix = entity('column_matrix_rows', { + columns: { + id: uuid().primaryKey(), + label: text({ max: 40 }), + mood: enumerated(MOODS), + link: url().nullable(), + count: integer(), + flag: boolean(), + at: timestamp(), + rate: decimal({ precision: 18, scale: 8 }), + day: date(), + big: bigint(), + data: json(t.object({ plan: t.string, seats: t.number })), + blob: bytes().nullable(), + price: money().nullable(), + uuids: arrayOf(uuid()).nullable(), + uuidGaps: arrayOf(uuid().nullable()).nullable(), + texts: arrayOf(text()).nullable(), + textGaps: arrayOf(text().nullable()).nullable(), + moods: arrayOf(enumerated(MOODS)).nullable(), + links: arrayOf(url()).nullable(), + counts: arrayOf(integer()).nullable(), + countGaps: arrayOf(integer().nullable()).nullable(), + flags: arrayOf(boolean()).nullable(), + flagGaps: arrayOf(boolean().nullable()).nullable(), + ats: arrayOf(timestamp()).nullable(), + atGaps: arrayOf(timestamp().nullable()).nullable(), + rates: arrayOf(decimal({ precision: 18, scale: 8 })).nullable(), + rateGaps: arrayOf(decimal().nullable()).nullable(), + days: arrayOf(date()).nullable(), + dayGaps: arrayOf(date().nullable()).nullable(), + bigs: arrayOf(bigint()).nullable(), + bigGaps: arrayOf(bigint().nullable()).nullable(), + }, +}); + +export type MatrixRow = typeof matrix.$row; +type MatrixWrite = Parameters['rows']['insert']>[0]; + +export const matrixDb = (driver: Driver) => database({ rows: matrix }, { driver }); + +const A = '01a12719-28e5-735e-b5bb-000000000001'; +const B = '01a12719-28e5-735e-b5bb-000000000002'; +const day = (value: string): PlainDate => value as PlainDate; + +const SCALARS = { + mood: 'ok', + count: 7, + flag: true, + at: new Date('2026-01-02T03:04:05.123Z'), + rate: '1.23456789', + day: day('2026-03-14'), + big: '9007199254740993', + data: { plan: 'team', seats: 12 }, +} as const; + +/** The element text a Postgres array literal has to quote or escape, each one a single element. */ +export const NASTY_TEXT = [ + 'plain', + 'with,comma', + 'quote"inside', + 'back\\slash', + '{braces}', + '', + ' padded ', + 'NULL', + 'null', + 'line\nbreak', + 'tab\there', + 'ünï©ode ✓', + "single'quote", +] as const; + +/** Named, so a failure says which row of the matrix it was. */ +export const MATRIX_ROWS: Readonly> = Object.freeze({ + 'every array holds two members': { + ...SCALARS, + label: 'full', + link: 'https://example.com/a?b=c,d', + blob: new Uint8Array([0, 255, 16]), + price: { minor: 1999, currency: 'EUR' }, + uuids: [A, B], + uuidGaps: [A, B], + texts: ['alpha', 'beta'], + textGaps: ['alpha', 'beta'], + moods: ['sad', 'happy'], + links: ['https://example.com/a,b', 'http://example.com/{c}'], + counts: [1, -2147483648], + countGaps: [2147483647, 0], + flags: [true, false], + flagGaps: [false, true], + ats: [new Date('2026-01-02T03:04:05.123Z'), new Date('1999-12-31T23:59:59.999Z')], + atGaps: [new Date('2026-06-01T00:00:00.000Z'), new Date(0)], + rates: ['1.23456789', '-0.5'], + rateGaps: ['12345678901234567890.123456789', '0'], + days: [day('2026-03-14'), day('1970-01-01')], + dayGaps: [day('2024-02-29'), day('2026-12-31')], + bigs: ['9007199254740993', '-9223372036854775808'], + bigGaps: ['9223372036854775807', '0'], + }, + 'every array holds one member': { + ...SCALARS, + label: 'single', + uuids: [A], + uuidGaps: [A], + texts: ['only'], + textGaps: ['only'], + moods: ['ok'], + links: ['https://example.com/'], + counts: [42], + countGaps: [42], + flags: [true], + flagGaps: [false], + ats: [new Date('2026-01-02T03:04:05.123Z')], + atGaps: [new Date('2026-01-02T03:04:05.123Z')], + rates: ['1.5'], + rateGaps: ['1.5'], + days: [day('2026-03-14')], + dayGaps: [day('2026-03-14')], + bigs: ['1'], + bigGaps: ['1'], + }, + 'every array is empty': { + ...SCALARS, + label: 'empty', + uuids: [], + uuidGaps: [], + texts: [], + textGaps: [], + moods: [], + links: [], + counts: [], + countGaps: [], + flags: [], + flagGaps: [], + ats: [], + atGaps: [], + rates: [], + rateGaps: [], + days: [], + dayGaps: [], + bigs: [], + bigGaps: [], + }, + 'every array column is null': { + ...SCALARS, + label: 'null columns', + link: null, + blob: null, + price: null, + uuids: null, + uuidGaps: null, + texts: null, + textGaps: null, + moods: null, + links: null, + counts: null, + countGaps: null, + flags: null, + flagGaps: null, + ats: null, + atGaps: null, + rates: null, + rateGaps: null, + days: null, + dayGaps: null, + bigs: null, + bigGaps: null, + }, + 'a null sits between two members': { + ...SCALARS, + label: 'gaps', + uuidGaps: [A, null, B], + textGaps: ['a', null, 'NULL', ''], + countGaps: [1, null, 3], + flagGaps: [true, null, false], + atGaps: [new Date('2026-01-02T03:04:05.123Z'), null], + rateGaps: ['1.5', null, '2'], + dayGaps: [day('2026-03-14'), null], + bigGaps: ['9007199254740993', null], + }, + 'every member is null': { + ...SCALARS, + label: 'all gaps', + uuidGaps: [null], + textGaps: [null, null], + countGaps: [null], + flagGaps: [null], + atGaps: [null], + rateGaps: [null], + dayGaps: [null], + bigGaps: [null], + }, + 'text the array literal has to quote': { + ...SCALARS, + label: 'nasty', + texts: [...NASTY_TEXT], + textGaps: [...NASTY_TEXT, null], + }, +}); + +const DROP = `drop table if exists "${matrix.$table}" cascade`; + +/** The matrix table, created from the entity's own migration — never hand-written DDL. */ +export const createMatrixTable = async (client: DbClient): Promise => { + await client.execute(raw(DROP)); + const migration = generateMigration({ + entities: [matrix.$describe()], + name: 'column matrix', + now: new Date('2026-10-10T00:00:00.000Z'), + }); + for (const statement of statementsOf(migration.up)) await client.execute(raw(statement)); +}; + +export const dropMatrixTable = async (client: DbClient): Promise => { + await client.execute(raw(DROP)); +}; + +/** + * One row through every read path that decodes: the `returning` of the insert, a keyed read (a + * statement WITH a parameter), an unfiltered read, and the `returning` of an update. A driver may + * decode a column differently per protocol — `Bun.SQL` reads `int4[]` as an `Array` on the simple + * protocol and as an `Int32Array` on the extended one (measured, 1.4.2) — so one path is not all. + */ +export const roundTrip = async (driver: Driver, write: MatrixWrite) => { + const db = matrixDb(driver); + const inserted = await db.rows.insert(write); + const keyed = await db.rows.where({ id: inserted.id }).one(); + const listed = (await db.rows.where({}).all()).find((row) => row.id === inserted.id) ?? null; + const updated = await db.rows.update(inserted.id, { label: `${write.label} again` }); + const detach = (row: R) => + row === null ? null : { ...row, label: write.label }; + return { inserted, keyed, listed, updated: detach(updated) }; +}; + +/** The array columns, read off the declaration so a column added above is a column tested below. */ +const ARRAY_COLUMNS = Object.entries(matrix.$columns) + .filter(([, column]) => column.$meta.kind === 'array') + .map(([property]) => property); + +/** + * The WRITE direction: a JS array bound as the operand of `@>` and `&&` against every array type. + * A literal that reads back correctly can still be one the server refuses to COMPARE — an untyped + * parameter takes the column's type, and `{"2026-01-02T03:04:05.123Z"}` has to parse as a + * `timestamptz[]` there. Asked of both drivers, because memory answering alone proves nothing. + */ +const matches = async (driver: Driver, id: string, column: string, member: unknown) => { + const repo = driver.repo(matrix); + const found = async (op: 'contains' | 'overlaps') => + ( + await repo.findMany({ + where: [ + { column: 'id', op: 'eq', value: id }, + { column, op, value: [member] }, + ], + orderBy: [{ column: 'id', direction: 'asc' }], + limit: 5, + }) + ).rows.length; + return { contains: await found('contains'), overlaps: await found('overlaps') }; +}; + +/** + * The cases both driver suites run. `open` is called per case, after the suite's `beforeAll` has + * installed its client, so the driver under test is whichever one that suite stands on. + */ +export const columnMatrixCases = (open: () => Driver): void => { + // The ratchet: a column kind added to `COLUMN_KINDS` with no column above is red here, so the + // next kind cannot ship having met only the driver its author happened to run. `char` has no + // constructor of its own — it is the currency half of `money()`. + test('the matrix declares every column kind, and an array of every element kind', () => { + const columns = Object.values(matrix.$columns).map((column) => column.$meta); + const declared = new Set(columns.map((meta) => meta.kind)); + const elements = new Set(columns.flatMap((meta) => meta.element?.$meta.kind ?? [])); + const scalar = COLUMN_KINDS.filter((kind) => kind !== 'char'); + expect(scalar.filter((kind) => !declared.has(kind))).toEqual([]); + expect(scalar.filter((kind) => !isRefusedElement(kind) && !elements.has(kind))).toEqual([]); + }); + + test('the corpus writes every column the entity declares', () => { + const written = new Set(Object.values(MATRIX_ROWS).flatMap((row) => Object.keys(row))); + const unwritten = Object.keys(matrix.$columns).filter( + (name) => name !== 'id' && !written.has(name), + ); + expect(unwritten).toEqual([]); + }); + + for (const [name, write] of Object.entries(MATRIX_ROWS)) { + test(`${name}: the driver reads back what memory reads back`, async () => { + const id = crypto.randomUUID(); + const memory = await roundTrip(memoryDriver(), { ...write, id }); + const real = await roundTrip(open(), { ...write, id }); + expect(real.inserted).toEqual(memory.inserted); + expect(real.keyed).toEqual(memory.keyed); + expect(real.listed).toEqual(memory.listed); + expect(real.updated).toEqual(memory.updated); + // Memory is not merely agreeing with itself: every array read back IS a JS array, of the + // members that were written — the one claim `toEqual` between two wrong answers cannot make. + const source = write as Readonly>; + const keyed = (real.keyed ?? {}) as Readonly>; + for (const column of ARRAY_COLUMNS) { + const wrote = new Map(Object.entries(source)).get(column) ?? null; + const read = new Map(Object.entries(keyed)).get(column); + if (wrote === null) expect(read).toBeNull(); + else expect(Array.isArray(read) && read.length === (wrote as unknown[]).length).toBe(true); + } + }); + } + + test('a bound array is an operand every array type compares against', async () => { + const write = MATRIX_ROWS['every array holds two members']; + if (write === undefined) return expect.unreachable('the two-member row left the corpus'); + const id = crypto.randomUUID(); + const memory = memoryDriver(); + const real = open(); + await matrixDb(memory).rows.insert({ ...write, id }); + await matrixDb(real).rows.insert({ ...write, id }); + const source = new Map(Object.entries(write as Readonly>)); + for (const column of ARRAY_COLUMNS) { + const [member] = source.get(column) as readonly unknown[]; + const answer = await matches(real, id, column, member); + expect({ column, ...answer }).toEqual({ column, contains: 1, overlaps: 1 }); + expect({ column, ...(await matches(memory, id, column, member)) }).toEqual({ + column, + ...answer, + }); + } + }); +}; diff --git a/packages/entity/src/column-matrix-parity.test.ts b/packages/entity/src/column-matrix-parity.test.ts new file mode 100644 index 000000000..c9b8de8e5 --- /dev/null +++ b/packages/entity/src/column-matrix-parity.test.ts @@ -0,0 +1,29 @@ +// Every column kind and every `arrayOf()` element kind, round-tripped through `postgresDriver()` +// over PGlite and compared with `memoryDriver()` — the embedded half of the matrix +// `column-matrix.live.test.ts` runs on `Bun.SQL`. The two drivers decode a cell by different +// tables (PGlite parses `uuid[]`, `Bun.SQL` hands back its literal), so green here says nothing +// about a server and green there says nothing about `x dev`: both are the bar. + +import { afterAll, beforeAll, describe } from 'bun:test'; +import { pgliteClient, setDbClient } from '@ultimat3/db'; +import { columnMatrixCases, createMatrixTable } from './column-matrix-fixture'; +import { postgresDriver } from './pg-driver'; +import { clearRegistry } from './registry'; + +const PGLITE_BOOT_MS = 30_000; +const client = pgliteClient(); + +beforeAll(async () => { + setDbClient(client); + await createMatrixTable(client); +}, PGLITE_BOOT_MS); + +afterAll(async () => { + setDbClient(undefined); + await client.close(); + clearRegistry(); +}); + +describe('pglite · every column kind round-trips', () => { + columnMatrixCases(() => postgresDriver()); +}); diff --git a/packages/entity/src/column-matrix.live.test.ts b/packages/entity/src/column-matrix.live.test.ts new file mode 100644 index 000000000..d38c8a5d0 --- /dev/null +++ b/packages/entity/src/column-matrix.live.test.ts @@ -0,0 +1,39 @@ +// Every column kind and every `arrayOf()` element kind, round-tripped through `Bun.SQL` against a +// real Postgres and compared with `memoryDriver()` — the test that would have caught `uuid[]` +// arriving as its array LITERAL (`'{01a1…}'`) and being refused by `arrayOf(uuid())` in production +// while PGlite, which every other suite and `x dev` run on, decoded it. A driver decides per type +// oid and per protocol what a cell is, so the proof is one row of every kind through every read +// path, on the driver production uses. Skips unless `TEST_DATABASE_URL` is set. + +import { afterAll, beforeAll, describe } from 'bun:test'; +import { type PostgresClient, postgresClient, setDbClient } from '@ultimat3/db'; +import { columnMatrixCases, createMatrixTable, dropMatrixTable } from './column-matrix-fixture'; +import { postgresDriver } from './pg-driver'; +import { clearRegistry } from './registry'; + +const adminUrl = Bun.env['TEST_DATABASE_URL']; +const hasPostgres = typeof adminUrl === 'string' && adminUrl.length > 0; + +describe.skipIf(!hasPostgres)('live · postgres · every column kind round-trips on Bun.SQL', () => { + let client: PostgresClient; + + beforeAll(async () => { + client = postgresClient({ url: adminUrl ?? '' }); + setDbClient(client); + await createMatrixTable(client); + }); + + afterAll(async () => { + await dropMatrixTable(client); + await client.close(); + setDbClient(undefined); + }); + + columnMatrixCases(() => postgresDriver()); +}); + +// Outside the block above and unconditional: bun runs no hook inside a skipped `describe`, and the +// registry is process-wide. `live-registry-cleanup.test.ts` is the rule that keeps it here. +afterAll(() => { + clearRegistry(); +}); diff --git a/packages/entity/src/index.ts b/packages/entity/src/index.ts index 9c2141e76..f62150141 100644 --- a/packages/entity/src/index.ts +++ b/packages/entity/src/index.ts @@ -88,10 +88,12 @@ export { memoryRepo, memoryTransactor } from './memory-repo'; export type { StatementLoop } from './n-plus-one'; export { N_PLUS_ONE_THRESHOLD, nPlusOne, preloadsFor } from './n-plus-one'; export { persistedRecordTypes } from './persisted-types'; -export type { PostgresDriverOptions } from './pg-driver'; -export { postgresDriver, postgresRepo } from './pg-driver'; // For a change feed holding a table name and raw columns: `entityForTable` + `decodeRow` is the row // the app declared, money and all. Exported for `@ultimat3/realtime` (plan 101, 06 i). +export type { PgArray } from './pg-array-literal'; +export { parsePgArray } from './pg-array-literal'; +export type { PostgresDriverOptions } from './pg-driver'; +export { postgresDriver, postgresRepo } from './pg-driver'; export { decodeRow } from './pg-row'; export { postgresTransactor } from './pg-transactor'; // The two page bounds, beside `N_PLUS_ONE_THRESHOLD` and for the same reason: an app validating diff --git a/packages/entity/src/pg-array-decode.test.ts b/packages/entity/src/pg-array-decode.test.ts new file mode 100644 index 000000000..0c347e857 --- /dev/null +++ b/packages/entity/src/pg-array-decode.test.ts @@ -0,0 +1,180 @@ +// An `arrayOf()` cell in each shape a driver hands it back, and `decodeRow` over them. The shapes +// are measured, not imagined: `Bun.SQL` 1.4.2 answers `uuid[]` with its literal and — on a pool +// that prepares statements — `int4[]` with an `Int32Array`; PGlite answers both with an `Array`. + +import { afterAll, describe, expect, test } from 'bun:test'; +import { isUltimateError } from '@ultimat3/core'; +import { boolean, integer, text, timestamp, uuid } from './columns'; +import { arrayOf, bigint, date, decimal } from './columns-data'; +import { entity } from './entity'; +import { arrayFromDriver, NOT_AN_ARRAY } from './pg-array-decode'; +import { decodeRow } from './pg-row'; +import { clearRegistry } from './registry'; + +const A = '01a12719-28e5-735e-b5bb-000000000001'; +const B = '01a12719-28e5-735e-b5bb-000000000002'; + +const rows = entity('array_decode_rows', { + columns: { + id: uuid().primaryKey(), + label: text(), + refs: arrayOf(uuid()).nullable(), + gaps: arrayOf(uuid().nullable()).nullable(), + tags: arrayOf(text()).nullable(), + counts: arrayOf(integer().nullable()).nullable(), + flags: arrayOf(boolean().nullable()).nullable(), + ats: arrayOf(timestamp()).nullable(), + rates: arrayOf(decimal()).nullable(), + days: arrayOf(date()).nullable(), + bigs: arrayOf(bigint()).nullable(), + }, +}); + +afterAll(() => { + clearRegistry(); +}); + +const decode = (cells: Readonly>) => + decodeRow(rows, { id: A, label: 'x', ...cells }); + +/** The refusal's code and cause, or the value that came back when nothing was refused. */ +const refusal = (work: () => unknown): { code: string; cause: string } | { value: unknown } => { + try { + return { value: work() }; + } catch (error) { + if (!isUltimateError(error)) throw error; + return { code: error.code, cause: String(error.cause) }; + } +}; + +describe('decodeRow · what is NOT an array is refused, by the column that declared one', () => { + test.each([ + ['a bare uuid', A], + ['a comma-joined list', `${A},${B}`], + ['an unclosed literal', `{${A}`], + ['a JSON array as text', `["${A}"]`], + ['an empty string', ''], + ])('%s is X_INVARIANT_VIOLATED and names the column', (_name, cell) => { + const outcome = refusal(() => decode({ refs: cell })); + expect(outcome).toMatchObject({ code: 'X_INVARIANT_VIOLATED' }); + expect(JSON.stringify(outcome)).toContain('array_decode_rows.refs'); + expect(JSON.stringify(outcome)).toContain('not a Postgres array literal'); + }); + + test('a number, an object and a boolean are refused the same way', () => { + for (const cell of [7, { 0: A }, true]) { + expect(refusal(() => decode({ refs: cell }))).toMatchObject({ code: 'X_INVARIANT_VIOLATED' }); + } + }); + + test('a well-formed literal of the wrong element is refused by the ELEMENT column', () => { + expect(refusal(() => decode({ refs: '{not-a-uuid}' }))).toMatchObject({ + code: 'X_INVARIANT_VIOLATED', + }); + expect(refusal(() => decode({ counts: '{1,abc}' }))).toMatchObject({ + code: 'X_INVARIANT_VIOLATED', + }); + expect(refusal(() => decode({ counts: '{1.5}' }))).toMatchObject({ + code: 'X_INVARIANT_VIOLATED', + }); + expect(refusal(() => decode({ flags: '{yes}' }))).toMatchObject({ + code: 'X_INVARIANT_VIOLATED', + }); + expect(refusal(() => decode({ ats: '{"not an instant"}' }))).toMatchObject({ + code: 'X_INVARIANT_VIOLATED', + }); + }); + + test('a second dimension is refused: arrayOf() declares one', () => { + expect(refusal(() => decode({ tags: '{{a,b},{c,d}}' }))).toMatchObject({ + code: 'X_INVARIANT_VIOLATED', + }); + }); + + test('a null member is refused where the element is not nullable', () => { + expect(refusal(() => decode({ refs: `{${A},NULL}` }))).toMatchObject({ + code: 'X_INVARIANT_VIOLATED', + }); + }); + + test('a text column holding a literal-shaped string keeps the string', () => { + expect(decode({ label: '{a,b}' }).label).toBe('{a,b}'); + }); +}); + +describe('decodeRow · the literal a driver left unparsed', () => { + test('uuid[] — the shape Bun.SQL hands back', () => { + expect(decode({ refs: `{${A},${B}}` }).refs).toEqual([A, B]); + expect(decode({ refs: '{}' }).refs).toEqual([]); + expect(decode({ gaps: `{${A},NULL}` }).gaps).toEqual([A, null]); + expect(decode({ refs: null }).refs).toBeNull(); + }); + + test('a uuid is still a value: upper case reads back lower-cased', () => { + expect(decode({ refs: `{${A.toUpperCase()}}` }).refs).toEqual([A]); + }); + + test('text[] with every character the literal quotes', () => { + expect(decode({ tags: '{plain,"a,b","q\\"x","b\\\\y","{c}",""," d ","NULL"}' }).tags).toEqual([ + 'plain', + 'a,b', + 'q"x', + 'b\\y', + '{c}', + '', + ' d ', + 'NULL', + ]); + }); + + test('int4[] is numbers, bool[] is booleans, with the null member kept', () => { + expect(decode({ counts: '{1,NULL,-2147483648}' }).counts).toEqual([1, null, -2147483648]); + expect(decode({ flags: '{t,f,NULL}' }).flags).toEqual([true, false, null]); + }); + + test('timestamptz[] is Dates at the instant the offset names', () => { + const { ats } = decode({ + ats: '{"2026-01-02 03:04:05.123456+00","2026-01-02 03:04:05-05","1969-12-31 23:59:59.9995+00"}', + }); + expect(ats?.map((at) => at.toISOString())).toEqual([ + '2026-01-02T03:04:05.123Z', + '2026-01-02T08:04:05.000Z', + '1969-12-31T23:59:59.999Z', + ]); + }); + + test('numeric[], int8[] and date[] keep the text Postgres wrote', () => { + expect(decode({ rates: '{1.50,-0.5}' }).rates).toEqual(['1.50', '-0.5']); + expect(decode({ bigs: '{9007199254740993,-1}' }).bigs).toEqual(['9007199254740993', '-1']); + expect(decode({ days: '{2026-03-14,1970-01-01}' }).days).toEqual(['2026-03-14', '1970-01-01']); + }); +}); + +describe('decodeRow · the other two shapes', () => { + test('a JS array is parsed member by member, as it always was', () => { + expect(decode({ refs: [A, B] }).refs).toEqual([A, B]); + expect(decode({ counts: [1, null] }).counts).toEqual([1, null]); + }); + + test('a typed array — Bun.SQL on a preparing pool — is a plain array of its numbers', () => { + const { counts } = decode({ counts: new Int32Array([1, 2, 3]) }); + expect(counts).toEqual([1, 2, 3]); + expect(Array.isArray(counts)).toBe(true); + expect(decode({ counts: new Int32Array([]) }).counts).toEqual([]); + }); +}); + +describe('arrayFromDriver', () => { + const meta = rows.$columns.refs.$meta; + + test('answers NOT_AN_ARRAY rather than guessing', () => { + for (const cell of ['a,b', '', 7, {}, new DataView(new ArrayBuffer(4)), undefined, null]) { + expect(arrayFromDriver(meta, cell)).toBe(NOT_AN_ARRAY); + } + }); + + test('hands a JS array back untouched, so no member is copied on the common path', () => { + const cell = [A]; + expect(arrayFromDriver(meta, cell)).toBe(cell); + }); +}); diff --git a/packages/entity/src/pg-array-decode.ts b/packages/entity/src/pg-array-decode.ts new file mode 100644 index 000000000..6806d0144 --- /dev/null +++ b/packages/entity/src/pg-array-decode.ts @@ -0,0 +1,82 @@ +// Single responsibility: an `arrayOf()` cell as the driver handed it back -> the JS array its +// element column can parse. A driver decides PER TYPE OID and PER PROTOCOL what an array cell is, +// and the repository may not depend on that table: +// +// * `Bun.SQL` parses the array types it knows and hands every other one back as its TEXT +// literal — `uuid[]` is `'{01a1…,01a2…}'` (measured, Bun 1.4.2 against Postgres 17 and 18), +// which `arrayOf(uuid())` refused as "expected an array, got a string" on every read while +// PGlite, which `x dev` and every non-live suite run on, parsed it; +// * `Bun.SQL` hands `int4[]` back as an `Int32Array` whenever the statement carries a parameter +// (the extended protocol's binary result) and as an `Array` when it carries none. +// +// Neither is decided by a type-parser registration: `Bun.SQL` has none, and a table keyed by oid +// would miss the next type it does not know — a domain or an extension type has a per-database +// oid. The DECLARED column is the only thing that knows this cell is an array, so the literal is +// read here, keyed by the declaration and by nothing the driver says. + +import { instantMicros } from './instant'; +import { parsePgArray } from './pg-array-literal'; +import type { ColumnKind, ColumnMeta } from './types'; + +const MICROS_PER_MILLI = 1000n; + +/** Floor, so a pre-1970 instant with a sub-millisecond part lands on the millisecond below it. */ +const millisOf = (micros: bigint): number => { + const remainder = ((micros % MICROS_PER_MILLI) + MICROS_PER_MILLI) % MICROS_PER_MILLI; + return Number((micros - remainder) / MICROS_PER_MILLI); +}; + +const WHOLE = /^-?\d+$/; + +/** + * One element's TEXT -> the value a driver that parsed the array would have handed over, which is + * what each column's `$parse` was written against: a `number` for `integer`, a `boolean` for + * `boolean`, a `Date` for `timestamptz`, and the text itself for every kind whose row value IS + * text (`uuid`, `text`, `numeric`, `bigint`, `date`). + * + * A text this cannot read is returned AS IT IS, never coerced: the element column refuses it with + * its own words, where `Number('abc')` would have stored a `NaN` nobody can find again. + */ +const elementOf = (kind: ColumnKind | undefined, raw: string): unknown => { + switch (kind) { + case 'integer': + return WHOLE.test(raw) ? Number(raw) : raw; + case 'boolean': + return raw === 't' || raw === 'true' ? true : raw === 'f' || raw === 'false' ? false : raw; + case 'timestamptz': { + // `2026-01-02 03:04:05.123456+00`: Postgres names the offset, so the text is an instant. + const micros = instantMicros(raw); + return micros === undefined || WHOLE.test(raw) ? raw : new Date(millisOf(micros)); + } + default: + return raw; + } +}; + +/** Every integer and float typed array; a `DataView` is bytes with no elements to list. */ +const isTypedArray = (value: unknown): value is ArrayLike => + ArrayBuffer.isView(value) && !(value instanceof DataView); + +/** What `arrayFromDriver` answers when the cell is neither an array nor a literal of one. */ +export const NOT_AN_ARRAY = Symbol('not-an-array'); + +/** + * The cell of a column DECLARED `arrayOf()`, as a plain JS array of driver-shaped elements — or + * `NOT_AN_ARRAY` when it is none of the three shapes a driver produces. Called for an array column + * only: a string in a `text()` column is a string, and nothing here ever asks whether it "looks + * like" a literal. + * + * A nested literal (`{{a,b},{c,d}}`) parses to nested arrays and is refused by the element column + * one line later — `arrayOf()` declares one dimension, so a second one is the table disagreeing + * with the entity, said by the column that knows its own type. + */ +export const arrayFromDriver = ( + meta: ColumnMeta, + value: unknown, +): readonly unknown[] | typeof NOT_AN_ARRAY => { + if (Array.isArray(value)) return value; + if (isTypedArray(value)) return Array.from(value); + if (typeof value !== 'string') return NOT_AN_ARRAY; + const kind = meta.element?.$meta.kind; + return parsePgArray(value, (raw) => elementOf(kind, raw)) ?? NOT_AN_ARRAY; +}; diff --git a/packages/entity/src/pg-array-literal.test.ts b/packages/entity/src/pg-array-literal.test.ts new file mode 100644 index 000000000..7eab97104 --- /dev/null +++ b/packages/entity/src/pg-array-literal.test.ts @@ -0,0 +1,90 @@ +// The Postgres array literal grammar: what is refused first, then what is read. The texts are the +// ones a server prints (`select array[...]::text`), so each case is a claim Postgres made and this +// reader has to agree with — the live half is `column-matrix.live.test.ts`. + +import { describe, expect, test } from 'bun:test'; +import { parsePgArray } from './pg-array-literal'; +import { arrayLiteral } from './pg-row'; + +const raw = (text: string) => parsePgArray(text, (element) => element); + +describe('parsePgArray · a text that is not an array literal is null, never a partial array', () => { + test.each([ + ['an empty string', ''], + ['a bare scalar', 'a'], + ['a uuid on its own', '01a12719-28e5-735e-b5bb-000000000000'], + ['an unclosed brace', '{a,b'], + ['an unopened brace', 'a,b}'], + ['a quote that never closes', '{"a,b}'], + ['a backslash at the very end', '{"a\\'], + ['content after the closing brace', '{a}b'], + ['a second array after the first', '{a}{b}'], + ['a quote inside a bare element', '{a"b}'], + ['a brace inside a bare element', '{a{b}'], + ['text straight after a closing quote', '{"a"b}'], + ['a dimension prefix', '[0:1]={a,b}'], + ['a JSON array', '["a","b"]'], + ['what Bun.SQL sends for a JS array', 'a,b'], + ])('%s', (_name, text) => { + expect(raw(text)).toBeNull(); + }); +}); + +describe('parsePgArray · elements', () => { + test('the empty array is an array of nothing, not of one empty string', () => { + expect(raw('{}')).toEqual([]); + }); + + test('bare elements are split on commas and kept as written', () => { + expect(raw('{a,b,c}')).toEqual(['a', 'b', 'c']); + expect(raw('{01a12719-28e5-735e-b5bb-000000000000}')).toEqual([ + '01a12719-28e5-735e-b5bb-000000000000', + ]); + }); + + test('a quoted element keeps its comma, its braces and its padding', () => { + expect(raw('{"a,b","{c}"," d "}')).toEqual(['a,b', '{c}', ' d ']); + }); + + test('a backslash escapes exactly one character inside quotes', () => { + expect(raw('{"q\\"x","b\\\\y"}')).toEqual(['q"x', 'b\\y']); + }); + + test('a quoted empty string is an element; an unquoted NULL is the null member', () => { + expect(raw('{"",NULL,null,"NULL"}')).toEqual(['', null, null, 'NULL']); + }); + + test('the null member never reaches the element decoder', () => { + const seen: string[] = []; + parsePgArray('{1,NULL,3}', (element) => seen.push(element)); + expect(seen).toEqual(['1', '3']); + }); + + test('a line break and a tab inside quotes are data', () => { + expect(raw('{"line\nbreak","tab\there"}')).toEqual(['line\nbreak', 'tab\there']); + }); + + test('a second dimension is a nested array', () => { + expect(raw('{{a,b},{c,d}}')).toEqual([ + ['a', 'b'], + ['c', 'd'], + ]); + }); + + test('the decoder decides the element type', () => { + expect(parsePgArray('{1,2}', Number)).toEqual([1, 2]); + }); +}); + +describe('parsePgArray · reads what arrayLiteral writes', () => { + test.each([ + [[]], + [['plain']], + [['with,comma', 'quote"inside', 'back\\slash', '{braces}', '', ' padded ']], + [['NULL', 'null', null]], + [['line\nbreak', 'tab\there', 'ünï©ode ✓', "single'quote", '\\"', '"\\']], + [[null, null]], + ])('%j', (members) => { + expect(raw(arrayLiteral(members))).toEqual(members); + }); +}); diff --git a/packages/entity/src/pg-array-literal.ts b/packages/entity/src/pg-array-literal.ts new file mode 100644 index 000000000..d7e527603 --- /dev/null +++ b/packages/entity/src/pg-array-literal.ts @@ -0,0 +1,106 @@ +// Single responsibility: the Postgres array TEXT literal — `{a,b}`, `{"a,b",NULL}`, +// `{{1,2},{3,4}}` — read into a JS array. The one grammar in the framework: the repository reads a +// literal a driver left unparsed (`pg-array-decode.ts`) and `@ultimat3/realtime` reads the one the +// WAL carries, and two readers of one grammar are how a quoted `"NULL"` becomes a null in one of +// them. The element decoder arrives as a parameter, so this file knows no type. +// +// The writer is `arrayLiteral` in `pg-row.ts`; `pg-array-literal.test.ts` holds the two to each +// other, so what this package writes is what it can read. + +/** Where the scan is, so every branch below advances one cursor rather than slicing the text. */ +interface Scan { + readonly text: string; + at: number; +} + +/** + * One array literal -> a nested JS array, or `null` when the text is not one this grammar + * describes. `null` is not an error: a dimension prefix (`[0:2]={…}`), a `DateStyle` this decoder + * does not read, or a corrupted literal all mean the same thing to the caller — keep the text it + * arrived as rather than deliver an array that is missing a member. + * + * `decode` is the ELEMENT decoder. It never sees a quoted element's quotes or its backslash + * escapes: an unquoted `NULL` is the null value and a quoted `"NULL"` is the four-character + * string, which is the one distinction the quoting exists to carry. + */ +export function parsePgArray(text: string, decode: (raw: string) => T): PgArray | null { + const scan: Scan = { text, at: 0 }; + const parsed = readArray(scan, decode); + // Trailing content means the literal was not what this grammar accepted, whatever it parsed. + return parsed === null || scan.at !== text.length ? null : parsed; +} + +/** A member is a decoded value, the null member, or — for a multidimensional array — a row of them. */ +export type PgArray = (T | null | PgArray)[]; + +function readArray(scan: Scan, decode: (raw: string) => T): PgArray | null { + if (scan.text[scan.at] !== '{') return null; + scan.at += 1; + const out: PgArray = []; + if (scan.text[scan.at] === '}') { + scan.at += 1; + return out; + } + for (;;) { + const member = readMember(scan, decode); + if (member === FAILED) return null; + out.push(member); + const next = scan.text[scan.at]; + scan.at += 1; + if (next === '}') return out; + if (next !== ',') return null; + } +} + +/** A sentinel, because `null` is a legal member and `undefined` would be a second spelling of it. */ +const FAILED = Symbol('pg-array-failed'); + +function readMember( + scan: Scan, + decode: (raw: string) => T, +): T | null | PgArray | typeof FAILED { + const head = scan.text[scan.at]; + if (head === '{') { + const nested = readArray(scan, decode); + return nested === null ? FAILED : nested; + } + if (head === '"') { + const quoted = readQuoted(scan); + return quoted === null ? FAILED : decode(quoted); + } + const start = scan.at; + while (scan.at < scan.text.length) { + const char = scan.text[scan.at]; + if (char === ',' || char === '}') break; + // A brace or a quote inside a bare element is a literal this grammar does not describe. + if (char === '{' || char === '"') return FAILED; + scan.at += 1; + } + if (scan.at === scan.text.length) return FAILED; + const raw = scan.text.slice(start, scan.at); + // Unquoted and case-insensitive is the ONLY spelling of the null member; `"NULL"` is a string. + return raw.toUpperCase() === 'NULL' ? null : decode(raw); +} + +/** The text between one pair of quotes, with `\\` and `\"` unescaped. `null` if it never closes. */ +function readQuoted(scan: Scan): string | null { + scan.at += 1; + let out = ''; + while (scan.at < scan.text.length) { + const char = scan.text[scan.at]; + if (char === '"') { + scan.at += 1; + return out; + } + if (char === '\\') { + const escaped = scan.text[scan.at + 1]; + if (escaped === undefined) return null; + out += escaped; + scan.at += 2; + continue; + } + out += char ?? ''; + scan.at += 1; + } + return null; +} diff --git a/packages/entity/src/pg-row.ts b/packages/entity/src/pg-row.ts index eaacd2e22..34ff9f93c 100644 --- a/packages/entity/src/pg-row.ts +++ b/packages/entity/src/pg-row.ts @@ -10,6 +10,7 @@ import { kindOf } from './cursor'; import type { EntityCore } from './entity'; import { invariantViolated } from './errors'; import { pgInstantMicros, seekAlias } from './instant'; +import { arrayFromDriver, NOT_AN_ARRAY } from './pg-array-decode'; import type { SortKey } from './tenancy'; import type { AnyColumn, MoneyValue, RowPatch } from './types'; @@ -168,6 +169,26 @@ const moneyOf = ( }; }; +/** + * What a column's `$parse` is handed. Every kind but one takes the driver's value as it is; an + * `arrayOf()` cell is first made the JS array its declaration says it is, because a driver hands + * one back three ways (`pg-array-decode.ts`) and only one of them is an `Array`. + * + * A cell that is none of the three is the table no longer matching the entity, and it is said + * here — `arrayOf`'s own refusal is written for a WRITE ("wrap it — [value]"), which instructs + * nobody about a value Postgres returned. + */ +const cellOf = (entityName: string, property: string, column: AnyColumn, value: unknown) => { + if (column.$meta.kind !== 'array') return value; + const members = arrayFromDriver(column.$meta, value); + if (members !== NOT_AN_ARRAY) return members; + throw invariantViolated( + entityName, + property, + `the database returned ${typeof value === 'string' ? `text that is not a Postgres array literal (${value.length} characters)` : `a ${typeof value}`} for an arrayOf() column — the table no longer matches the entity`, + ); +}; + /** * Physical row -> entity row. A column the projection left out is left out here too, so a * `select` narrows the object as well as the statement. @@ -188,7 +209,10 @@ export const decodeRow = (entity: EntityCore, source: PhysicalRow): Ro // A sealed column's stored value is not the author's to parse: it goes up as it is, and the // sealing seam opens it — so a row somebody left in the clear is `X_SEAL_INVALID` there // rather than whatever the plaintext parser thinks of its length here. - row[property] = column.$meta.sealed === undefined ? column.$parse(value) : value; + row[property] = + column.$meta.sealed === undefined + ? column.$parse(cellOf(entity.$name, property, column, value)) + : value; continue; } if (column.$meta.notNull) { diff --git a/packages/realtime/src/pg-array.ts b/packages/realtime/src/pg-array.ts index 9a8f562c8..08bd48ea1 100644 --- a/packages/realtime/src/pg-array.ts +++ b/packages/realtime/src/pg-array.ts @@ -1,10 +1,10 @@ -// The Postgres array TEXT literal — `{a,b}`, `{"a,b",NULL}`, `{{1,2},{3,4}}` — and the element -// type behind an array oid. Split from `pg-values.ts` so that file stays the oid switch and this -// one stays the grammar; the element decoder arrives as a parameter, so neither imports the other. +// The element type behind an array type oid. The WAL names only the ARRAY type and carries the +// value as its TEXT literal (`{a,b}`), while `@ultimat3/entity`'s repository hands an `arrayOf()` +// column back as a JS array — and a live row and a repository row have to be the same object. // -// It exists because `@ultimat3/entity`'s repository hands an `arrayOf()` column back as a JS array -// (the driver parses the literal) while the WAL carries the literal itself. A live row and a -// repository row have to be the same object, so the literal is parsed here rather than shipped. +// The GRAMMAR of that literal is `@ultimat3/entity`'s `parsePgArray`: the repository reads the +// same text when a driver leaves an array unparsed, and one grammar has one reader. This file +// keeps the half only the WAL needs — which element decoder an oid selects. /** * `text[]` -> `text`. The wire names only the ARRAY type, so the element type behind it is a @@ -41,101 +41,3 @@ const ELEMENT_OF = Object.freeze>({ export function arrayElementOid(typeOid: number): number | undefined { return Object.hasOwn(ELEMENT_OF, typeOid) ? ELEMENT_OF[typeOid] : undefined; } - -/** Where the scan is, so every branch below advances one cursor rather than slicing the text. */ -interface Scan { - readonly text: string; - at: number; -} - -/** - * One array literal -> a nested JS array, or `null` when the text is not one this grammar - * describes. `null` is not an error: a dimension prefix (`[0:2]={…}`), a `DateStyle` this decoder - * does not read, or a corrupted literal all mean the same thing to the caller — keep the text it - * arrived as rather than deliver an array that is missing a member. - * - * `decode` is the ELEMENT decoder. It never sees a quoted element's quotes or its backslash - * escapes: an unquoted `NULL` is the null value and a quoted `"NULL"` is the four-character - * string, which is the one distinction the quoting exists to carry. - */ -export function parsePgArray(text: string, decode: (raw: string) => T): PgArray | null { - const scan: Scan = { text, at: 0 }; - const parsed = readArray(scan, decode); - // Trailing content means the literal was not what this grammar accepted, whatever it parsed. - return parsed === null || scan.at !== text.length ? null : parsed; -} - -/** A member is a decoded value, the null member, or — for a multidimensional array — a row of them. */ -export type PgArray = (T | null | PgArray)[]; - -function readArray(scan: Scan, decode: (raw: string) => T): PgArray | null { - if (scan.text[scan.at] !== '{') return null; - scan.at += 1; - const out: PgArray = []; - if (scan.text[scan.at] === '}') { - scan.at += 1; - return out; - } - for (;;) { - const member = readMember(scan, decode); - if (member === FAILED) return null; - out.push(member); - const next = scan.text[scan.at]; - scan.at += 1; - if (next === '}') return out; - if (next !== ',') return null; - } -} - -/** A sentinel, because `null` is a legal member and `undefined` would be a second spelling of it. */ -const FAILED = Symbol('pg-array-failed'); - -function readMember( - scan: Scan, - decode: (raw: string) => T, -): T | null | PgArray | typeof FAILED { - const head = scan.text[scan.at]; - if (head === '{') { - const nested = readArray(scan, decode); - return nested === null ? FAILED : nested; - } - if (head === '"') { - const quoted = readQuoted(scan); - return quoted === null ? FAILED : decode(quoted); - } - const start = scan.at; - while (scan.at < scan.text.length) { - const char = scan.text[scan.at]; - if (char === ',' || char === '}') break; - // A brace or a quote inside a bare element is a literal this grammar does not describe. - if (char === '{' || char === '"') return FAILED; - scan.at += 1; - } - if (scan.at === scan.text.length) return FAILED; - const raw = scan.text.slice(start, scan.at); - // Unquoted and case-insensitive is the ONLY spelling of the null member; `"NULL"` is a string. - return raw.toUpperCase() === 'NULL' ? null : decode(raw); -} - -/** The text between one pair of quotes, with `\\` and `\"` unescaped. `null` if it never closes. */ -function readQuoted(scan: Scan): string | null { - scan.at += 1; - let out = ''; - while (scan.at < scan.text.length) { - const char = scan.text[scan.at]; - if (char === '"') { - scan.at += 1; - return out; - } - if (char === '\\') { - const escaped = scan.text[scan.at + 1]; - if (escaped === undefined) return null; - out += escaped; - scan.at += 2; - continue; - } - out += char ?? ''; - scan.at += 1; - } - return null; -} diff --git a/packages/realtime/src/pg-values.test.ts b/packages/realtime/src/pg-values.test.ts index f60bd826f..ac3b9569e 100644 --- a/packages/realtime/src/pg-values.test.ts +++ b/packages/realtime/src/pg-values.test.ts @@ -4,7 +4,8 @@ // `Date`'s epoch number and which `post.tags.map(…)` throws on. import { describe, expect, test } from 'bun:test'; -import { arrayElementOid, parsePgArray } from './pg-array'; +import { parsePgArray } from '@ultimat3/entity'; +import { arrayElementOid } from './pg-array'; import { decodeValue } from './pg-values'; const TIMESTAMPTZ = 1184; diff --git a/packages/realtime/src/pg-values.ts b/packages/realtime/src/pg-values.ts index 5871da50e..148b041f9 100644 --- a/packages/realtime/src/pg-values.ts +++ b/packages/realtime/src/pg-values.ts @@ -11,9 +11,10 @@ // in the component the first patch reached. import { renderThrowable } from '@ultimat3/core'; +import { parsePgArray } from '@ultimat3/entity'; import { ReplicationProtocolError } from './errors'; import type { JsonValue } from './json'; -import { arrayElementOid, parsePgArray } from './pg-array'; +import { arrayElementOid } from './pg-array'; /** * What a row value can be between the WAL and the wire: JSON, plus the two JS shapes a repository diff --git a/wiki/Entities-And-Migrations.md b/wiki/Entities-And-Migrations.md index 1f9c31545..33d617b5d 100644 --- a/wiki/Entities-And-Migrations.md +++ b/wiki/Entities-And-Migrations.md @@ -493,7 +493,7 @@ is the shape a table already has, `As of 2026-08`. | `date()` | `date` | `PlainDate` — a calendar date, no time, no zone | | `bigint()` | `bigint` | `string`. A JS `bigint` is what `JSON.stringify` throws on and a `number` loses digits past 2^53, which is where a legacy `int8` key lives | | `bytes()` | `bytea` | `Uint8Array`, normalised across both drivers | -| `arrayOf(column)` | `[]` | `readonly T[]`, each member parsed by the element column | +| `arrayOf(column)` | `[]` | `readonly T[]`, each member parsed by the element column. Elements: `uuid`, `text` (and `url`, `enumerated`, `tz`, `locale`), `integer`, `boolean`, `timestamp`, `decimal`, `date`, `bigint`, each optionally `.nullable()`. The row is the same on both drivers: a driver that returns an array as its text literal (`Bun.SQL` does for `uuid[]`) is read by the column's declaration | **No NUL in text.** `text()` and `url()` refuse a string carrying U+0000 (`X_INVARIANT_VIOLATED`, `column.format`) — the one character a Postgres `text` value cannot hold (SQLSTATE 22021), so the From 411ca72e272b69cd51b7592ce49d634b94c36faf Mon Sep 17 00:00:00 2001 From: sebi Date: Sat, 10 Oct 2026 15:11:20 -0500 Subject: [PATCH 2/2] test(entity,cli): the date[] decode test compares plain dates; serve-graph pins follow the two entity modules (measured) Co-Authored-By: Claude Opus 5.5 --- packages/cli/src/serve-graph.test.ts | 14 +++++++++++--- packages/entity/src/pg-array-decode.test.ts | 6 +++++- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/packages/cli/src/serve-graph.test.ts b/packages/cli/src/serve-graph.test.ts index 29d943e74..068e10f3a 100644 --- a/packages/cli/src/serve-graph.test.ts +++ b/packages/cli/src/serve-graph.test.ts @@ -92,8 +92,12 @@ const NEVER_AT_BOOT: readonly (readonly [string, RegExp])[] = [ * both reached through the core barrel. * raised 636 → 637, measured 637 (2026-10-09, #710 row 9): `auth/src/login-identity.ts` — which * account a login names, an email or a handle `defineAuth({ handles })` resolves, through the auth barrel. + * raised 637 → 639, measured 639 (2026-10-10, #752): `entity/src/pg-array-literal.ts` and + * `entity/src/pg-array-decode.ts` — an `arrayOf()` cell read by its declaration, because `Bun.SQL` + * hands `uuid[]` back as its text literal and every read of such a row was refused. The grammar + * moved out of `realtime/src/pg-array.ts`, which stays for its oid table, so nothing left the graph. */ -const MIGRATE_CEILING = 637; +const MIGRATE_CEILING = 639; /** * measured: 796 — the 558 above plus what `serve-boot.ts` adds: the services and the roles. @@ -175,8 +179,10 @@ const MIGRATE_CEILING = 637; * raised 935 → 936, measured 936 (2026-10-10, channel events from app code): * `realtime/src/channel-publish.ts` — `publishChannelEvent` and the process bus the boot installs * (`setChannelTransport`), so a job or an action publishes on a declared channel with no hub. + * raised 936 → 938, measured 938 (2026-10-10, #752): the two entity modules named on + * `MIGRATE_CEILING` for #752, which every role reaches through the entity barrel. */ -const SERVING_ROLE_CEILING = 936; +const SERVING_ROLE_CEILING = 938; /** * measured: 888 — the 796 above plus the 92 `serve-web.ts` adds (41 CLI, 36 MCP, 15 PWA). @@ -255,8 +261,10 @@ const SERVING_ROLE_CEILING = 936; * raised 1062 → 1063, measured 1063 (2026-10-10, events-only islands are never held): * `cli/src/island-realtime-state.ts` — which realtime islands read the record store or the outbox * (held, #506) and which only follow a channel, read off the names each imports from the barrel. + * raised 1063 → 1065, measured 1065 (2026-10-10, #752): the two entity modules named on + * `MIGRATE_CEILING` for #752. */ -const WEB_ROLE_CEILING = 1063; +const WEB_ROLE_CEILING = 1065; interface MetaInput { readonly imports: readonly { readonly path: string; readonly kind: string }[]; diff --git a/packages/entity/src/pg-array-decode.test.ts b/packages/entity/src/pg-array-decode.test.ts index 0c347e857..dc66d8e4f 100644 --- a/packages/entity/src/pg-array-decode.test.ts +++ b/packages/entity/src/pg-array-decode.test.ts @@ -4,6 +4,7 @@ import { afterAll, describe, expect, test } from 'bun:test'; import { isUltimateError } from '@ultimat3/core'; +import { plainDate } from '@ultimat3/time'; import { boolean, integer, text, timestamp, uuid } from './columns'; import { arrayOf, bigint, date, decimal } from './columns-data'; import { entity } from './entity'; @@ -146,7 +147,10 @@ describe('decodeRow · the literal a driver left unparsed', () => { test('numeric[], int8[] and date[] keep the text Postgres wrote', () => { expect(decode({ rates: '{1.50,-0.5}' }).rates).toEqual(['1.50', '-0.5']); expect(decode({ bigs: '{9007199254740993,-1}' }).bigs).toEqual(['9007199254740993', '-1']); - expect(decode({ days: '{2026-03-14,1970-01-01}' }).days).toEqual(['2026-03-14', '1970-01-01']); + expect(decode({ days: '{2026-03-14,1970-01-01}' }).days).toEqual([ + plainDate('2026-03-14'), + plainDate('1970-01-01'), + ]); }); });