Script IO bridges: Collections + wire all five bridges (dbQuery via Data Sources)

docs/design/10-collections.md

@@ -0,0 +1,153 @@
+# 10 — Collections
+
+> Status: design (approved 2026-07-13). A durable, queryable, TTL-pruned keyed
+> record store that channel scripts read and write via a `getCollection()` bridge.
+
+## Why
+
+Channels frequently need to **stash a record now and look it up later from a
+different channel** — the canonical case (live in the user's Mirth prod at
+~100K msgs/day) is order/report matching: an orders channel stores each inbound
+order keyed by accession + institution; later a reports channel looks up the
+newest matching order (filtered by order-control type) to build the outbound
+report. See `reference` memory `vns-mirth-prod-source` →
+`portalApi.hl7message.store/find` and `valor-network/core/populateReportMsg.js`.
+
+This is **not** a [Resource](#relationship-to-resources): resources are static
+config blobs (read-only, cache-until-invalidated). A collection is mutable,
+high-churn, keyed, queryable state. They are separate features.
+
+Nor is it a **map** (`globalMap`/`channelMap`): maps are in-memory, ephemeral,
+lost on restart, and not queryable. A collection is durable and indexed.
+
+## Model
+
+A **Collection** is defined once (in the UI / API):
+
+| Field | Meaning |
+|---|---|
+| `name` | unique; how scripts address it (`getCollection('orders')`) |
+| `description` | free text |
+| `indexedFields` | ordered list of user-defined field names that are queryable (e.g. `["accessionNumber","institutionName","orderControl","messageCode","triggerEvent"]`) |
+| `defaultTtlSeconds` | default record lifetime; `null` = never expire. Applied at write time unless the write overrides it. |
+
+A **CollectionRecord** is one stored row:
+
+| Field | Meaning |
+|---|---|
+| `id` | uuid, unique per record (append — many records per key) |
+| `collectionId` | FK to the collection |
+| `fields` | jsonb — the indexed field values supplied at write time |
+| `payload` | the stored value (HL7 text, JSON, whatever) |
+| `expireAt` | timestamptz, nullable — when the pruner removes it |
+| `createdAt` | timestamptz — drives newest-wins ordering |
+
+Records are **append-only**: `store()` inserts, never upserts. "Newest wins" is a
+query result (`ORDER BY created_at DESC LIMIT 1`), not a storage constraint. This
+matches the domain (many orders accrue per accession over time) and avoids
+read-modify-write races at high write volume.
+
+## Storage
+
+One static table (no per-collection dynamic DDL — parameterized queries only,
+Postgres-native, Drizzle-friendly):
+
+```
+collection_records(
+  id           uuid pk default gen_random_uuid(),
+  collection_id uuid not null references collections(id) on delete cascade,
+  fields       jsonb not null,
+  payload      text,                    -- opaque to the store; scripts parse
+  expire_at    timestamptz,             -- null = never
+  created_at   timestamptz not null default now()
+)
+  index gin (fields)                              -- @> containment (equality match)
+  index btree (collection_id, created_at desc)    -- newest-wins hot path
+  index btree (expire_at) where expire_at is not null  -- pruner scan
+```
+
+## Query surface (bounded — not a query language)
+
+`find(match, options)`:
+- **`match`** — object of `{field: value}`, equality on indexed fields, AND'd.
+  Implemented as `fields @> $match` (GIN-indexed). This is the fast key lookup.
+- **`options.filter`** — optional `{field: value | value[]}`, **multiple fields**,
+  AND'd; a scalar is equality, an array is `IN`. Implemented as
+  `fields->>'f' = $v` / `fields->>'f' = ANY($arr)`.
+- **`options.latest`** — boolean; return the single newest match (or `null`).
+- **`options.limit` / `options.order`** — optional; default order is
+  `created_at desc`.
+
+No joins, ranges, or partial-key scans in v1. Both known use cases fit.
+
+## TTL
+
+`defaultTtlSeconds` on the collection is the primary mechanism (the user's prod
+sets a backend default and rarely overrides — and when they do it's per-customer,
+not per-message). At `store()`:
+- `expireAt = now + (override ?? defaultTtlSeconds)`; if both are null → never
+  expires.
+- Override is an optional `{ expireAt }` (or `{ ttlSeconds }`) on the write.
+
+Pruning reuses the existing data-pruner scheduler: periodic
+`DELETE FROM collection_records WHERE expire_at < now()`.
+
+## Script bridge
+
+Shaped to mirror the prod `portalApi.hl7message` API so migrating existing
+channel code (`getOrder`) is mechanical:
+
+```js
+// Write (orders channel)
+getCollection('orders').store(
+  { accessionNumber, institutionName, orderControl, messageCode, triggerEvent },
+  hl7.toString(),
+  { expireAt }                       // optional; else collection default TTL
+);
+
+// Read newest match (reports channel) — same semantics as prod getOrder()
+const order = getCollection('orders').find(
+  { accessionNumber, institutionName },
+  { filter: { orderControl: ['XO', 'NW', 'SC'] }, latest: true }
+);   // → { id, fields, payload, createdAt } | null
+```
+
+The bridge is injected the same way as the other IO bridges, at the currently
+unpassed-deps construction point `packages/server/src/engine.ts` (`new
+VmSandboxExecutor(...)`). Reads hit Postgres directly — collections are mutable,
+so (unlike `getResource`) there is **no read cache**.
+
+## Relationship to Resources
+
+| | Resource | Collection |
+|---|---|---|
+| Shape | one named text blob | many keyed records |
+| Mutability | rarely edited config | high-churn writes |
+| Read | `getResource(name)` → string | `getCollection(name).find(...)` → records |
+| Caching | cache-until-invalidated | no read cache |
+| TTL | none | per-collection default + per-write override |
+| UI | Resources page | Collections page |
+
+Separate DB tables, services, routes, bridges, and pages. Wiring `getResource`
+(config) is tracked separately; this doc covers Collections only.
+
+## Security / limits
+
+- Any channel script can read/write any collection by name (no per-channel
+  scoping in v1 — noted as a future option). PHI lives in `payload`; access is
+  auditable via the same event trail as other services.
+- Enforce a max `payload` size on write (reject oversized).
+- Not a secret store.
+
+## Build order
+
+1. **core-models** — Zod schemas (collection def, record, `find` query input),
+   branded `CollectionId`, field-value canonicalization.
+2. **server** — `collections` + `collection_records` tables + migration;
+   `CollectionService` (define/list/delete; `store`/`find` parameterized);
+   routes + RBAC (`collections:read/write/delete`); pruner hook.
+3. **engine** — inject `getCollection` into `VmSandboxExecutor`; in-sandbox
+   global; `.d.ts` for script IntelliSense.
+4. **web** — Collections page (define fields + TTL, browse/inspect records).
+5. **tests + docs** — sandbox bridge tests, server unit + integration tests,
+   `docs/testing/` checklist, scripting-api docs, progress docs.

docs/design/11-datasources.md

@@ -0,0 +1,131 @@
+# 11 — Data Sources (`dbQuery` bridge)
+
+> Status: design (approved 2026-07-13). Named, admin-managed database connection
+> profiles that channel scripts query via `dbQuery(dataSourceName, sql, params)`.
+> This replaces the original, unwired `dbQuery(driver, connectionUrl, sql, params)`
+> signature — see the reframe below.
+
+## Why not the original signature
+
+The sandbox declared `dbQuery(driver, connectionUrl, sql, params)` — the **script**
+supplies the driver, host, and credentials inline. For a healthcare engine that is
+the wrong shape, and it is why the bridge was left unwired (the other four bridges
+shipped; see [`10-collections.md`](10-collections.md) and D-177):
+
+- **Credential sprawl** — DB passwords live in channel scripts (plaintext in config,
+  surfaced in error logs, no rotation).
+- **Unbounded reach** — a script can connect to any host with any credentials it can
+  construct: an exfiltration/SSRF primitive with no allowlist.
+- **No stable pool key** — per-call URLs can't be pooled cleanly.
+
+The bridge is unwired, so the signature is free to change. **Decision: named Data
+Sources.** An admin defines connection profiles server-side; scripts reference them
+by name. Credentials never touch scripts, the admin decides which databases are
+reachable (allowlist *by construction*), and the pool key is the data-source id.
+
+```js
+const rows = await dbQuery('reporting-db',
+  'SELECT report FROM reports WHERE accession = $1 ORDER BY created_at DESC LIMIT 1',
+  [accession]);
+```
+
+This also sidesteps URL parsing: `ConnectionPool.create` already takes structured
+`PoolConfig` (host/port/db/user/password), so a Data Source *is* a `PoolConfig` + a
+name + policy.
+
+## Data model — `data_sources`
+
+| Field | Notes |
+|---|---|
+| `id` | uuid |
+| `name` | unique; how scripts address it |
+| `description` | free text |
+| `driver` | `'postgres'` only in v1 (validated); stored so more drivers can be added |
+| `host`, `port`, `database`, `user` | structured connection config |
+| `password` | **encrypted at rest** via `content-crypto` (`encryptContent`/`decryptContent`, same `CONTENT_ENCRYPTION_KEY` as PHI); decrypted only when building the pool; never logged; never returned by the API |
+| `readOnly` | policy flag, **default true** (see below) |
+| `maxConnections` | pool size (default e.g. 5) |
+| `statementTimeoutMs` | per-query DB-side timeout (default 30s) |
+| `maxRows` | hard cap on rows returned to a script (default e.g. 10 000) |
+| `createdAt`, `updatedAt` | |
+
+New feature parallel to Resources/Collections: CRUD API + a Data Sources admin page,
+RBAC `datasources:read/write/delete` (write/delete for admin + deployer only — they
+hold DB credentials). Note this is *stricter* than today: connector passwords
+currently sit unencrypted in connector-config JSONB, so encrypted Data Sources set
+the pattern to backport later.
+
+**No SSRF host-blocking here** (unlike `httpFetch`): internal databases are the whole
+point, and the admin gate is the control.
+
+## Bridge semantics
+
+`dbQuery(dataSourceName, sql, params?) → readonly Record<string, unknown>[]`:
+
+- **Parameterized only** — `params` is separate from `sql`; docs stress "never
+  string-interpolate values." Passed straight to `pg` as `$1, $2, …`.
+- **Read-only default** — a data source is read-only unless an admin flags it
+  read-write. Enforced by the **DB role** (admin configures a read-only user) plus
+  Postgres `default_transaction_read_only` on read-only pools — *not* by SQL-string
+  sniffing (fragile). A read-write query against a read-only source fails at the DB.
+- **Timeout** — `statement_timeout`/`query_timeout` on the pool bound each query
+  DB-side; the sandbox wall-clock `AbortSignal` still caps the whole script.
+- **Row cap** — enforce `maxRows`; exceeding it fails loud (not silent truncation).
+- **Errors** — surface as thrown errors inside the script (via the bridge's
+  `ioDispatch` ok/err envelope), like the other bridges.
+
+## Pool manager
+
+`DataSourcePoolManager` singleton keyed by data-source id, wrapping the existing
+`packages/connectors/src/database/ConnectionPool`:
+
+- Lazily `create()` a pool on first use; reuse across calls.
+- **Invalidate** (destroy + drop) a pool when its Data Source is updated or deleted.
+  No `RESOURCE_UPDATED`-style event bus exists, so `DataSourceService` mutations call
+  the manager directly (both live in the server process).
+- **Shutdown** — destroy all pools during server teardown (register with the existing
+  graceful-shutdown sequence).
+- Wired into `EngineManager` like the other bridges: one more dep in the
+  `new VmSandboxExecutor({...})` construction.
+
+## Drivers
+
+v1 is **Postgres-only** (reuse `pg`/`ConnectionPool`); `driver` validated to
+`'postgres'`. Define a thin `DbDriver` interface (`createPool(config)`,
+`query(sql, params)`) so MySQL/MSSQL/Oracle are additive later — but don't build them
+now (YAGNI).
+
+## Security summary
+
+- Credentials server-side, encrypted at rest, never logged, never returned by the API.
+- Reachable databases limited to configured sources (allowlist by construction).
+- Read-only by default, enforced by DB role + read-only transactions.
+- Parameterized queries; statement timeout; row cap.
+- `datasources:write/delete` gated to admin + deployer.
+- Optional audit event `DB_QUERY` recording data-source name + row count (not SQL
+  values / params — they may carry PHI).
+
+## Build order (mirrors Collections)
+
+1. **core-models** — `DataSource` schemas (create/update incl. `password`, query
+   input), branded `DataSourceId`. Change the sandbox `dbQuery` bridge signature to
+   `dbQuery(dataSourceName, sql, params)`.
+2. **server** — `data_sources` table (password stored via `encryptContent`);
+   `DataSourceService` (CRUD + `runQuery(name, sql, params)` with row cap +
+   read-only enforcement); `DataSourcePoolManager`; routes + `datasources:*` RBAC;
+   password redaction on all responses.
+3. **engine/sandbox** — update the `dbQuery` types + bootstrap global in
+   `bridge-functions.ts` / `sandbox-executor.ts`; wire `createDbQueryBridge()` into
+   `EngineManager`; restore the `dbQuery` IntelliSense decl in `sandbox-types.ts`.
+4. **web** — Data Sources page (define connection, "Test Connection" button, no
+   password readback), nav item, RBAC gating.
+5. **tests + docs** — service unit tests (CRUD, read-only enforcement, row cap,
+   redaction), a real-Postgres integration test (`dbQuery` against the test DB),
+   pool-manager lifecycle/invalidation tests, sandbox bridge test; `scripting-api.md`
+   update, `docs/testing/` checklist, DECISIONS entry, and this doc.
+
+## Open follow-ups (not v1)
+
+- Additional drivers (MySQL/MSSQL/Oracle) behind the `DbDriver` interface.
+- Backport encrypted-at-rest credentials to the Database *connector* config.
+- An optional config-gated ad-hoc-URL escape hatch (only if a real need appears).

docs/progress/CHANGELOG.md

@@ -1811,3 +1811,40 @@ Fixed six verified release-blocking bugs plus three cheaper related issues in th
 
 ### Security
 - **Sandbox RCE closed** — hardened `VmSandboxExecutor` so no host-realm object/function is reachable from user scripts; `logger.info.constructor('return process')()` and friends can no longer reach host `process`/env. Added escape-attempt tests, an async wall-clock timeout, and removed the dead `memoryLimit` knob. See `packages/engine/src/sandbox/README.md`. (D-129)
+
+## 2026-07-13 — Collections: keyed record store + getCollection bridge (branch: feature/collections)
+
+New feature: a durable, queryable, TTL-pruned keyed record store that channel scripts read/write via `getCollection()`. Motivated by the user's live Mirth prod order/report matching (~100K msgs/day; `portalApi.hl7message.store/find`). Design in `docs/design/10-collections.md`; rationale in D-177.
+
+- **core-models** — `collection.schema.ts` (create/update/store/find + params), branded `CollectionId`, `COLLECTION_UPDATED` event (22 schema tests).
+- **server** — `collections` + `collection_records` tables (JSONB `fields` with GIN + newest-wins + partial-expiry indexes, FK cascade); migration 0009. `CollectionService`: define/list/get/update/delete + `store`/`find` (parameterized `@>` match + multi-field `->>` filter, newest-wins, limit) + `listRecords` + `pruneExpired`; field-value string coercion for GIN consistency; unknown-field + 1 MiB payload guards. Controller/routes at `/collections`; RBAC `collections:read/write/delete` (deployer RWD, developer RW, viewer R). Real-Postgres integration suite (`collection.itest.ts`).
+- **engine** — `getCollection(name)` → `{ store, find }` sandbox bridge (new `collections` BridgeDependency). Fixed `hasAsyncBridges` omitting `collections` (a script using only getCollection was wrapped sync and `await` threw). Wired `createCollectionBridge()` into `VmSandboxExecutor` at `engine.ts` — **the first IO bridge to go live in production** (the executor was constructed with no deps); Zod-validates store/find at the script boundary.
+- **web** — Collections page (define name/indexed-fields/TTL, browse records), `use-collections` hook, `/collections` route + nav, `formatTtl`/`parseFields` lib (6 tests). Restored `getCollection` in the channel-script editor IntelliSense (`sandbox-types.ts`).
+- **docs** — `docs/user/scripting-api.md` (getCollection + order/report example), `docs/testing/66-collections.md`, `e2e/collections.spec.ts`.
+
+Engine 359, server 979, core-models 243, web tests green; full build + `pnpm lint --max-warnings 0` green.
+
+### 2026-07-13 (cont.) — Wire the remaining sound IO bridges
+
+Migration 0009 applied to the dev DB; full real-Postgres integration suite green (16 tests) incl. all 7 collection tests. Then wired the tractable IO bridges into `EngineManager` (they were sandbox-only):
+
+- **getResource** — `ResourceService.getByName(name)` (content-by-name, null if absent) + a one-line bridge closure; removed the ResourcesPage "not wired" banner; restored the `getResource` editor IntelliSense decl.
+- **httpFetch** — host closure over Node global `fetch` (method default, header/status/body mapping, per-request `AbortSignal.timeout`); SSRF host-blocking already enforced in the sandbox bridge layer.
+- **routeMessage** — cross-channel routing via existing `sendMessage`/`processMessage`, with a name→id resolver and a `MAX_ROUTE_DEPTH=25` hop-depth loop guard (`EngineManager.routeMessage`).
+- IntelliSense restored for all three; **`dbQuery` deliberately left unwired** — needs a driver registry, URL-keyed connection pooling, and a security model for script-supplied connection URLs (flagged in ROADMAP/scripting-api).
+- Tests: `resource.service` getByName (content/null), `engine-bridges.test.ts` (httpFetch mapping/forwarding + routeMessage happy/unknown/loop-guard). Server 984, engine 359, all green.
+
+## 2026-07-13 (cont.) — Data Sources: dbQuery wired (branch: feature/collections)
+
+Implemented the dbQuery bridge per D-178 / `docs/design/11-datasources.md` — the last IO bridge, so all five are now live. Also applied migration 0009 (collections) + 0010 (data_sources) and ran the real-Postgres integration suites.
+
+- **core-models** — `datasource.schema.ts` (create/update/test/query; read-only default, pool/row bounds), branded `DataSourceId`, `DATASOURCE_UPDATED` event (13 schema tests).
+- **server** — `data_sources` table (password stored as a content-crypto envelope) + migration 0010; `DataSourceService` (CRUD + `runQuery` + `testConnection`; encrypt-before-insert fail-loud without `CONTENT_ENCRYPTION_KEY`; password never returned); `DataSourcePoolManager` (one `ConnectionPool` per source keyed by id, read-only enforced via `SET TRANSACTION READ ONLY`, `maxRows` cap, invalidation on edit/delete, shutdown teardown wired into the graceful-shutdown sequence); routes + `datasources:*` RBAC (deployer RWD, developer/viewer R).
+- **engine/sandbox** — changed the `dbQuery` bridge signature from `(driver, connectionUrl, sql, params)` to `(dataSourceName, sql, params)`; wired `createDbQueryBridge()` into `EngineManager`; restored the `dbQuery` editor IntelliSense decl.
+- **web** — Data Sources page (connection form, Test Connection, read-only toggle, write-only password), `use-datasources` hook, `/datasources` route + nav.
+- **tests** — schema (13), pool-manager unit (7, mocked ConnectionPool), service unit (2, encryption guard + redaction), real-Postgres integration (6: encrypted round-trip, params, read-only-blocks-writes, read-write-allows, row cap, NOT_FOUND-after-delete); dbQuery sandbox bridge test updated to the new signature.
+- **docs** — `docs/design/11-datasources.md`, D-178, `scripting-api.md` (dbQuery + Data Sources section), `docs/testing/67-datasources.md`, `e2e/datasources.spec.ts`.
+
+All 5 unit suites green; both integration suites green (12 tests on `mirthless_test`); full build + `pnpm lint --max-warnings 0` clean.
+
+> Infra note: `drizzle-kit migrate`'s config loads `../../.env` via dotenv, which took precedence over a `DATABASE_URL` env override — migrations meant for `mirthless_test` hit the dev DB. To migrate a non-`.env` database, apply the migration SQL directly (or point `.env` at it). Applied 0010 to `mirthless_test` directly for the integration run.