Script IO bridges: Collections + wire all five bridges (dbQuery via Data Sources) - #34
Merged
Conversation
New feature — durable, queryable, TTL-pruned keyed record store for channel scripts (getCollection bridge). Motivated by prod order/report matching. - docs/design/10-collections.md — full design - DECISIONS.md D-177 — append-only, user-definable indexed fields, single JSONB table, collection-default TTL w/ per-write override, separate from Resources - core-models: collection.schema.ts (create/update/store/find + params), branded CollectionId, 22 schema tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
- db schema: collections + collection_records (JSONB fields, GIN + newest-wins + partial-expiry indexes, FK cascade); migration 0009 (dropped the redundant response_transformer ALTER already shipped by hand-written 0008) - CollectionService: define/list/get/update/delete + store/find (parameterized @> match + multi-field ->> filter, newest-wins, limit) + listRecords + pruneExpired; field-value string coercion for GIN consistency; unknown-field and 1 MiB payload guards; COLLECTION_UPDATED audit events - controller + routes (collections:read/write/delete), registered at /collections - RBAC: collections:* permissions seeded; deployer=RWD, developer=RW, viewer=R - core-models: COLLECTION_UPDATED event name - integration test (collection.itest.ts): order/report newest-wins match, multi-filter, default+override TTL + prune, unknown-field rejection, cascade Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
- sandbox: getCollection(name) → { store, find } bridge, mirroring the IO-bridge
dispatch pattern; new `collections` BridgeDependency; payload flag getCollection
- fix: hasAsyncBridges omitted `collections`, so a script using only getCollection
was wrapped in a sync fn and `await` threw — now included
- engine.ts: wire createCollectionBridge() into VmSandboxExecutor (was constructed
with no deps — the first IO bridge to go live); Zod-validate store/find inputs at
the script boundary; map service records to ISO-string dates
- tests: getCollection store/find/error + not-exposed-without-deps (engine 359 pass)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
- use-collections hook: list/records/create/update/delete (TanStack Query) - CollectionsPage: table (name, description, indexed-field chips, TTL), create/ edit dialog (name, description, comma-separated indexed fields, default TTL), read-only record-browser dialog; RBAC-gated via collections:* permissions - lib/collections.ts: formatTtl + parseFields (pure, unit-tested, 6 tests) - route /collections + sidebar nav item; web PERMISSION.COLLECTIONS_* Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
…ress - sandbox-types.ts: restore getCollection in the channel-script editor IntelliSense (CollectionHandle store/find + CollectionRecord); note the other IO bridges remain unwired - docs/user/scripting-api.md: getCollection section + order/report example - docs/testing/66-collections.md; e2e/collections.spec.ts - progress: CHANGELOG (2026-07-13), ROADMAP (getCollection = first live IO bridge), DECISIONS D-177 (added in 1/5) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
The engine constructed the sandbox with only the collections dep; wire the other tractable IO bridges (all were sandbox-only): - getResource — ResourceService.getByName (content-by-name, null if absent) + bridge closure; removed the ResourcesPage "not wired" banner - httpFetch — host closure over global fetch (method default, header/status/body mapping, per-request AbortSignal.timeout); SSRF blocking already in the sandbox - routeMessage — cross-channel routing via sendMessage/processMessage with a name→id resolver and MAX_ROUTE_DEPTH=25 hop-depth loop guard - restored getResource/httpFetch/routeMessage editor IntelliSense; dbQuery left unwired (needs driver registry + URL-keyed pooling + script-URL security model) - tests: resource.service getByName (content/null); engine-bridges.test.ts (httpFetch mapping/forwarding + routeMessage happy/unknown/loop-guard) Migration 0009 applied to dev DB; full integration suite green (16). Server 986, engine 359; full build + lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
…178) Reframes the unwired dbQuery(driver, connectionUrl, sql, params) bridge to dbQuery(dataSourceName, sql, params) backed by admin-managed Data Sources: - credentials server-side & encrypted at rest (content-crypto); never in scripts - reachable DBs are an allowlist by construction (no SSRF-style risk) - read-only by default (DB role + read-only txn), opt-in read-write per source - Postgres-only v1 (reuse ConnectionPool); DbDriver interface for later - DataSourcePoolManager keyed by source id; statement timeout + row cap - phased build plan + open follow-ups Design only — no code yet. User chose named-data-sources + read-only-default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
- datasource.schema.ts: create/update (password, read-only default, pool/row bounds) + dbQuery input + uuid param; DB_DRIVER (postgres v1) - branded DataSourceId; DATASOURCE_UPDATED event; 13 schema tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
Implements dbQuery per D-178 — the last IO bridge (all five now live). Server: - data_sources table (password as content-crypto envelope) + migration 0010 - DataSourceService: CRUD + runQuery + testConnection; encrypt-before-insert (fail loud without CONTENT_ENCRYPTION_KEY); password never returned - DataSourcePoolManager: one ConnectionPool per source keyed by id; read-only enforced via SET TRANSACTION READ ONLY; maxRows cap; invalidate on edit/delete; shutdown teardown wired into the graceful-shutdown sequence - routes + datasources:* RBAC (deployer RWD, developer/viewer R) Engine/sandbox: - dbQuery signature (driver, connectionUrl, sql, params) → (dataSourceName, sql, params); createDbQueryBridge() wired into EngineManager; IntelliSense restored Web: Data Sources page (form, Test Connection, read-only toggle, write-only password), use-datasources hook, /datasources route + nav. Tests: schema (13), pool-manager (7, mocked pool), service (2, encryption guard + redaction), real-Postgres integration (6), updated dbQuery sandbox test. Full build + lint clean; unit + both integration suites green. Docs: scripting-api (dbQuery + Data Sources), 67-datasources testing checklist, e2e/datasources.spec, CHANGELOG/ROADMAP. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
There was a problem hiding this comment.
Pull request overview
This PR ships the “script IO bridges” program end-to-end by introducing Collections and Data Sources, and wiring all sandbox IO bridges (getCollection, getResource, httpFetch, routeMessage, dbQuery) into the production engine so channel scripts can use them at runtime.
Changes:
- Adds Collections (schema/migrations/service/routes/web UI + helper lib/tests) and exposes them to scripts via
getCollection(name). - Adds Data Sources (schema/migrations/service/pool manager/routes/web UI + tests) and updates
dbQuerytodbQuery(dataSourceName, sql, params?)with encrypted-at-rest credentials. - Wires all IO bridges into
EngineManagerand updates sandbox runtime/types/docs/tests accordingly.
Reviewed changes
Copilot reviewed 58 out of 61 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/web/src/pages/ResourcesPage.tsx | Removes “not wired” banner now that getResource() is live. |
| packages/web/src/pages/DataSourcesPage.tsx | New admin UI for managing/test-connecting Data Sources. |
| packages/web/src/pages/CollectionsPage.tsx | New admin UI for managing Collections and browsing records. |
| packages/web/src/lib/sandbox-types.ts | Restores/extends scripting IntelliSense for IO bridges + collection types. |
| packages/web/src/lib/permissions.ts | Adds collections:* and datasources:* permissions constants. |
| packages/web/src/lib/collections.ts | Adds pure UI helpers (formatTtl, parseFields). |
| packages/web/src/lib/tests/collections.test.ts | Unit tests for formatTtl and parseFields. |
| packages/web/src/hooks/use-datasources.ts | TanStack Query hooks for Data Source CRUD + test endpoint. |
| packages/web/src/hooks/use-collections.ts | TanStack Query hooks for Collection CRUD + record browsing. |
| packages/web/src/components/layout/AppLayout.tsx | Adds sidebar navigation entries for Collections and Data Sources. |
| packages/web/src/App.tsx | Adds routes for /collections and /datasources. |
| packages/server/test/integration/data-source.itest.ts | Real Postgres integration tests for Data Sources + dbQuery path. |
| packages/server/test/integration/collection.itest.ts | Real Postgres integration tests for Collections storage/query/TTL/prune. |
| packages/server/test/integration/_setup.ts | Exposes Collection/DataSource services + pool manager to integration harness. |
| packages/server/src/services/resource.service.ts | Adds getByName for getResource bridge. |
| packages/server/src/services/data-source.service.ts | Adds Data Source CRUD, encryption guard, runQuery, and testConnection. |
| packages/server/src/services/data-source-pool-manager.ts | New per-data-source pool lifecycle, read-only enforcement, row caps, teardown. |
| packages/server/src/services/collection.service.ts | Adds Collections CRUD + store/find/listRecords/pruneExpired. |
| packages/server/src/services/tests/resource.service.test.ts | Unit tests for ResourceService.getByName. |
| packages/server/src/services/tests/data-source.service.test.ts | Unit tests for “fail loud without encryption key” + password redaction. |
| packages/server/src/services/tests/data-source-pool-manager.test.ts | Unit tests for pool reuse/rebuild, row cap, read-only transaction enforcement. |
| packages/server/src/server.ts | Wires DataSourcePoolManager shutdown into graceful shutdown. |
| packages/server/src/routes/index.ts | Registers /collections and /datasources routes. |
| packages/server/src/routes/data-source.routes.ts | New CRUD/test routes with RBAC and validation. |
| packages/server/src/routes/collection.routes.ts | New CRUD/records routes with RBAC and validation. |
| packages/server/src/lib/shutdown.ts | Adds shutdown step for data source pools. |
| packages/server/src/lib/tests/shutdown.test.ts | Updates shutdown order test for new pool shutdown step. |
| packages/server/src/engine.ts | Injects all IO bridges into VmSandboxExecutor; adds route loop guard; adds collection/dbQuery wiring. |
| packages/server/src/db/seeds/roles.ts | Adds new permissions to default roles. |
| packages/server/src/db/seeds/permissions.ts | Adds new permission definitions for collections/datasources. |
| packages/server/src/db/schema/index.ts | Exports new collections + data_sources schemas. |
| packages/server/src/db/schema/data-sources.ts | New data_sources table schema (encrypted password). |
| packages/server/src/db/schema/collections.ts | New collections and collection_records schemas + indexes. |
| packages/server/src/db/migrations/meta/_journal.json | Registers migrations 0009 and 0010. |
| packages/server/src/db/migrations/0009_spicy_killer_shrike.sql | Migration for collections tables + indexes. |
| packages/server/src/db/migrations/0010_ambitious_wild_pack.sql | Migration for data_sources table. |
| packages/server/src/controllers/data-source.controller.ts | New controller for Data Sources endpoints. |
| packages/server/src/controllers/collection.controller.ts | New controller for Collections endpoints. |
| packages/server/src/tests/engine-bridges.test.ts | Adds host-side bridge wiring tests for httpFetch + routeMessage. |
| packages/engine/src/sandbox/sandbox-executor.ts | Updates dbQuery signature; adds getCollection bridge plumbing + async bridge detection. |
| packages/engine/src/sandbox/bridge-functions.ts | Updates bridge dependency types; adds collections bridge interfaces. |
| packages/engine/src/sandbox/tests/bridge-io-functions.test.ts | Updates dbQuery tests and adds getCollection tests. |
| packages/core-models/src/schemas/index.ts | Exports new collection/datasource schemas. |
| packages/core-models/src/schemas/event.schema.ts | Adds COLLECTION_UPDATED and DATASOURCE_UPDATED event names. |
| packages/core-models/src/schemas/datasource.schema.ts | Adds Zod schemas/types for Data Sources + dbQuery inputs. |
| packages/core-models/src/schemas/collection.schema.ts | Adds Zod schemas/types for Collections + store/find inputs. |
| packages/core-models/src/schemas/tests/datasource.schema.test.ts | Adds schema tests for data sources + dbQuery input. |
| packages/core-models/src/schemas/tests/collection.schema.test.ts | Adds schema tests for collections/store/find. |
| packages/core-models/src/brands.ts | Adds branded IDs for CollectionId and DataSourceId. |
| e2e/datasources.spec.ts | Playwright coverage for Data Sources UI flows. |
| e2e/collections.spec.ts | Playwright coverage for Collections UI flows. |
| docs/user/scripting-api.md | Updates scripting API docs for new/rewired bridges + examples. |
| docs/testing/66-collections.md | Manual testing checklist for Collections. |
| docs/testing/67-datasources.md | Manual testing checklist for Data Sources/dbQuery. |
| docs/progress/ROADMAP.md | Marks IO bridge work as implemented/wired. |
| docs/progress/DECISIONS.md | Adds D-177 (Collections) and D-178 (Data Sources/dbQuery). |
| docs/progress/CHANGELOG.md | Adds changelog entries for Collections + wired IO bridges + Data Sources. |
| docs/design/10-collections.md | Design doc for Collections. |
| docs/design/11-datasources.md | Design doc for Data Sources and dbQuery reframe. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+336
to
+379
| /** Current routeMessage nesting depth (one process chain); guards against routing loops. */ | ||
| private routeDepth = 0; | ||
|
|
||
| constructor(serverId?: string) { | ||
| this.sandbox = new VmSandboxExecutor(); | ||
| this.sandbox = new VmSandboxExecutor({ | ||
| collections: createCollectionBridge(), | ||
| getResource: createResourceBridge(), | ||
| httpFetch: createHttpFetchBridge(), | ||
| dbQuery: createDbQueryBridge(), | ||
| routeMessage: this.createRouteMessageBridge(), | ||
| }); | ||
| this.serverId = serverId ?? 'server-01'; | ||
| } | ||
|
|
||
| /** Resolve a deployed channel's id by its name (in-memory; deployed channels only). */ | ||
| private resolveChannelIdByName(name: string): string | undefined { | ||
| for (const [id, deployed] of this.runtimes) { | ||
| if (deployed.config.name === name) return id; | ||
| } | ||
| return undefined; | ||
| } | ||
|
|
||
| /** | ||
| * Route a raw message into another deployed, STARTED channel by name. Enforces a | ||
| * hop-depth cap so a routing cycle (A→B→A) fails loud instead of recursing until | ||
| * timeout on every hop. | ||
| */ | ||
| async routeMessage(channelName: string, rawData: string): Promise<Result<{ messageId: number }>> { | ||
| return tryCatch(async () => { | ||
| if (this.routeDepth >= MAX_ROUTE_DEPTH) { | ||
| throw new Error(`routeMessage exceeded max hop depth (${String(MAX_ROUTE_DEPTH)}) — possible routing loop`); | ||
| } | ||
| const targetId = this.resolveChannelIdByName(channelName); | ||
| if (!targetId) { | ||
| throw new Error(`routeMessage: no deployed channel named "${channelName}"`); | ||
| } | ||
| this.routeDepth++; | ||
| try { | ||
| const result = await this.sendMessage(targetId, rawData); | ||
| if (!result.ok) throw new Error(result.error.message); | ||
| return result.value; | ||
| } finally { | ||
| this.routeDepth--; | ||
| } |
Comment on lines
+299
to
+306
| /** dbQuery() script bridge: parameterized query against a named Data Source (read-only by default). */ | ||
| function createDbQueryBridge(): NonNullable<BridgeDependencies['dbQuery']> { | ||
| return async (dataSourceName, sql, params) => { | ||
| const result = await DataSourceService.runQuery(dataSourceName, sql, params); | ||
| if (!result.ok) throw new Error(result.error.message); | ||
| return result.value; | ||
| }; | ||
| } |
Comment on lines
+316
to
+322
| globalThis.getCollection = function (name) { | ||
| return { | ||
| store: async function (fields, payload, options) { | ||
| var r = JSON.parse(await dispatch('collectionStore', name, fields || {}, String(payload == null ? '' : payload), options || {})); | ||
| if (!r.ok) { throw new Error(r.e); } | ||
| return r.v; | ||
| }, |
Comment on lines
+239
to
+247
| static async pruneExpired(): Promise<Result<number>> { | ||
| return tryCatch(async () => { | ||
| const deleted = await db | ||
| .delete(collectionRecords) | ||
| .where(lt(collectionRecords.expireAt, new Date())) | ||
| .returning({ id: collectionRecords.id }); | ||
| return deleted.length; | ||
| }); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ships the script IO-bridge program: Collections + wiring all five sandbox IO bridges into the production engine, culminating in
dbQueryvia named Data Sources. Before this, the engine constructed the sandbox with no dependencies, so none of the IO bridges were live despite being implemented in the sandbox layer.Collections — durable keyed record store
A new
getCollection(name)bridge for the order/report matching pattern (stash records now, look up newest-by-key-and-filter later). Append-only, user-definable indexed fields, single JSONB table (GIN + newest-wins + partial-expiry indexes), per-collection default TTL with per-write override (pruned by the existing data-pruner). Full stack: core-models schema,collections/collection_recordstables + migration 0009,CollectionService, routes +collections:*RBAC, sandbox bridge, web Collections page. Design:docs/design/10-collections.md(D-177).IO bridges wired into the engine
getCollection,getResource,httpFetch, androuteMessagewired intoEngineManager:getResource—ResourceService.getByName; removed the ResourcesPage "not wired" banner.httpFetch— host closure over globalfetch(SSRF blocking already in the sandbox layer).routeMessage— cross-channel routing with aMAX_ROUTE_DEPTH=25hop-depth loop guard.hasAsyncBridgesomittingcollections(a script using onlygetCollectionwas wrapped sync andawaitthrew).dbQuery — named Data Sources (the last bridge)
Reframed the unsafe
dbQuery(driver, connectionUrl, sql, params)(creds-in-scripts, arbitrary host) todbQuery(dataSourceName, sql, params)backed by admin-managed Data Sources. Credentials live server-side, encrypted at rest (content-crypto); scripts can only reach configured sources (allowlist by construction). Read-only by default (enforced viaSET TRANSACTION READ ONLY), per-source statement timeout +maxRowscap,DataSourcePoolManagerkeyed by source id with invalidation + graceful-shutdown teardown. Postgres-only v1 behind aDbDriverinterface. Full stack + web Data Sources page (with Test Connection). Design:docs/design/11-datasources.md(D-178).Testing
*.itest.ts): 22 tests across 6 suites, including new collection (7) and data-source (6) suites covering jsonb match/filter, newest-wins, TTL prune, encrypted-credential round-trip, read-only enforcement, row cap, and pool invalidation.pnpm build+pnpm lint --max-warnings 0clean.Docs
docs/design/10-collections.md,docs/design/11-datasources.md, DECISIONS D-177/D-178,docs/user/scripting-api.md, testing checklists66-collections.md/67-datasources.md, e2e specs, ROADMAP/CHANGELOG.Follow-ups (parked)
Additional
dbQuerydrivers (MySQL/MSSQL/Oracle behindDbDriver); backport encrypted-at-rest creds to the Database connector config.🤖 Generated with Claude Code
https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi