fix(vector): provision per-field embedding storage via durable markers (least-privilege safe)

.changeset/marker-gated-vector-storage.md

@@ -0,0 +1,37 @@
+---
+"@nicia-ai/typegraph": minor
+---
+
+Vector storage now rides the #135 durable-contribution machinery, so the
+runtime never issues DDL on the embedding hot path.
+
+Previously every vector op (`upsertEmbedding` / `deleteEmbedding` /
+`vectorSearch` / `createVectorIndex`) lazily ran `CREATE TABLE IF NOT EXISTS`
+for its per-`(kind, field)` table on whatever connection it executed on. On a
+least-privilege Postgres role (USAGE on `public`, full DML, but no `CREATE`)
+this failed with `permission denied for schema public` (SQLSTATE 42501) — even
+when the table already existed, because Postgres runs the schema aclcheck before
+the `IF NOT EXISTS` short-circuit. The fulltext path already avoided this via
+durable markers; vectors now do too.
+
+What changed:
+
+- **Boot (privileged):** `createStoreWithSchema` provisions every embedding
+  `(kind, field)` table + a durable contribution marker, enumerated from the
+  graph. `evolve()` provisions any embedding fields it introduces.
+- **Runtime (DML-only):** the hot path asserts the durable marker with a cached
+  SELECT and runs DML — never DDL. `createVerifiedStore` verifies vector markers
+  at attach, alongside fulltext.
+- `reembedVectorField` re-stamps the marker after recreating storage at a new
+  dimension; vector-field reclaim (`materializeRemovals`) clears the marker when
+  it drops a table.
+
+**Breaking:** vector ops now require a prior privileged `createStoreWithSchema`
+(exactly as fulltext already does). A plain `createStore` + embedding write with
+no provisioning step throws `StoreNotInitializedError` instead of lazily
+creating the table.
+
+**Migration:** after upgrading, run `createStoreWithSchema(graph, adminBackend)`
+once under the schema-owner role. It creates the per-field vector tables +
+markers; least-privilege runtimes then assert markers (SELECT) and run vector
+DML with zero DDL — no `GRANT CREATE` required.

apps/docs/src/content/docs/architecture.md

@@ -442,9 +442,10 @@ backend.capabilities.vector; // { supported, metrics, indexTypes, maxDimensions,
 ### Storage
 
 Embeddings are stored in **per-field typed tables**, one per `(graphId, nodeKind, fieldPath)`, each carrying
-that field's fixed dimension. Tables are created lazily on first write and named
-`tg_vec_<graphId>_<kind>_<field>`. Graph-scoping the table name lets multiple graphs in one database declare the
-same kind+field at different dimensions without collision.
+that field's fixed dimension. Tables are provisioned by the privileged migrator (`createStoreWithSchema`, and
+`evolve()` for runtime-added fields), with a durable contribution marker; the runtime hot path asserts the
+marker and never issues DDL. They are named `tg_vec_<graphId>_<kind>_<field>`. Graph-scoping the table name lets
+multiple graphs in one database declare the same kind+field at different dimensions without collision.
 
 ```sql
 -- PostgreSQL with pgvector: one table per (graphId, kind, field). The kind
@@ -463,7 +464,7 @@ CREATE INDEX ON tg_vec_my_graph_document_embedding
 ```
 
 `generatePostgresMigrationSQL()` runs `CREATE EXTENSION IF NOT EXISTS vector` but creates no embedding table —
-the per-field tables are created at runtime on first write.
+the per-field tables are provisioned by `createStoreWithSchema` at boot (under the privileged role).
 
 ### Query Flow
 

apps/docs/src/content/docs/backend-setup.md

@@ -32,11 +32,12 @@ const { backend, db } = createLocalSqliteBackend({ path: "./app.db" });
 const store = createStore(graph, backend);
 ```
 
-:::caution[Fulltext requires `createStoreWithSchema`]
-`createLocalSqliteBackend` creates the tables but does not durably
-materialize fulltext storage. If your graph has `searchable()` fields,
-boot with `const [store] = await createStoreWithSchema(graph, backend);`
-instead of bare `createStore()` — otherwise the first fulltext operation
+:::caution[Fulltext and embeddings require `createStoreWithSchema`]
+`createLocalSqliteBackend` creates the base tables but does not durably
+materialize strategy-owned storage. If your graph has `searchable()` or
+`embedding()` fields, boot with
+`const [store] = await createStoreWithSchema(graph, backend);` instead of
+bare `createStore()` — otherwise the first fulltext or embedding operation
 throws `StoreNotInitializedError`.
 :::
 
@@ -99,7 +100,9 @@ const backend = createSqliteBackend(db, { vector: sqliteVecStrategy });
 ```
 
 sqlite-vec stores embeddings in `vec0` virtual tables and supports the `cosine` and `l2` metrics. Per-field
-vector tables are created lazily on first write — no embedding-table DDL is part of the migration.
+vector tables are provisioned by `createStoreWithSchema` at boot (not by the generated migration SQL), and the
+runtime asserts a durable marker rather than issuing DDL on first write — see
+[Database roles & least privilege](#database-roles--least-privilege).
 
 See [Semantic Search](/semantic-search) for query examples.
 
@@ -435,9 +438,9 @@ const db = drizzle(pool);
 const backend = createPostgresBackend(db);
 ```
 
-pgvector stores embeddings in per-field typed `vector(N)` tables (created lazily on first write — the migration
-creates no embedding table) with HNSW or IVFFlat indexes, and supports the `cosine`, `l2`, and `inner_product`
-metrics.
+pgvector stores embeddings in per-field typed `vector(N)` tables (provisioned by `createStoreWithSchema` at boot
+— the generated migration SQL creates no embedding table) with HNSW or IVFFlat indexes, and supports the
+`cosine`, `l2`, and `inner_product` metrics.
 
 See [Semantic Search](/semantic-search) for query examples.
 
@@ -807,19 +810,25 @@ least-privilege, DML-only database role.
 
 - **`createStoreWithSchema(graph, backend)` runs DDL.** It bootstraps the
   base tables on a fresh database, applies safe auto-migrations, and
-  durably materializes strategy-owned runtime storage (e.g. fulltext). It
-  re-issues idempotent DDL on every cold boot — at minimum a
-  `CREATE TABLE IF NOT EXISTS` for the contribution-marker table — so the
-  role it runs under **must hold `CREATE` / DDL privileges**. Run it once
-  at startup, outside request handlers and transactions.
+  durably materializes strategy-owned runtime storage — both fulltext and
+  each `embedding()` field's per-`(kind, field)` vector table, plus a
+  durable marker for each. It re-issues idempotent DDL on every cold boot
+  — at minimum a `CREATE TABLE IF NOT EXISTS` for the contribution-marker
+  table — so the role it runs under **must hold `CREATE` / DDL
+  privileges**. Run it once at startup, outside request handlers and
+  transactions. (`store.evolve()` likewise provisions any embedding field
+  it introduces, so it too needs DDL privileges.)
 
 - **`createStore(graph, backend)` is a synchronous, zero-I/O attach.**
   It does not create tables, repair DDL, or record that runtime storage
   is materialized — it issues **no DDL ever**. Use it only to attach to a
   database a prior `createStoreWithSchema` boot already initialized. A
-  fulltext read or write against a database that was never initialized
-  throws `StoreNotInitializedError` rather than silently emitting DDL on
-  the hot path. Graphs with no `searchable()` fields are unaffected.
+  fulltext or **embedding** read/write against a database that was never
+  initialized — a `create({ embedding })` write or a `store.search.vector`
+  / `.similarTo()` query — throws `StoreNotInitializedError` rather than
+  silently emitting `CREATE TABLE` on the hot path. This is what lets a
+  least-privilege role run vector ops: the table already exists. Graphs
+  with no `searchable()` or `embedding()` fields are unaffected.
 
 - **`createVerifiedStore(graph, backend)` is the same zero-DDL attach
   with a verification gate.** It reads the active schema row, folds the

apps/docs/src/content/docs/graph-extensions.md

@@ -587,8 +587,12 @@ const Manual = defineNode("Manual", {
 
 Embeddings live in per-`(graphId, nodeKind, fieldPath)` typed tables
 named `tg_vec_<graphId>_<kind>_<field>` (each carrying the field's
-fixed dimension), created lazily on first write — there is no single
-shared embeddings table.
+fixed dimension) — there is no single shared embeddings table. The
+privileged migrator provisions each table plus a durable marker: at boot
+via `createStoreWithSchema`, and for a field a runtime `evolve()`
+introduces, by that `evolve()` call. The runtime hot path then asserts
+the marker (never DDL), so a least-privilege role can read/write
+embeddings.
 
 On `materializeIndexes()`:
 

apps/docs/src/content/docs/schema-evolution.md

@@ -528,11 +528,13 @@ if (result.status === "migrated" && result.toVersion === 3) {
 
 ## Reclaiming Removed Embedding Storage
 
-Embeddings live in per-`(graphId, kind, field)` tables (`tg_vec_*`), created
-lazily on first write. When you remove an `embedding()` field from a **surviving**
+Embeddings live in per-`(graphId, kind, field)` tables (`tg_vec_*`), provisioned
+by the privileged migrator (`createStoreWithSchema`, or `evolve()` for a
+runtime-added field). When you remove an `embedding()` field from a **surviving**
 kind, the schema change commits fast but the field's now-orphaned vector table
-remains until you reconcile it. `store.materializeRemovals()` drops it (this is
-the same pass that cleans up storage for fully removed kinds):
+remains until you reconcile it. `store.materializeRemovals()` drops it — and
+clears its durable contribution marker so a later re-add re-provisions cleanly
+(this is the same pass that cleans up storage for fully removed kinds):
 
 ```typescript
 const result = await store.materializeRemovals();

apps/docs/src/content/docs/schema-management.md

@@ -272,9 +272,11 @@ const result = await ensureSchema(backend, graph, {
 ## Migrating Legacy Embedding Storage
 
 Embeddings now live in per-`(graphId, kind, field)` typed tables
-(`tg_vec_<graphId>_<kind>_<field>`), created lazily on first write. This
-replaces the single shared `typegraph_node_embeddings` table. New deployments
-need no action — the per-field tables materialize as you write vectors.
+(`tg_vec_<graphId>_<kind>_<field>`), provisioned by `createStoreWithSchema` (the
+privileged migrator) at boot. This replaces the single shared
+`typegraph_node_embeddings` table. New deployments need no action — the per-field
+tables are materialized by `createStoreWithSchema`, which the legacy migration
+below also relies on having run.
 
 Deployments that already hold rows in the legacy table run a one-time, idempotent
 cutover with `migrateLegacyEmbeddings()`, exported from the package root:

apps/docs/src/content/docs/semantic-search.md

@@ -149,8 +149,8 @@ import { generatePostgresMigrationSQL } from "@nicia-ai/typegraph/postgres";
 
 // Generates DDL including `CREATE EXTENSION IF NOT EXISTS vector;`.
 // It does NOT create a single embeddings table — each embedding field gets
-// its own typed `vector(N)` table, created lazily on first write (see
-// Storage Layout below).
+// its own typed `vector(N)` table, provisioned by `createStoreWithSchema`
+// (the privileged migrator) at boot (see Storage Layout below).
 const migrationSQL = generatePostgresMigrationSQL();
 ```
 
@@ -227,12 +227,17 @@ sqlite-vec feature set.
 
 Each embedding field is stored in its own typed, graph-scoped table named
 `tg_vec_<graphId>_<kind>_<field>`, carrying that field's fixed dimension
-(pgvector `vector(N)`, libSQL `F32_BLOB(N)`, sqlite-vec `vec0`). Tables are
-created lazily on the first write, so no migration step provisions them.
-Graph-scoping means several graphs in one database can declare the same
-`kind`+`field` at different dimensions without collision. This is transparent
-to queries — `.similarTo()`, `store.search.vector`, and `store.search.hybrid`
-read it for you.
+(pgvector `vector(N)`, libSQL `F32_BLOB(N)`, sqlite-vec `vec0`). The privileged
+migrator (`createStoreWithSchema`, and `evolve()` for runtime-added fields)
+provisions each table plus a durable contribution marker at boot; the runtime
+hot path then asserts the marker (a cached SELECT) and never issues DDL, so a
+least-privilege, DML-only role can read and write embeddings. A vector op
+against an un-provisioned slot throws `StoreNotInitializedError` rather than
+lazily creating the table — see [Database roles & least
+privilege](/backend-setup#database-roles--least-privilege). Graph-scoping means
+several graphs in one database can declare the same `kind`+`field` at different
+dimensions without collision. This is transparent to queries — `.similarTo()`,
+`store.search.vector`, and `store.search.hybrid` read it for you.
 
 ### Changing an embedding dimension
 

apps/docs/src/content/docs/troubleshooting.md

@@ -366,8 +366,12 @@ markers. See
 that never materializes runtime storage) against a database that no
 `createStoreWithSchema()` boot has initialized — commonly the runtime
 started before the privileged migration step ran, or the wrong role/
-database is configured. `createVerifiedStore()` catches this case at
-boot rather than at first hot-path operation.
+database is configured. This covers both fulltext and **embedding**
+operations: a `store.nodes.*.create({ embedding })` write or a
+`store.search.vector` / `.similarTo()` query against an un-provisioned
+per-`(kind, field)` table throws here rather than lazily issuing
+`CREATE TABLE` on the hot path. `createVerifiedStore()` catches this
+case at boot rather than at the first hot-path operation.
 
 **Solution:** Run `createStoreWithSchema(graph, adminBackend)` once
 under the privileged role before the runtime attaches (it writes the

packages/typegraph/examples/11-semantic-search.ts

@@ -20,7 +20,7 @@
 import { z } from "zod";
 
 import {
-  createStore,
+  createStoreWithSchema,
   defineGraph,
   defineNode,
   embedding,
@@ -183,7 +183,10 @@ export async function main() {
   console.log("\n=== Part 2: Storing Documents with Embeddings ===\n");
 
   const backend = createExampleBackend();
-  const store = createStore(graph, backend);
+  // createStoreWithSchema provisions each embedding field's per-(kind, field)
+  // vector table + durable marker under the (privileged) migrator role, so
+  // runtime embedding writes/searches never issue DDL. Run it once at boot.
+  const [store] = await createStoreWithSchema(graph, backend);
 
   // Sample documents to index
   const documents = [

packages/typegraph/examples/12-knowledge-graph-rag.ts

@@ -13,7 +13,7 @@
 import { z } from "zod";
 
 import {
-  createStore,
+  createStoreWithSchema,
   defineEdge,
   defineGraph,
   defineNode,
@@ -109,7 +109,10 @@ export async function main() {
   console.log("=== Knowledge Graph for RAG ===\n");
 
   const backend = createExampleBackend();
-  const store = createStore(graph, backend);
+  // createStoreWithSchema provisions each embedding field's per-(kind, field)
+  // vector table + durable marker under the (privileged) migrator role, so
+  // runtime embedding writes/searches never issue DDL. Run it once at boot.
+  const [store] = await createStoreWithSchema(graph, backend);
 
   // ----------------------------------------------------------
   // Seed data: Two documents about AI companies