Skip to content

fix(vector): provision per-field embedding storage via durable markers (least-privilege safe)#169

Open
pdlug wants to merge 1 commit into
mainfrom
fix/vector-least-privilege-marker-gate
Open

fix(vector): provision per-field embedding storage via durable markers (least-privilege safe)#169
pdlug wants to merge 1 commit into
mainfrom
fix/vector-least-privilege-marker-gate

Conversation

@pdlug

@pdlug pdlug commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Problem

On a least-privilege Postgres role (USAGE on public, full DML, but no CREATE), every runtime vector op failed with permission denied for schema public (SQLSTATE 42501). The cause: each vector op (upsertEmbedding / deleteEmbedding / vectorSearch / createVectorIndex) lazily ran CREATE TABLE IF NOT EXISTS for its per-(kind,field) table on the request connection — and Postgres runs the schema aclcheck before the IF NOT EXISTS short-circuit, so it failed even when the table already existed.

Fulltext already avoided this via the #135 durable-contribution machinery; the #161 vector cutover (v0.29) shipped an in-process-latch shortcut that bypassed it — exactly what the design doc said not to do.

Fix

Vectors now ride the same durable-marker machinery as fulltext:

  • Boot (privileged): createStoreWithSchema provisions every embedding (kind,field) table + a durable marker; evolve() provisions fields it introduces.
  • Runtime (DML-only): the hot path asserts the marker with a cached SELECT — never DDL. An un-provisioned slot throws StoreNotInitializedError (not a raw 42501). createVerifiedStore verifies vector markers at attach.
  • reembedVectorField re-stamps the marker after recreating storage at a new dimension; vector-field reclaim clears it when it drops a table.
  • The in-process vectorSlotLatch is removed; a shared resolveGraphVectorSlots enumerator is the single source of truth for boot materialization and runtime assertion.

Key files

  • backend/drizzle/contribution-materializations.ts — generalized the materializer to any contribution set; added ensureVectorSlot / assertVectorSlot / dropVectorSlot (+ force, deleteMarker).
  • backend/drizzle/{postgres,sqlite}.ts — marker-gated vector methods; shared materializer across tx-scoped backends.
  • store/store.ts, schema/manager.ts, store/materialize-removals.ts, backend/migrate-vectors.ts, core/embedding.ts, errors/index.ts.

⚠️ Breaking change (minor)

Vector ops now require a prior privileged createStoreWithSchema — exactly as fulltext already does. A plain createStore + embedding write with no provisioning step throws StoreNotInitializedError instead of lazily creating the table.

Migration: run createStoreWithSchema(graph, adminBackend) once under the schema-owner role after upgrading. It creates the per-field tables + markers; least-privilege runtimes then assert markers (SELECT) and run vector DML with zero DDL — no GRANT CREATE needed. Changeset included.

Tests

  • New postgres-vector-least-privilege.test.ts — the canonical regression: a USAGE-only role (no CREATE) runs vector upsert+search after the owner provisions, and gets StoreNotInitializedError (not 42501) when unprovisioned.
  • Extended the cross-backend materializer unit tests; updated lazy-usage tests/examples/fixtures to provision via createStoreWithSchema.
  • SQLite 3493 passed · Postgres + integration 914 passed · property 320 passed · typecheck + lint clean.

Docs updated (semantic-search, backend-setup, troubleshooting, architecture, graph-extensions, schema-evolution, schema-management) — the old "created lazily on first write" claims were the trap that led here.

Vector ops no longer issue DDL on the runtime hot path, so a
least-privilege (DML-only) Postgres role can read and write embeddings.

Previously every vector op lazily ran CREATE TABLE IF NOT EXISTS for its
per-(kind,field) table on the request connection, failing with
"permission denied for schema public" (42501) under a USAGE-only role —
even when the table existed, since Postgres runs the schema aclcheck
before the IF NOT EXISTS short-circuit. Vectors now ride the same #135
durable-contribution machinery as fulltext: the privileged migrator
(createStoreWithSchema, and evolve() for runtime-added fields) provisions
each table + durable marker at boot, and the runtime asserts the marker
with a cached SELECT — never DDL.

- contribution-materializations: generalized to any contribution set;
  ensureVectorSlot / assertVectorSlot / dropVectorSlot (+ force,
  deleteMarker); per-contribution-identity cache
- postgres/sqlite: marker-gated vector methods, shared materializer
  across tx-scoped backends; removed the in-process vectorSlotLatch
- store/manager: boot + evolve provision, createVerifiedStore asserts,
  reembedVectorField re-stamps the marker, reclaim clears it
- shared resolveGraphVectorSlots enumerator (one source of truth for
  boot materialization and runtime assertion)
- StoreNotInitializedError now names the actual storage (e.g.
  vector storage "Doc.embedding")

Adds a Postgres least-privilege regression test that reproduces the prod
failure and proves a USAGE-only role runs vector DML after the owner
provisions.

BREAKING CHANGE: vector ops now require a prior privileged
createStoreWithSchema (as fulltext already does); a plain createStore +
embedding write without provisioning throws StoreNotInitializedError.
Migration: run createStoreWithSchema(graph, adminBackend) once under the
schema-owner role after upgrading — no GRANT CREATE needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant