fix(vector): provision per-field embedding storage via durable markers (least-privilege safe)#169
Open
pdlug wants to merge 1 commit into
Open
fix(vector): provision per-field embedding storage via durable markers (least-privilege safe)#169pdlug wants to merge 1 commit into
pdlug wants to merge 1 commit into
Conversation
Vector ops no longer issue DDL on the runtime hot path, so a least-privilege (DML-only) Postgres role can read and write embeddings. Previously every vector op lazily ran CREATE TABLE IF NOT EXISTS for its per-(kind,field) table on the request connection, failing with "permission denied for schema public" (42501) under a USAGE-only role — even when the table existed, since Postgres runs the schema aclcheck before the IF NOT EXISTS short-circuit. Vectors now ride the same #135 durable-contribution machinery as fulltext: the privileged migrator (createStoreWithSchema, and evolve() for runtime-added fields) provisions each table + durable marker at boot, and the runtime asserts the marker with a cached SELECT — never DDL. - contribution-materializations: generalized to any contribution set; ensureVectorSlot / assertVectorSlot / dropVectorSlot (+ force, deleteMarker); per-contribution-identity cache - postgres/sqlite: marker-gated vector methods, shared materializer across tx-scoped backends; removed the in-process vectorSlotLatch - store/manager: boot + evolve provision, createVerifiedStore asserts, reembedVectorField re-stamps the marker, reclaim clears it - shared resolveGraphVectorSlots enumerator (one source of truth for boot materialization and runtime assertion) - StoreNotInitializedError now names the actual storage (e.g. vector storage "Doc.embedding") Adds a Postgres least-privilege regression test that reproduces the prod failure and proves a USAGE-only role runs vector DML after the owner provisions. BREAKING CHANGE: vector ops now require a prior privileged createStoreWithSchema (as fulltext already does); a plain createStore + embedding write without provisioning throws StoreNotInitializedError. Migration: run createStoreWithSchema(graph, adminBackend) once under the schema-owner role after upgrading — no GRANT CREATE needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a least-privilege Postgres role (USAGE on
public, full DML, but noCREATE), every runtime vector op failed withpermission denied for schema public(SQLSTATE 42501). The cause: each vector op (upsertEmbedding/deleteEmbedding/vectorSearch/createVectorIndex) lazily ranCREATE TABLE IF NOT EXISTSfor its per-(kind,field)table on the request connection — and Postgres runs the schema aclcheck before theIF NOT EXISTSshort-circuit, so it failed even when the table already existed.Fulltext already avoided this via the #135 durable-contribution machinery; the #161 vector cutover (v0.29) shipped an in-process-latch shortcut that bypassed it — exactly what the design doc said not to do.
Fix
Vectors now ride the same durable-marker machinery as fulltext:
createStoreWithSchemaprovisions every embedding(kind,field)table + a durable marker;evolve()provisions fields it introduces.StoreNotInitializedError(not a raw 42501).createVerifiedStoreverifies vector markers at attach.reembedVectorFieldre-stamps the marker after recreating storage at a new dimension; vector-field reclaim clears it when it drops a table.vectorSlotLatchis removed; a sharedresolveGraphVectorSlotsenumerator is the single source of truth for boot materialization and runtime assertion.Key files
backend/drizzle/contribution-materializations.ts— generalized the materializer to any contribution set; addedensureVectorSlot/assertVectorSlot/dropVectorSlot(+force,deleteMarker).backend/drizzle/{postgres,sqlite}.ts— marker-gated vector methods; shared materializer across tx-scoped backends.store/store.ts,schema/manager.ts,store/materialize-removals.ts,backend/migrate-vectors.ts,core/embedding.ts,errors/index.ts.Vector ops now require a prior privileged
createStoreWithSchema— exactly as fulltext already does. A plaincreateStore+ embedding write with no provisioning step throwsStoreNotInitializedErrorinstead of lazily creating the table.Migration: run
createStoreWithSchema(graph, adminBackend)once under the schema-owner role after upgrading. It creates the per-field tables + markers; least-privilege runtimes then assert markers (SELECT) and run vector DML with zero DDL — noGRANT CREATEneeded. Changeset included.Tests
postgres-vector-least-privilege.test.ts— the canonical regression: a USAGE-only role (noCREATE) runs vector upsert+search after the owner provisions, and getsStoreNotInitializedError(not 42501) when unprovisioned.createStoreWithSchema.Docs updated (semantic-search, backend-setup, troubleshooting, architecture, graph-extensions, schema-evolution, schema-management) — the old "created lazily on first write" claims were the trap that led here.