feat(w2): capability-label materialization, Redis cache, pgvector retrieval#22
Merged
Conversation
…rieval Closes #7, #8, #9. Delivers the W2 acceptance gate: LabelFilter(u) is a permissive superset of Authorized(u), verified across 5000 generated graphs by a new differential property test. core/labels.py Stable 31-bit label encoding via zlib+type-prefix. materialize_doc_labels is a doc-side forward walk over incident grant tuples. labels_for is a principal-side reverse walk that produces (L, B) — self-label + every container the principal is transitively a member of, plus opposite-side barrier tags. Two cache implementations behind a LabelCache Protocol: InMemoryCache for property tests, RedisCache for production. Asymmetric invalidation is DOCUMENTED HERE: grants must call invalidate() before returning (missed = silent recall loss); revocations can rely on TTL (Gate 2 catches). evals/differential.py test_label_filter_is_permissive_superset — the W2 acceptance property. For every principal in every generated graph, docs the oracle allows MUST also pass the label filter (acl_labels && L, minus barrier_tags & B). Reverse inclusion is allowed and expected; under-permission is the failure mode. Green across 5000 examples locally in ~2m25s (all 4 differential properties combined). sql/002_documents.sql documents table with pgvector embedding(768), int[] acl_labels and barrier_tags, LIST partitioning by org_id. Partitioning is defense-in- depth: a label-materialization bug cannot cross partition boundaries because the query never touches another partition's heap. retrieval/index.py ensure_org_partition (idempotent CREATE ... IF NOT EXISTS + per- partition GIN + HNSW), upsert_document. Partition creation uses SQL- literal interpolation of org_id (Postgres DDL doesn't accept bind params for FOR VALUES IN) — gated by an alphanumeric+_ regex. retrieval/strategies.py Three filtered-ANN strategies behind a Strategy enum: exact (flat scan on filtered set), iterative (pgvector 0.8 iterative_scan), partitioned (iterative scoped to one org partition). All three return the same Candidate shape so W4 benchmarks can compare them head-to-head on identical corpora. tests/test_labels.py 19 handwritten fixtures: label determinism, namespace disjointness, 31-bit fit for int4, doc materialization for direct/group/owner grants, L transitively contains group+org memberships, B contains opposite barrier side, cache stale-vs-invalidated semantics. tests/test_retrieval_pgvector.py 10 integration tests. All 3 strategies parametrized: seed 5 docs, retrieve top-3, exclude barrier-blocked docs, empty on no overlap, partitioned scopes to org. Skipped without WARDEN_TEST_DB_URL. tests/test_labels_redis.py 4 tests against real Redis: set/get roundtrip, miss returns None, invalidate removes entry, TTL applied (catches the "forgot ex=" bug that makes revocation lag unbounded). docker-compose.yml Switched postgres:16-alpine → pgvector/pgvector:pg16. Added redis:7- alpine service. .github/workflows/ci.yml Renamed postgres-integration → integration; now spins up both pgvector and redis service containers and runs all three integration test files. Fast job also includes tests/test_labels.py. Test state: 61 unit tests + 4 differential properties (5000 examples, in CI) + 14 integration tests (Postgres 3 + pgvector 10 + Redis 4). All green. Deferred to W4 per the milestone split: recall-vs-selectivity chart, head-to-head strategy benchmarks, full concurrent grant/revoke stress test. This PR ships the machinery; W4 ships the plotted numbers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Milestone W2 — capability-label materialization + pgvector retrieval + Redis-cached labels. Closes #7, #8, #9.
The W2 acceptance gate is the superset invariant:
LabelFilter(u) ⊇ Authorized(u). If it ever fails, Warden has a silent recall bug (an authorized doc that the pre-filter would never surface to Gate 2). Verified across 5,000 randomly generated authorization graphs per CI run.What ships
core/labels.py— stable 31-bit label encoding,materialize_doc_labels(doc-side forward walk),labels_for(principal-side reverse walk),LabelCacheProtocol withInMemoryCache+RedisCache. Asymmetric invalidation is documented at the top of the file and its correctness is exercised by the fixture tests: grants invalidate eagerly, revocations rely on TTL.sql/002_documents.sql—documentswithint[] acl_labels,int[] barrier_tags,vector(768) embedding, LIST-partitioned byorg_id. GIN + HNSW per partition.retrieval/index.py— idempotentensure_org_partition+upsert_document.retrieval/strategies.py— three filtered-ANN strategies (exact,iterative,partitioned) behind an enum. All three verified to return correct results; recall-vs-selectivity plotting is a W4 concern per the milestone split.test_label_filter_is_permissive_superset. This is the acceptance gate.Bug caught while wiring
Postgres partition DDL (
PARTITION OF documents FOR VALUES IN (%s)) doesn't accept bind parameters. Fixed by validatingorg_idagainst[A-Za-z0-9_]+and interpolating as a SQL literal. Injection surface is bounded by the same charset that the partition table name uses, so if it's safe for one it's safe for both.Test plan
make oracle— 15 tests green.make rebac— 18 tests green.make labels— 19 tests green.make test— 61 unit tests green, 21 integration skipped w/o env vars.make differential-gate(5000 examples, 4 properties) — green in ~2m25s locally.make integration(pgvector+Redis, real containers) — 21 tests green.Invariant checklist
make oraclegreen.make testgreen.core/oracle.pyandcore/rebac.py/core/labels.py.Security-sensitive?
core/labels.py(new), the pre-filter (Gate 1). It does not touchcore/algebra.py,core/oracle.py,core/rebac.py, or the gate wiring.The failure mode this PR could introduce is silent recall loss (under-permissive filter). The new superset property test is specifically designed to catch that class of bug, and it runs at 5000 examples in CI.
Notes for reviewers
core/labels.pyand the two fixture teststest_cache_returns_stale_after_write_without_invalidate(proves the bug is real) andtest_cache_invalidate_forces_recompute(proves the fix works)._incoming_grant_tuplesusesstore._outgoingbucket iteration forInMemoryStore. Storage-backed stores will override with a real reverse-index query in W3+. Same pattern asexpand()from W1 — kept minimal to avoid designing an API we don't need yet.tag(b, side) = b*2 + side) is reused fromcore/algebra.py. Users on both sides of the same barrier get both tags inB(u)— verified by fixture.