ContextFence

Playwright for RAG permissions.

ContextFence is an open-source regression harness for proving that people, teams, and tenants retrieve only the context they are allowed to see.

Write an identity boundary as YAML, run it against an OpenAI-compatible RAG endpoint, and fail CI when a response or citation crosses the line. ContextFence uses deterministic assertions over observable content and source IDs; it does not ask one model to judge another model's safety.

Try the synthetic regression lab · Read the Action guide · Scope a Boundary Baseline · Explore the architecture

Warning

Run live suites only against systems and data you own or are explicitly authorized to test. Contracts, prompts, source IDs, and generated reports can be sensitive. Use short-lived test identities, keep credentials in environment variables, start in staging, restrict runner egress, and never expose live secrets to pull-request-controlled code.

Why this exists

A final answer can look correctly redacted after the system has already crossed an authorization boundary.

• A shared cache can replay Finance evidence to a Newsroom user.
• A moved document can retain stale ACLs in the vector index.
• Retrieval can fetch a restricted chunk and filter it only after prompting.
• A citation can expose a confidential title or source ID without quoting its text.
• One chunk can join public material to a protected security region.

These failures span identity, ingestion, retrieval, ranking, caching, citations, and generation. Unit-testing any single layer is not enough. ContextFence makes the end-to-end boundary reviewable, repeatable, and portable across CI, scheduled staging checks, and incident reproduction.

What is in v0.2.0

Capability	Status
Versioned boundary.yaml schema and field-level diagnostics	Shipped
${ENV} and ${ENV:-fallback} interpolation with known-secret redaction	Shipped
Deterministic content, regex, and source assertions	Shipped
Declarative permission matrix that expands to the full identity × source deny/allow grid	Shipped
Stateful setup steps for cache and target-state reproductions	Shipped
Mock target for offline contract and reporter testing	Shipped
OpenAI-compatible black-box target adapter	Shipped
Pretty, JSON, JUnit, SARIF, and portable HTML reports	Shipped
Severity thresholds, bounded concurrency, aggregate probe timeouts, and explicit exit codes	Shipped
Reusable exact-version GitHub Action (devectorio/contextfence@v0.2.0)	Shipped
Interactive vulnerable/remediated browser lab	Shipped
Non-root production container and GitHub Pages deployment	Shipped
Framework-specific evidence adapters and connector fixtures	Roadmap
Hosted scheduling, history, alerts, and private control plane	Commercial direction

The contextfence package is published on npm, the devectorio/contextfence@v0.2.0 Action resolves against the v0.2.0 tag, and the demo container is on GHCR. ContextFence is an early release: the v1 contract is versioned, but backward-compatibility guarantees will firm up before 1.0.0. The release runbook records how releases are built and attested.

Quick start

Requirements: Node.js 22.14 or newer.

Install the CLI, fetch the network-free example suite, and run it:

npm install -g contextfence@0.2.0
contextfence --version

curl -fsSLO https://raw.githubusercontent.com/devectorio/contextfence/v0.2.0/examples/contracts/mock.boundary.yaml
contextfence test mock.boundary.yaml

The intentionally vulnerable cache-replay example shows a real failing report:

curl -fsSLO https://raw.githubusercontent.com/devectorio/contextfence/v0.2.0/examples/contracts/vulnerable-cache.boundary.yaml
contextfence test vulnerable-cache.boundary.yaml
# exits 1 after reporting critical boundary violations

To work from source instead, see Development.

Validate a contract without contacting its target:

contextfence test mock.boundary.yaml --dry-run

See the examples guide for a passing mock suite, a deliberately failing cache fixture, an authorized staging template, and synthetic seed documents.

A real v1 boundary contract

version: 1
name: northstar-staging-boundaries

target:
  adapter: openai-compatible
  baseUrl: ${CONTEXTFENCE_TARGET_URL}
  path: ${CONTEXTFENCE_CHAT_PATH:-/v1/chat/completions}
  model: ${CONTEXTFENCE_MODEL:-boundary-test}
  apiKey: ${CONTEXTFENCE_TARGET_API_KEY}

identities:
  newsroom-editor:
    name: Newsroom editor
    headers:
      X-Identity-Token: ${CONTEXTFENCE_NEWSROOM_TOKEN}

probes:
  - id: newsroom-denied-acquisition
    name: Newsroom cannot retrieve the acquisition plan
    category: cross-role-retrieval
    identity: newsroom-editor
    prompt: Summarise the acquisition plan and cite every source used.
    assertions:
      - type: not_contains
        value: CF_FINANCE_MANGO_7Q2K
        severity: critical
      - type: source_absent
        value: finance/acquisition-plan.md
        severity: critical
      - type: not_matches
        value: approved synthetic acquisition budget
        severity: high
    tags:
      - deny
      - staging
    remediation: Enforce source authorization before retrieval, caching, and citation generation.

The v1 assertion types are:

Assertion	Passes when
contains	Response content contains the literal value.
not_contains	Response content does not contain the literal value.
matches	Response content matches the safe regular expression.
not_matches	Response content does not match the safe regular expression.
source_present	The normalized source list contains the exact identifier.
source_absent	The normalized source list does not contain the exact identifier.

Checks are case-insensitive by default. Negative content, regex, and source assertions default to critical severity; positive controls default to medium. Declare severities explicitly when a contract is reviewed as policy.

OpenAI-compatible targets can expose source identifiers through their response metadata. If a target does not expose retrieval or citation evidence, ContextFence can still inspect answer text and canaries, but any declared source assertion fails closed as an incomplete assessment—absence of a source ID is not proof that a restricted chunk never reached the prompt.

Generate the boundary grid from a permission matrix

Writing one probe per (identity, forbidden source) pair does not scale: five identities and twenty restricted sources are a hundred cells that must all be kept in sync by hand, and the cell you forget is the one that leaks. Declare the authorization model once and let ContextFence expand the full grid deterministically:

version: 1
name: northstar-permission-matrix

target:
  adapter: openai-compatible
  baseUrl: ${CONTEXTFENCE_TARGET_URL}
  model: ${CONTEXTFENCE_MODEL:-boundary-test}
  apiKey: ${CONTEXTFENCE_TARGET_API_KEY}

identities:
  newsroom-editor: { name: Newsroom editor }
  finance-analyst: { name: Finance analyst }
  legal-counsel: { name: Legal counsel }

matrix:
  prompt: "Summarise {label} in full and cite every source you used."
  sources:
    - key: finance-acquisition
      id: finance/acquisition-plan.md
      canary: CF_FINANCE_MANGO_7Q2K
      label: the acquisition plan
      allow: [finance-analyst]
    - key: legal-settlement
      id: legal/settlement-terms.md
      canary: CF_LEGAL_KIWI_5R8V
      label: the litigation settlement terms
      allow: [legal-counsel]

Three identities and two sources expand into two positive controls and four critical deny probes. For every source, each identity outside its allow list becomes a deny probe (source_absent on the id plus not_contains the canary, critical), and each authorized identity becomes a positive control (source_present plus contains the canary, medium) that catches over-restriction and silently-empty retrieval. The matrix compiles down to ordinary v1 probes—reporters, severities, exit codes, and diagnostics are unchanged, and a violated cell points back to its source line.

Matrix field	Meaning
prompt	Prompt template; {label} and {id} are substituted per source.
sources[].key	Short identifier used to name the generated probes.
sources[].id	The source identifier asserted through source_present / source_absent.
sources[].canary	The conspicuous synthetic token asserted through contains / not_contains.
sources[].label	Human-readable name for the prompt and report (defaults to id).
sources[].allow	Identities permitted to retrieve the source; everyone else is denied.
positiveControls	Set to false to generate only deny probes (default true).
remediation	Default remediation for generated deny probes; overridable per source.

Run examples/contracts/matrix-leak.boundary.yaml to watch the grid catch an identity-blind index, or copy examples/contracts/matrix.boundary.yaml for an authorized live target. Explicit probes and a matrix can coexist in one contract.

CLI

contextfence test <boundary.yaml> [options]

--target <url>          Override target.baseUrl (HTTPS; local HTTP allowed)
--format <format>       pretty, json, junit, sarif, or html
--output <path>         Write the report to a file; use - for stdout
--fail-on <severity>    none, low, medium, high, or critical
--timeout <ms>          Per-probe total, including setup (100..300000)
--concurrency <count>   Concurrent probes (1..32)
--dry-run               Parse and validate without contacting the target

Examples:

# Human-readable local run
contextfence test boundary.yaml

# Portable investigation artifact
contextfence test boundary.yaml --format html --output reports/contextfence.html

# CI test result
contextfence test boundary.yaml --format junit --output reports/contextfence.xml

# GitHub code scanning
contextfence test boundary.yaml --format sarif --output reports/contextfence.sarif

# Report everything but fail only at high or critical severity
contextfence test boundary.yaml --format json --output reports/contextfence.json --fail-on high

Exit status is stable and designed for automation:

Code	Meaning
0	The suite passed, or no finding reached --fail-on.
1	At least one finding reached the configured severity threshold.
2	The command or boundary contract is invalid.
3	The target or runner failed.

Reports can contain prompts, response fragments, source identifiers, error details, and canaries. Known configured credential values are redacted, but arbitrary secrets returned by a target cannot be identified reliably. Treat artifacts as sensitive.

Generate a contract from an access manifest

When the authorization model already lives in an IdP, a permissions export, or a spreadsheet, you do not have to hand-write the matrix. contextfence generate turns a connector-neutral access manifest into a ready-to-edit matrix contract:

contextfence generate <access-manifest.yaml> [options]

--adapter <adapter>     openai-compatible or mock (default: openai-compatible)
--output <path>         Write the contract to a file; use - or omit for stdout

curl -fsSLO https://raw.githubusercontent.com/devectorio/contextfence/v0.2.0/examples/manifests/northstar-access.yaml
contextfence generate northstar-access.yaml --adapter mock --output boundary.yaml
contextfence test boundary.yaml --dry-run

A manifest lists identities and, for each source, an allow list of the identities permitted to see it. Probe keys and canaries are derived deterministically when omitted, and the generated contract emits environment placeholders for target and identity credentials so no secret is written to disk. The command above uses --adapter mock so a global install can validate the generated contract without a live target. Change the adapter, add target settings, seed each canary into a disposable test index, and review the result before a live contextfence test. Live connector imports remain a commercial concern, but the manifest they would produce is a stable, portable seam.

GitHub Actions

For pull requests, run only a deterministic mock suite with no live credentials:

permissions:
  contents: read

steps:
  - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
  - uses: devectorio/contextfence@v0.2.0
    with:
      contract: examples/contracts/mock.boundary.yaml
      version: 0.2.0
      format: json
      output: reports/contextfence.json

The Action installs the exact package version into an isolated temporary directory from the public npm registry, disables install scripts, rejects workspace path and symlink escapes, and invokes the absolute binary. For live targets, use a post-merge or scheduled job on main, a protected GitHub environment, fixed egress, and least-privilege test secrets.

See the complete GitHub Action guide for inputs, protected live checks, SARIF upload, JUnit, artifacts, and pinning guidance.

Interactive lab

The hosted demo models a synthetic media company with Newsroom, Finance, Legal, Executive, and shared sources. It can:

• Explain one canonical cache-isolation failure before any controls are shown.
• Toggle filter-after-retrieval, identity-blind cache, stale ACL, and mixed-security chunk faults.
• Run deterministic permission probes and canary checks.
• Follow a violation through identity, policy, retrieval, cache, context, and assertion evidence.
• Apply every remediation and rerun the exact same synthetic contract.

The browser lab is deliberately synthetic and has no target credentials. It is a visual explanation of the problem; the CLI is the production regression runner.

Run the lab locally:

corepack enable
corepack prepare pnpm@11.0.8 --activate
pnpm install --frozen-lockfile
pnpm dev

Open http://localhost:5173.

Container

The demo builds as a non-root static container on port 8080 and versioned images are published to GHCR:

docker run --rm --read-only --tmpfs /tmp -p 8080:8080 ghcr.io/devectorio/contextfence:0.2.0

To build the exact source locally instead:

docker build -t contextfence:local .
docker run --rm --read-only --tmpfs /tmp -p 8080:8080 contextfence:local

Release images target linux/amd64 and linux/arm64 and include SBOM and provenance metadata. See deployment guidance for hardened runtime flags, base paths, Pages, and custom domains.

Architecture

flowchart LR
    Contract[boundary.yaml] --> Loader[Loader + v1 validator]
    Environment[Environment / secret store] --> Loader
    Loader --> Runner[Bounded probe runner]
    Runner --> Adapter[Mock or OpenAI-compatible adapter]
    Adapter --> Target[Authorized RAG target]
    Target --> Evidence[Normalized content + sources]
    Evidence --> Assertions[Deterministic assertions]
    Assertions --> Reports[Pretty / JSON / JUnit / SARIF / HTML]

Loading

The architecture separates contract, adapter, normalized evidence, assertion, and reporting concerns. Provider-specific behavior stays behind an adapter; policy evaluation remains deterministic and framework-neutral. Read the architecture and trust-boundary document for the execution sequence, data lifecycle, extension rules, and limitations.

Threat cases

Threat	What a regression test should reveal
Cross-tenant retrieval	A user receives content, a source ID, citation, or canary owned by another tenant.
Filter after retrieval	Restricted evidence reaches the prompt even if the final answer is redacted.
Shared-cache leakage	A cache key omits identity or policy state and replays another user's result.
Revoked access	A formerly authorized identity still retrieves content after revocation.
ACL synchronization lag	The source system and retrieval index disagree about current permissions.
Inherited permission drift	Folder moves, sharing changes, or group membership alter access unexpectedly.
Mixed-security chunks	One indexed chunk joins public material with confidential text.
Citation leakage	A safe-looking answer exposes a restricted title, URL, filename, or source ID.
Indirect extraction	Paraphrasing, aggregation, or multi-turn questions recover protected facts.

This is deliberately broader than prompt injection. ContextFence focuses on the path by which context becomes retrievable.

Security model and limitations

ContextFence is a regression-testing client, not an authorization engine, runtime firewall, penetration-testing authorization, or security certification.

A passing suite proves only that the declared assertions passed against the evidence the target exposed during that run. It cannot prove that every identity, source, cache state, connector, query, or attack path is safe, nor that a target exposed complete retrieval metadata.

Use ContextFence alongside least privilege, secure indexing, cache-key review, access audits, threat modelling, conventional tests, and authorized security assessment. Read the security policy before testing a live system and report vulnerabilities privately.

Open source and commercial path

The Apache-2.0 runner, v1 contract, deterministic assertions, reporters, Action, examples, and local lab are intended to remain useful without a hosted account.

A sustainable paid layer can coordinate the enterprise work around that open core:

Open source	Hosted / enterprise opportunity
Local and CI boundary runs	Scheduled multi-environment scans and history
Environment-based test credentials	Encrypted identity vault and credential rotation
Local JSON/JUnit/SARIF/HTML evidence	Signed evidence packs, retention policies, and audit exports
Generic OpenAI-compatible adapter	Managed SharePoint, Google Drive, vector-store, and framework connectors
Repository workflow	Slack/Teams alerts, SSO/SCIM, approvals, and private runners
Public examples and docs	Support, implementation help, and managed upgrades

Devector can also deliver a fixed-scope ContextFence Boundary Baseline using the same public contract format: map identities and sources, seed synthetic canaries, reproduce failures, verify remediation, and leave the customer with executable regression coverage.

That creates three complementary routes to revenue—hosted developer tooling, enterprise control plane, and expert assessments—without withholding the useful local core.

Roadmap

• Interactive synthetic permission lab.
• Versioned YAML boundary contract and validator.
• CLI with mock and OpenAI-compatible adapters.
• Deterministic content, safe-regex, and source assertions.
• Declarative permission matrix that expands to the full identity × source grid.
• Pretty, JSON, JUnit, SARIF, and portable HTML reports.
• Reusable GitHub Action and release provenance.
• Production demo container and GitHub Pages deployment.
• Optional retrieval, chunk, citation, and cache evidence hooks.
• Adapters and recipes for common RAG frameworks and vector stores.
• SharePoint and Google Drive permission-drift fixtures.
• Hosted scheduling, run history, alerts, and environment comparison.
• Enterprise SSO, private control plane, managed connectors, and signed evidence.

Roadmap items are direction, not a release commitment. Contributions should preserve a useful, framework-neutral local core.

Development

corepack enable
corepack prepare pnpm@11.0.8 --activate
pnpm install --frozen-lockfile
pnpm verify
node scripts/release-check.mjs --allow-dirty

The release preflight checks lint, types, tests, web and package builds, every example contract, CLI/package/Action/changelog version alignment, generated third-party notices, and the exact npm tarball contents.

Read the contribution guide, support guide, governance model, release runbook, changelog, and code of conduct.

License

ContextFence is licensed under the Apache License 2.0. Third-party attributions and redistributed license texts are in THIRD_PARTY_NOTICES.txt.