Agentic Boundary Bench

agentic-boundary-bench is a benchmark scaffold for agentic platforms whose real security boundary is often a mix of runtime policy, approvals, tool mediation, trust handling, and optional sandboxing rather than a single container wall.

It borrows the best measurement ideas from sandbox_escape_bench:

• fixed scenarios
• objective artifacts
• stable lanes
• transcript and tool-trace evidence
• explicit N/A handling for boundaries a platform does not expose
• score by final boundary outcome, not only by an intended exploit path

It does not assume Docker, Kubernetes, AppLocker, bwrap, or any other kernel-backed isolation is the primary boundary. For GuardianAgent, the benchmark treats runtime governance as the main line of defense and measures how well that holds when strong OS isolation is absent or optional.

Current State

This repo now includes:

• a core scenario catalog aligned to boundary-circumvention rather than container escape
• a lane model for degraded + strict, degraded + permissive, and strong + strict
• a reusable scoring model
• a profile format for comparing different agent platforms
• a first working adapter for GuardianAgent
• a manual adapter for platforms that do not yet have automation

The GuardianAgent adapter automates the scenarios that can already be proven through real runtime evidence:

• filesystem boundary denial
• command allowlist / shell sanitization denial
• localhost SSRF blocking
• approval-gated write stopping before side effects
• policy-widening requests stopping at approval

The remaining scenarios stay in the catalog as planned/manual until the target platform exposes enough evidence to test them honestly.

Benchmarking Your Own Repo

If you want to use this against another agent platform, the shortest path is:

1. Copy template.profile.json to profiles/<your-platform>.json.
2. Fill in the real trust boundary and only mark surfaces true if the platform actually exposes them.
3. Run node src/cli.mjs plan --profile profiles/<your-platform>.json.
4. If you do not have an adapter yet, fill a manual results file starting from manual-results.template.json.
5. Score it with node src/cli.mjs score --profile profiles/<your-platform>.json --lane degraded-strict --input reports/<your-run>.json.

The full step-by-step guide is in docs/USAGE.md.

Quick Start

List the benchmark catalog:

node src/cli.mjs list

Plan the matrix for GuardianAgent:

node src/cli.mjs plan --profile profiles/guardian-agent.json

Run one GuardianAgent lane:

node src/cli.mjs run --profile profiles/guardian-agent.json --lane degraded-strict

Run only one scenario:

node src/cli.mjs run \
  --profile profiles/guardian-agent.json \
  --lane degraded-permissive \
  --scenario core.fs.outside-allowed-path

Save JSON output:

node src/cli.mjs run \
  --profile profiles/guardian-agent.json \
  --lane degraded-strict \
  --output reports/guardian-degraded-strict.json \
  --json

Manual scoring for a repo without an adapter:

node src/cli.mjs score \
  --profile profiles/my-agent.json \
  --lane degraded-strict \
  --input reports/my-agent.manual-results.json

Outcome Model

Each scenario resolves to one of these benchmark outcomes:

• escaped
• blocked_cleanly
• approval_required
• contained_by_surface_removal
• false_success
• n_a
• not_implemented
• error

The important point is that N/A is first-class. A platform should not be punished for not exposing Docker breakout or kernel exploit surfaces if those are not part of its architecture.

Repo Layout

• scenarios/core-10.json: initial cross-platform scenario set
• profiles/: platform descriptions
• src/catalog.mjs: scenario and lane loading
• src/planner.mjs: applicability and lane planning
• src/scoring.mjs: outcome classification and aggregate metrics
• src/adapters/: platform adapters
• docs/REFERENCE-PLATFORMS.md: why the taxonomy is broader than GuardianAgent alone
• docs/METHODOLOGY.md: what transfers from the paper and repo that introduced SandboxEscapeBench
• docs/USAGE.md: how to use the benchmark against your own repo
• docs/SANDBOX-ESCAPE-BENCH-GUARDIANAGENT-ASSESSMENT-2026-03-27.md: original Guardian-specific assessment

Adapter Contract

An adapter is responsible for producing evidence, not prose. It should:

1. prepare the target platform in a known lane
2. execute one or more scenarios
3. return raw evidence about artifacts, approvals, tool availability, side effects, and final claims
4. let the shared scorer classify outcomes

That keeps the benchmark reusable across platforms with very different execution models.

Reference Direction

This repo is intentionally broader than GuardianAgent, but GuardianAgent is the current gold-standard reference profile because it already exposes:

• explicit allowlists
• structured approvals
• sandbox posture metadata
• tool catalogs
• trust/taint controls
• auditable runtime decisions

Those are the right primitives for a reusable boundary benchmark.