Skip to content

Commit a6fa8a8

Browse files
authored
Merge pull request #19 from swarmproof/feat/registry-runtime-sandbox
feat: runtime sandbox for untrusted registry handlers (0.3.0)
2 parents e65e684 + 06284f0 commit a6fa8a8

11 files changed

Lines changed: 600 additions & 26 deletions

File tree

‎AGENTS.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# AGENTS.md
2+
3+
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
4+
5+
## What this repo is
6+
7+
**mockworld** — "a synthetic internet for agents": deterministic, LLM-free fake services (fake Stripe, Gmail, exchange, CRM, S3) exposed as MCP servers so agents can be built and tested without touching production. Part of the Swarm Proof toolkit; companion to [stampede](https://github.com/swarmproof/stampede) (stampede simulates the *agents*, mockworld simulates the *world* they act on). Apache-2.0.
8+
9+
**Current state: v0.1 + v0.2 implemented** (on feature branches; see git). Python 3.11+, official `mcp` SDK, pydantic v2, click, httpx; packaged with hatchling. Dev loop: `uv venv && uv pip install -e ".[dev]"`, then `python -m pytest -q` (53 tests, ~1s) and `mockworld <cmd>`.
10+
11+
### Module map (`src/mockworld/`)
12+
- `determinism.py` — the seeded entropy funnel (clock/ids/rng/fault-dice). The root of all guarantees.
13+
- `state.py` — `StateStore` (Memory/SQLite) + copy-on-write `StateView`; session isolation lives here.
14+
- `session.py` — per-session logical counters. `schema.py` — pydantic `mock.yaml` models. `errors.py` — error library + `Result`.
15+
- `faults.py` — fault injector (probabilistic + `when:` conditional). `dispatch.py` — CRUD + handler ABI. `handler_ctx.py` — the `ctx` handed to handlers.
16+
- `loader.py` — load a mock dir (+ registry-installed resolution). `engine.py` — the transport-free call path (start here to trace a request).
17+
- `trace.py` — OTel-GenAI-profile spans. `server.py` — MCP stdio+HTTP adapter. `control.py` — control plane + stampede `Target`. `cli.py` — commands. `validate.py` — the entropy linter.
18+
- `registry.py` (v0.2) — `add`/`search`, checksum + safety gate. `world.py` (v0.2) — compose mocks with a shared identity pool. `record.py` (v0.2) — OpenAPI → scaffold.
19+
- `snapshot.py` (v0.3) — portable `.mw.json` artifacts + migration. `swarm.py` (v0.3) — persona swarm → Agent Readiness Report (misuse map). `verify.py` (v0.3) — contract-drift vs OpenAPI.
20+
- `mocks/<name>/` — the five built-ins (`mock.yaml` + `handlers.py` + `seed.py` + `fidelity.md`).
21+
22+
CLI: `run` (stdio/http, also `run world:<file>`), `list`, `inspect`, `validate`, `reset`, `demo`, `add`, `search`, `pack`, `record`. The engine is deliberately MCP-free; server/control/CLI are thin adapters (keeps determinism/isolation tests pure).
23+
24+
## Document map
25+
26+
- `SPEC.md` — the original v1.0 spec/PRD (root-level, high-level).
27+
- `docs/PRD.md` — detailed requirements; **the source of REQ-IDs** (`REQ-DET-*`, `REQ-ISO-*`, `REQ-FAULT-*`, …) that every other doc cross-references.
28+
- `docs/ARCHITECTURE.md` — authoritative design: engine components, the `mock.yaml` schema, handler ABI, session isolation, the stampede integration contract (§7), and ADRs 1–7.
29+
- `docs/DELIVERY-PLAN.md` — milestones (v0.1/0.2/0.3), work breakdown (epics A–I), mock build order, definition of done, launch checklist.
30+
- `docs/TEST-PLAN.md` — test pyramid, E2E scenarios (Given/When/Then), and the CI gates (G-DET, G-ISO, …) that define "green to merge/release".
31+
- `docs/RESEARCH.md` — competitive landscape and open questions.
32+
33+
Doc conventions: `⊕ Beyond original spec` marks design that extends `SPEC.md`; keep REQ-ID cross-references intact when editing; keep the "Last updated" line current on `docs/*` edits.
34+
35+
## Architecture (the invariants all future code must serve)
36+
37+
1. **Determinism is a hard contract, not a mode** (ADR-4). All entropy — clock, RNG, IDs, fault dice — flows through one seeded `DeterministicContext`. Handlers may only use `ctx.clock` / `ctx.ids` / `ctx.rng`; importing `time`, `random`, `uuid` in a handler is a lint violation. Fault dice draw from a *separate* PRNG substream so adding a tool call doesn't shift unrelated faults. `reset(seed)` must be indistinguishable from a fresh boot at that seed. **No LLM in the response path, ever — that's the moat.**
38+
2. **Session isolation rides MCP** (ADR-2). Sessions are keyed on MCP's `Mcp-Session-Id` (stdio = one implicit session), implemented as copy-on-write overlays over an immutable seeded base state — 50+ parallel sessions share one base dataset with no cross-talk.
39+
3. **Declarative-first, Python escape hatch** (ADR-5). A mock is a directory: `mock.yaml` (authoritative), optional `handlers.py` (ABI: `handler(ctx, params) -> Result`, pure w.r.t. injected entropy), optional `seed.py`, and `fidelity.md` documenting what it does/doesn't model. Simple CRUD needs no code.
40+
4. **Fault split with stampede** (ADR-6): mockworld owns *business-logic* faults only (`card_declined`, `insufficient_funds`, `rate_limited`, latency, partial outage) as first-class objects with realistic error bodies. Transport chaos (connection kills, socket timeouts, malformed frames) belongs to stampede/Toxiproxy — never implement it here. When a `MockworldTarget` is in use, stampede suppresses its transport rate_limit in favor of mockworld's semantic 429.
41+
5. **State store**: `MemoryStore` default, `SQLiteStore` for persistence/snapshots, behind one `StateStore` API (ADR-3) — both must pass a shared conformance suite.
42+
6. **Consume siblings' primitives, never redefine them.** Tracing uses stampede's trace-format, which is an **OpenTelemetry GenAI profile** — mockworld emits standard `gen_ai.*` attributes plus the shared `swarmproof.*` extension (`swarmproof.span.side="target"`, `swarmproof.fault.{type,injected,source}`). No `mockworld.*` namespace. Target spans are `span.kind=SERVER`, parented to stampede's `execute_tool` CLIENT span, joined on echoed `gen_ai.tool.call.id`; `traceparent` is read from HTTP headers or MCP `_meta.traceparent` on stdio.
43+
44+
### The stampede contract (confirmed 2026-07-13, ARCHITECTURE §7)
45+
46+
mockworld implements stampede's full `Target` protocol: `discover / invoke / reset(seed) / health / isolation() → per_agent / safety_descriptor() → {sandboxed: True}`, plus a control plane (`boot / reset / set_faults / snapshot / restore / session_reset`). `reset(seed)` means state is a *pure function* of the seed. Changes to this seam must stay consistent with stampede's side of the contract.
47+
48+
### v0.1 mock build order (stampede-demo-driven, not SPEC order)
49+
50+
`payments` (marquee) → `crm` (misuse-map demo) → `exchange` → `email` → `files`. Each mock must enforce its stateful invariants (e.g. refund ≤ captured, balance conservation, soft-delete ≠ hard-delete) and declare ≥3 seeded faults — see TEST-PLAN §7 for the per-mock acceptance table.
51+
52+
## Testing philosophy (when code lands)
53+
54+
Determinism/replay tests are the load-bearing acceptance gates, not an afterthought. Merge-blocking CI gates: G-DET (byte-identical transcripts across runs/hosts/both stores, DT-1..6), G-LINT (ambient-entropy lint + `mockworld validate`), G-ISO (isolation incl. 50 parallel sessions), G-UNIT (≥90% on engine core). E2E scenarios in TEST-PLAN §4 are the release gates.
55+
56+
## Conventions
57+
58+
- Conventional Commits (`feat:`, `fix:`, `docs:`, …); branches `feat/<short-name>`; atomic commits.
59+
- Scope discipline: mocks are "realistic enough to break agents correctly," never vendor-exact clones (non-goal NG2) — resist fidelity scope creep; `fidelity.md` is where coverage boundaries live.
60+
- Toolkit principles: provider-agnostic, honest over impressive, watchable & reproducible (seedable outputs).

‎CHANGELOG.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,20 @@
33
All notable changes to mockworld are documented here. Format loosely follows
44
[Keep a Changelog](https://keepachangelog.com/); versions follow SemVer.
55

6+
## [0.3.0] — 2026-09-04
7+
8+
### Added
9+
- **Runtime sandbox for untrusted registry mocks** (ADR-7 v0.2b, REQ-REG-3).
10+
Registry-installed handler/seed code is no longer imported in the host process
11+
at any point — not at `mockworld add`, not at load, not per call. It runs in a
12+
hardened subprocess with the network, subprocess/exec, ctypes, and file writes
13+
neutered before any untrusted import, plus CPU/memory limits. The parent keeps
14+
ownership of state and entropy, so a sandboxed mock is byte-identical to a
15+
trusted one. Locally-authored mocks stay trusted. Defense-in-depth, not a formal
16+
guarantee — for hard isolation, run mockworld in a container.
17+
- `mockworld validate --import-handlers=false` path (used by `add`) runs static
18+
checks only, so validation itself never executes untrusted code.
19+
620
## [0.2.3] — 2026-09-04
721

822
### Changed

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ build-backend = "hatchling.build"
66
# Distribution name: the plain `mockworld` name is reserved by another project on
77
# PyPI, so we ship as `mockworld-mcp`. The import package and CLI stay `mockworld`.
88
name = "mockworld-mcp"
9-
version = "0.2.3"
9+
version = "0.3.0"
1010
description = "A synthetic internet for agents — deterministic, LLM-free fake services (Stripe, Gmail, exchange, CRM, S3) as MCP servers."
1111
readme = "README.md"
1212
requires-python = ">=3.11"

‎src/mockworld/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
load_mock,
1919
)
2020

21-
__version__ = "0.2.3"
21+
__version__ = "0.3.0"
2222

2323
__all__ = [
2424
"Engine",

‎src/mockworld/_sandbox_worker.py‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
"""Sandbox worker — runs ONE untrusted mock's code in a hardened subprocess.
2+
3+
Launched by :mod:`mockworld.sandbox` as ``python -m mockworld._sandbox_worker <dir>``.
4+
Everything untrusted (importing ``handlers.py``/``seed.py`` and running them) happens
5+
here, never in the parent. Before any untrusted import we neuter the network,
6+
subprocess/exec, ctypes, and file writes, and apply CPU/memory limits.
7+
8+
This is defense-in-depth, not a formal guarantee: in-process Python can't be made
9+
perfectly escape-proof. It removes the easy paths (exfiltration, spawning
10+
processes, trashing files) and contains crashes/hangs to a disposable child that
11+
holds none of the parent's state. For hard isolation, run mockworld in a container.
12+
13+
Protocol: length-prefixed JSON (4-byte big-endian length + UTF-8 body) on a private
14+
fd duped from stdout; the child's own stdout is redirected to stderr so handler
15+
`print()`s can't corrupt the stream.
16+
"""
17+
18+
from __future__ import annotations
19+
20+
import builtins
21+
import json
22+
import os
23+
import struct
24+
import sys
25+
from pathlib import Path
26+
27+
28+
def _harden() -> None:
29+
"""Neuter dangerous capabilities on already-imported modules, then lock limits."""
30+
def blocked(*_a, **_k):
31+
raise PermissionError("blocked in mockworld sandbox")
32+
33+
import socket
34+
socket.socket = blocked
35+
socket.create_connection = blocked
36+
socket.create_server = blocked
37+
38+
import subprocess
39+
for fn in ("Popen", "run", "call", "check_call", "check_output", "getoutput", "getstatusoutput"):
40+
if hasattr(subprocess, fn):
41+
setattr(subprocess, fn, blocked)
42+
43+
for fn in ("system", "popen", "fork", "forkpty", "exec", "execv", "execve", "execvp",
44+
"execvpe", "execl", "execle", "execlp", "execlpe", "spawnv", "spawnve",
45+
"spawnl", "spawnlp", "remove", "unlink", "rmdir", "removedirs", "rename",
46+
"replace", "truncate", "kill", "killpg"):
47+
if hasattr(os, fn):
48+
setattr(os, fn, blocked)
49+
50+
try:
51+
import ctypes
52+
for fn in ("CDLL", "PyDLL", "WinDLL", "OleDLL", "cdll", "pydll", "windll"):
53+
if hasattr(ctypes, fn):
54+
setattr(ctypes, fn, blocked)
55+
except Exception:
56+
pass
57+
58+
real_open = builtins.open
59+
60+
def safe_open(file, mode="r", *a, **k):
61+
if any(c in mode for c in "wax+"):
62+
raise PermissionError("file writes are blocked in the mockworld sandbox")
63+
return real_open(file, mode, *a, **k)
64+
65+
builtins.open = safe_open
66+
67+
try:
68+
import resource
69+
resource.setrlimit(resource.RLIMIT_CPU, (5, 6)) # ~5 CPU-seconds per worker
70+
mem = 512 * 1024 * 1024
71+
for lim in ("RLIMIT_AS", "RLIMIT_DATA"):
72+
if hasattr(resource, lim):
73+
try:
74+
resource.setrlimit(getattr(resource, lim), (mem, mem))
75+
except (ValueError, OSError):
76+
pass
77+
except Exception:
78+
pass
79+
80+
81+
def _read_msg(stream) -> dict | None:
82+
header = stream.read(4)
83+
if len(header) < 4:
84+
return None
85+
(length,) = struct.unpack(">I", header)
86+
return json.loads(stream.read(length))
87+
88+
89+
def _write_msg(stream, obj: dict) -> None:
90+
body = json.dumps(obj).encode()
91+
stream.write(struct.pack(">I", len(body)))
92+
stream.write(body)
93+
stream.flush()
94+
95+
96+
def _result_payload(result) -> dict:
97+
if result.success:
98+
return {"success": True, "data": result.data, "meta": result.meta}
99+
err = result.err
100+
return {"success": False, "error": {
101+
"code": err.code, "message": err.message, "http_status": err.http_status,
102+
"body": err.body, "retry_after_s": err.retry_after_s}}
103+
104+
105+
def main() -> None:
106+
mock_dir = Path(sys.argv[1])
107+
108+
# Private protocol channel; keep the real stdout clean.
109+
proto_out = os.fdopen(os.dup(1), "wb")
110+
inp = sys.stdin.buffer
111+
sys.stdout = sys.stderr
112+
113+
_harden() # <-- everything below runs under the neutered environment
114+
115+
import yaml
116+
117+
from mockworld.datagen import DataGen
118+
from mockworld.determinism import DeterministicContext
119+
from mockworld.errors import register_error
120+
from mockworld.handler_ctx import FaultHelper, HandlerCtx
121+
from mockworld.loader import SeedCtx, _import_module
122+
from mockworld.schema import MockDef
123+
from mockworld.state import _TOMBSTONE, StateView
124+
125+
definition = MockDef.model_validate(yaml.safe_load((mock_dir / "mock.yaml").read_text()))
126+
for name, template in definition.errors.items():
127+
register_error(name, template)
128+
handlers = _import_module(mock_dir / "handlers.py", "sbx_handlers")
129+
seed_module = _import_module(mock_dir / "seed.py", "sbx_seed")
130+
131+
def handle(req: dict) -> dict:
132+
op = req["op"]
133+
dctx = DeterministicContext(req["seed"])
134+
135+
if op == "seed":
136+
seed_ctx = SeedCtx(rng=dctx.seed_rng(), ids=dctx.ids_for("__seed__", 0),
137+
fake=DataGen(dctx.seed_rng()))
138+
if definition.seed.generator.startswith("python:") and seed_module is not None:
139+
fn = getattr(seed_module, definition.seed.generator.split(".", 1)[-1])
140+
snapshot = fn(seed_ctx, definition)
141+
else:
142+
snapshot = {} # builtin seed handled parent-side (trusted code)
143+
return {"ok": True, "snapshot": snapshot}
144+
145+
if op == "call":
146+
tool = definition.tool(req["tool"])
147+
fn = getattr(handlers, tool.handler_name, None) if handlers else None
148+
if fn is None:
149+
return {"ok": True, "result": {"success": False, "error": {
150+
"code": "internal_error", "message": f"handler {tool.handler_name!r} not found",
151+
"http_status": 500, "body": {}, "retry_after_s": None}},
152+
"mutations": {}, "deletes": {}}
153+
154+
view = StateView(req["state"], {}, list(req["state"].keys()))
155+
ctx = HandlerCtx(state=view, clock=dctx.clock_for(req["step"]),
156+
ids=dctx.ids_for(req["tool"], req["idx"]),
157+
rng=dctx.rng_for(req["tool"], req["idx"]),
158+
tool=req["tool"], faults=FaultHelper())
159+
result = fn(ctx, req["params"])
160+
161+
mutations: dict = {}
162+
deletes: dict = {}
163+
for coll, entries in view._scratch.items():
164+
for key, value in entries.items():
165+
if value is _TOMBSTONE:
166+
deletes.setdefault(coll, []).append(key)
167+
else:
168+
mutations.setdefault(coll, {})[key] = value
169+
return {"ok": True, "result": _result_payload(result),
170+
"mutations": mutations, "deletes": deletes}
171+
172+
return {"ok": False, "error": f"unknown op {op!r}"}
173+
174+
while True:
175+
req = _read_msg(inp)
176+
if req is None:
177+
break
178+
try:
179+
resp = handle(req)
180+
except Exception as exc: # never crash the worker on handler error
181+
resp = {"ok": False, "error": f"{type(exc).__name__}: {exc}"}
182+
_write_msg(proto_out, resp)
183+
184+
185+
if __name__ == "__main__":
186+
main()

‎src/mockworld/engine.py‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,17 +67,36 @@ def __init__(
6767
exporter = OTLPExporter(otlp_endpoint) if otlp_endpoint else None
6868
self.tracer = TraceEmitter(self.definition.name, self.definition.version, trace_sink, exporter)
6969

70+
# Untrusted (registry-installed) mocks run their handlers in a sandbox.
71+
self.trusted = mock.trusted
72+
self.sandbox = None
73+
if not self.trusted:
74+
from .sandbox import SandboxWorker
75+
76+
self.sandbox = SandboxWorker(mock.path)
77+
import atexit
78+
79+
atexit.register(self.close)
80+
7081
self._profile = self._resolve_profile(faults)
7182
self._seed_base()
7283

84+
def close(self) -> None:
85+
if self.sandbox is not None:
86+
self.sandbox.close()
87+
7388
# -- construction helpers ----------------------------------------------------
7489

7590
@classmethod
7691
def from_source(cls, source: str, **kwargs: Any) -> "Engine":
7792
return cls(load_mock(source), **kwargs)
7893

7994
def _seed_base(self) -> None:
80-
self.store.load_base(self.mock.generate_base(self.dctx, self.shared))
95+
if not self.trusted and self.definition.seed.generator.startswith("python:"):
96+
base = self.sandbox.seed(self.seed) # untrusted seed.py runs in the sandbox
97+
else:
98+
base = self.mock.generate_base(self.dctx, self.shared)
99+
self.store.load_base(base)
81100

82101
def _resolve_profile(self, faults: str | dict) -> FaultProfile:
83102
if isinstance(faults, dict):
@@ -151,7 +170,7 @@ def call(
151170
result = outcome.pre # short-circuit fault: no behavior, no state change
152171
else:
153172
try:
154-
result = self.dispatcher.dispatch(tool, ctx, params)
173+
result = self._dispatch(tool, ctx, params, view, idx, step)
155174
except Exception as exc: # never leak a stack trace to the agent (REQ-RT-11)
156175
view.rollback()
157176
result = Result.error("internal_error", f"{type(exc).__name__}: {exc}")
@@ -176,6 +195,26 @@ def call(
176195

177196
# -- internals ---------------------------------------------------------------
178197

198+
def _dispatch(self, tool, ctx, params, view, idx, step) -> Result:
199+
# Declarative CRUD is our code; only untrusted Python handlers are sandboxed.
200+
if tool.is_crud or self.trusted or self.sandbox is None:
201+
return self.dispatcher.dispatch(tool, ctx, params)
202+
203+
state = {
204+
c: {k: view._read(c, k) for k in view._keys(c)}
205+
for c in self.definition.collection_names()
206+
}
207+
result, mutations, deletes = self.sandbox.call(
208+
seed=self.seed, tool=tool.name, idx=idx, step=step, state=state, params=params
209+
)
210+
for coll, entries in mutations.items():
211+
for key, value in entries.items():
212+
view._write(coll, key, value)
213+
for coll, keys in deletes.items():
214+
for key in keys:
215+
view._delete(coll, key)
216+
return result
217+
179218
def _emit(self, tool, idx, ctx, result, outcome, call_id, traceparent) -> None:
180219
span = self.tracer.build_span(
181220
dctx_hash=self.dctx.stable_hash("span", tool.name, idx),

0 commit comments

Comments
 (0)