cap-bench

A self-contained, offline reproduction of the central empirical results in the CAP paper "A Model-Checked Coordination Protocol for Shared-Resource Handover in Autonomous Earthwork Fleets."

cap-bench runs the real reservation arbiter (from cap-coordination-kit) and the real CAP gRPC runtime (from cap-reference's cap_sdk) against a deterministic, instant-actuation kinematic mock of the fleet. There is no LLM and no physics simulator: a FakeModelClient replays a fixed four-step site plan, and the machines drive scripted straight-line cooperative cycles. The protocol layer being measured — reservation arbitration, the typed Handover lifecycle, frame validation — is exercised exactly as in the closed reference implementation.

It carries no dependency on the closed cap_pangaea implementation (enforced by tests/test_no_private_imports.py) and exposes no LLM/Anthropic surface.

What it reproduces

Result in the paper	Command	Expected qualitative output
Per-guard fault-injection ablation (tab:fault_injection, tab:fault_guard_attribution)	python scripts/run_fault_campaign.py --runs 612 --workers 12 --out runs	reservation-keeping configs (cap_full, mutex_only): 0 exclusivity violations, 0 double-grants; unguarded (no_reservation, flag_bt): violate under contention. Frame-validation NACKs confined to state-machine-guarded configs.
Multi-carrier fleet-size sweep (tab:multicarrier_sweep)	python scripts/run_fleet_sweep.py --seeds 30 --workers 12 --out runs_sweep	cap_full/mutex_only: 0 exclusivity violations + 0 double-grants at 1+{2,3,5} carriers; denied-while-held grows ~30 → ~110 → ~270 (mean) with the fleet.

Both scripts/run_fault_campaign.py (single-fleet per-guard campaign) and scripts/run_fleet_sweep.py (the 1+{2,3,5}-carrier sweep) ship here. Runtimes: ~10–30 min for the 612-run campaign on 12 workers; the full 3,240-run sweep is a few hours. Completing arms ~60 s/run, unguarded arms hit the watchdog.

Install / run

# From a checkout of the cooperative-agent-protocol monorepo, dependencies are siblings.
# Put the public protobuf bindings + SDK + kit on the path (or `pip install` the packages):
export PYTHONPATH="$PWD:../cap-coordination-kit:../cap-spec/gen/python:../cap-reference/python"

python -m pytest tests -q --asyncio-mode=auto     # gates: smoke reproduction + no-private-imports
python scripts/run_fault_campaign.py --runs 12 --workers 4 --out runs   # quick pilot

The package targets Python ≥ 3.10 and depends only on cap-sdk, cap-coordination-kit, grpcio, protobuf, numpy, and pydantic.

Honest scope

cap-bench reproduces the protocol-level coordination results: exclusivity, no-double-grant, contention (denied-while-held), and frame-validation NACKs, all read from the authoritative arbiter transaction log. Timing is relative (instant-actuation kinematic mock), not physical. The physics-based witness, the LLM agents, and the live operator integration belong to the closed reference implementation and are not part of this benchmark.

Apache-2.0.