Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,14 @@ invart adapter profiles
invart adapter profiles --track managed_wrapper
invart real-agent check --agent claude-code --out-dir .invart/real-agent
invart real-agent run --agent claude-code --require-live --out-dir .invart/live-claude -- <claude-or-fixture-command>
invart real-agent run --agent opencode --require-live --out-dir .invart/live-opencode -- <opencode-or-fixture-command>
invart real-agent run --agent gemini-cli --require-live --out-dir .invart/live-gemini -- <gemini-or-fixture-command>
invart real-agent run --agent aider --require-live --out-dir .invart/live-aider -- <aider-or-fixture-command>
invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html
```

Use `--require-live` when you want missing local agent binaries to fail the run instead of being recorded as blocked evidence. Fixture-backed runs validate the Invart adapter contract; live runs validate the installed product surface.
The plural `adapter profiles` command lists priority agent tracks: reference full adapter, managed wrapper, native bridge, vendor/cloud evidence import, and framework trace import. Vendor import tracks are audit evidence, not Invart mediation.
The plural `adapter profiles` command lists priority agent tracks: reference full adapter, managed wrapper, native bridge, vendor/cloud evidence import, and framework trace import. Vendor import tracks are audit evidence, not Invart mediation. `real-agent check` emits a conformance contract row for each product so imported, discovered, fixture-backed, and live evidence cannot be mixed into a stronger claim.

### Claude Code reference adapter

Expand Down Expand Up @@ -111,5 +114,8 @@ invart eval benchmark --suite v0.9.5-priority-agent-tracks
invart eval benchmark --suite v0.9.6-layer-runtime-workflow
invart eval benchmark --suite v0.9.7-evidence-workspace-gate
invart eval benchmark --suite v0.9.8-claude-full-live-adapter
invart eval benchmark --suite v0.9.9-conformance-contract-v2
invart eval benchmark --suite v0.9.10-opencode-real-adapter
invart eval benchmark --suite v0.9.11-terminal-agent-managed-wrappers
invart roadmap status --require-full
```
8 changes: 7 additions & 1 deletion docs/html/cli-reference.html
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,10 @@ <h2>User intent to command</h2>
invart adapter profiles --track managed_wrapper
invart real-agent check --agent claude-code --out-dir .invart/real-agent
invart real-agent run --agent claude-code --require-live --out-dir .invart/live-claude -- &lt;claude-or-fixture-command&gt;
invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html</pre><p>Use <code>--require-live</code> when missing local binaries should fail instead of being recorded as blocked evidence. <code>adapter profiles</code> lists priority tracks and keeps vendor/cloud import separate from Invart-mediated control.</p></div>
invart real-agent run --agent opencode --require-live --out-dir .invart/live-opencode -- &lt;opencode-or-fixture-command&gt;
invart real-agent run --agent gemini-cli --require-live --out-dir .invart/live-gemini -- &lt;gemini-or-fixture-command&gt;
invart real-agent run --agent aider --require-live --out-dir .invart/live-aider -- &lt;aider-or-fixture-command&gt;
invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html</pre><p>Use <code>--require-live</code> when missing local binaries should fail instead of being recorded as blocked evidence. <code>adapter profiles</code> lists priority tracks and keeps vendor/cloud import separate from Invart-mediated control. <code>real-agent check</code> emits a conformance contract row so imported, discovered, fixture-backed, and live evidence cannot be mixed into stronger claims.</p></div>
<div class="card"><h3>Claude Code reference adapter</h3><pre>invart adapter claude-code \
--target . \
--out-dir .invart/claude-reference \
Expand All @@ -58,6 +61,9 @@ <h2>User intent to command</h2>
invart eval benchmark --suite v0.9.6-layer-runtime-workflow
invart eval benchmark --suite v0.9.7-evidence-workspace-gate
invart eval benchmark --suite v0.9.8-claude-full-live-adapter
invart eval benchmark --suite v0.9.9-conformance-contract-v2
invart eval benchmark --suite v0.9.10-opencode-real-adapter
invart eval benchmark --suite v0.9.11-terminal-agent-managed-wrappers
invart roadmap status --require-full</pre></div>
</section>
</main>
Expand Down
3 changes: 3 additions & 0 deletions docs/html/release-history.html
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ <h2>0.9 Patch Track</h2>
<tr><td>v0.9.6</td><td>Implemented</td><td>Ledger-derived L1-L5 runtime operation workflow via <code>runtime layers</code>, exporting a stage x layer matrix, layer timeline, operator command guide, and linked proof/replay/path graph/coverage/audit/evidence artifacts.</td></tr>
<tr><td>v0.9.7</td><td>Implemented</td><td>L5 evidence workspace and gate hardening via <code>evidence inspect</code>, verifying bundle hashes, required artifacts, layer workflow/adapter package links, and whether proof/audit evidence answers who, what, why, policy, approval, outcome, and coverage.</td></tr>
<tr><td>v0.9.8</td><td>Implemented</td><td>Claude Code strict-live adapter: binary probing, missing-binary failure, pre-side-effect managed risk block, full adapter package, layer-runtime workflow, evidence workspace, and explicit degraded process-tree coverage when only portable subprocess supervision is active.</td></tr>
<tr><td>v0.9.9</td><td>Implemented</td><td>Live adapter conformance contract v2: each real-agent row now records evidence level, control position, side-effect timing, required artifacts, claimable coverage, and a claim gate that rejects vendor/import/discovery evidence inflated into Invart-mediated or enforced coverage.</td></tr>
<tr><td>v0.9.10</td><td>Implemented</td><td>OpenCode real adapter track: binary-backed managed wrapper run, plugin/MCP config inventory, L5 artifact export, benign autonomy preservation, and managed risk blocking before side effects without treating plugin-only inventory as mediation.</td></tr>
<tr><td>v0.9.11</td><td>Implemented</td><td>Terminal-agent managed wrappers for Gemini CLI and Aider: binary-backed wrapper runs, Gemini MCP/config inventory, Aider config/repo-context inventory, artifact parity, and low approval-noise checks for benign workflows.</td></tr>
</table>
</section>
<section>
Expand Down
3 changes: 3 additions & 0 deletions docs/release-history.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ This page is a reference, not the first-run learning path. Start with [Product O
| v0.9.6 | Implemented | Ledger-derived L1-L5 runtime operation workflow via `runtime layers`, exporting a stage x layer matrix, layer timeline, operator command guide, and linked proof/replay/path graph/coverage/audit/evidence artifacts. |
| v0.9.7 | Implemented | L5 evidence workspace and gate hardening via `evidence inspect`, verifying bundle hashes, required artifacts, layer workflow/adapter package links, and whether proof/audit evidence answers who, what, why, policy, approval, outcome, and coverage. |
| v0.9.8 | Implemented | Claude Code strict-live adapter: binary probing, missing-binary failure, pre-side-effect managed risk block, full adapter package, layer-runtime workflow, evidence workspace, and explicit degraded process-tree coverage when only portable subprocess supervision is active. |
| v0.9.9 | Implemented | Live adapter conformance contract v2: each real-agent row now records evidence level, control position, side-effect timing, required artifacts, claimable coverage, and a claim gate that rejects vendor/import/discovery evidence inflated into Invart-mediated or enforced coverage. |
| v0.9.10 | Implemented | OpenCode real adapter track: binary-backed managed wrapper run, plugin/MCP config inventory, L5 artifact export, benign autonomy preservation, and managed risk blocking before side effects without treating plugin-only inventory as mediation. |
| v0.9.11 | Implemented | Terminal-agent managed wrappers for Gemini CLI and Aider: binary-backed wrapper runs, Gemini MCP/config inventory, Aider config/repo-context inventory, artifact parity, and low approval-noise checks for benign workflows. |

## Internal History

Expand Down
6 changes: 6 additions & 0 deletions src/invart/benchmarks/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,12 @@
run_agent_adapter_contract_benchmark,
run_claude_full_live_adapter_benchmark,
run_claude_reference_adapter_benchmark,
run_conformance_contract_v2_benchmark,
run_evidence_workspace_gate_benchmark,
run_layer_runtime_workflow_benchmark,
run_opencode_real_adapter_benchmark,
run_priority_agent_tracks_benchmark,
run_terminal_agent_managed_wrappers_benchmark,
)

BenchmarkRunner = Callable[[], dict[str, Any]]
Expand Down Expand Up @@ -123,6 +126,9 @@ def benchmark_runner_registry() -> dict[str, BenchmarkRunner]:
"v0.9.6-layer-runtime-workflow": run_layer_runtime_workflow_benchmark,
"v0.9.7-evidence-workspace-gate": run_evidence_workspace_gate_benchmark,
"v0.9.8-claude-full-live-adapter": run_claude_full_live_adapter_benchmark,
"v0.9.9-conformance-contract-v2": run_conformance_contract_v2_benchmark,
"v0.9.10-opencode-real-adapter": run_opencode_real_adapter_benchmark,
"v0.9.11-terminal-agent-managed-wrappers": run_terminal_agent_managed_wrappers_benchmark,
"progressive-external-validation": run_progressive_external_validation_benchmark,
"real-world-agent-risk-demo": run_real_world_risk_benchmark,
"containerized-risk-demo": run_container_risk_demo_benchmark,
Expand Down
146 changes: 145 additions & 1 deletion src/invart/benchmarks/releases_v52_v57.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@
from invart.control.runtime import close_session, record_action, start_session
from invart.evaluation.release_candidate import verify_release_candidate
from invart.evaluation.product_control_matrix import run_product_control_matrix
from invart.evaluation.real_agent_conformance import run_real_agent_conformance
from invart.evaluation.real_agent_conformance import run_real_agent_conformance, validate_conformance_contract
from invart.surfaces.adapter import run_adapter_command
from invart.surfaces.claude_adapter import run_claude_code_adapter
from invart.surfaces.live_adapter import run_live_agent_adapter
from invart.surfaces.adapter_profiles import adapter_track_matrix, list_adapter_profiles, validate_adapter_profile_truthfulness


Expand Down Expand Up @@ -325,11 +326,154 @@ def run_claude_full_live_adapter_benchmark() -> dict[str, object]:
)


def run_conformance_contract_v2_benchmark() -> dict[str, object]:
with tempfile.TemporaryDirectory(prefix="invart_v099_") as tmp:
root = Path(tmp)
fake = root / "fake-agent"
fake.write_text("#!/usr/bin/env python3\nimport sys\nsys.exit(0)\n", encoding="utf-8")
fake.chmod(0o755)
report = run_real_agent_conformance(
out_dir=root / "conformance",
agents=["claude-code", "openclaw"],
binary_overrides={"claude-code": str(fake), "openclaw": str(fake)},
require_live=True,
)
by_agent = {row["agent"]: row for row in report["agents"]}
inflated = dict(by_agent["openclaw"])
inflated["contract"] = {**dict(inflated["contract"]), "claimable_coverage": "managed_wrapper"}
inflated_gate = validate_conformance_contract([inflated])
checks = {
"contract_schema_present": report.get("conformance_contract", {}).get("schema_version") == "invart.adapter_conformance_contract.v0.9.9",
"managed_wrapper_claim_has_artifacts": by_agent["claude-code"]["contract"]["artifact_completeness"]["status"] == "pass",
"vendor_import_not_mediated": by_agent["openclaw"]["contract"]["claimable_coverage"] == "vendor_import",
"vendor_cannot_claim_pre_side_effect_mediation": "invart_pre_side_effect_mediation" in by_agent["openclaw"]["contract"]["cannot_claim"],
"claim_gate_passes_truthful_rows": report.get("conformance_contract", {}).get("claim_gate", {}).get("status") == "pass",
"claim_gate_fails_inflated_row": inflated_gate.get("status") == "fail",
}
return _suite_result(
"v0.9.9-conformance-contract-v2",
checks,
artifacts=report.get("artifacts", {}),
)


def run_opencode_real_adapter_benchmark() -> dict[str, object]:
with tempfile.TemporaryDirectory(prefix="invart_v0910_") as tmp:
root = Path(tmp)
(root / "opencode.json").write_text('{"plugin":["demo"],"mcp":{"fs":{}}}\n', encoding="utf-8")
fake = root / "fake-opencode"
marker = root / "opencode-marker.txt"
fake.write_text(
"#!/usr/bin/env python3\n"
"import pathlib, sys\n"
"if '--version' in sys.argv:\n"
" raise SystemExit(0)\n"
"if '--write-marker' in sys.argv:\n"
" pathlib.Path(sys.argv[sys.argv.index('--write-marker') + 1]).write_text('ran')\n",
encoding="utf-8",
)
fake.chmod(0o755)
run = run_live_agent_adapter(
agent="opencode",
target=root,
out_dir=root / "opencode",
command=[str(fake), "--write-marker", str(marker)],
binary=str(fake),
require_live=True,
policy_mode="advisory",
)
risk_marker = root / "risk-marker.txt"
risk = run_live_agent_adapter(
agent="opencode",
target=root,
out_dir=root / "opencode-risk",
command=[str(fake), "--write-marker", str(risk_marker), "rm -rf ."],
binary=str(fake),
require_live=True,
policy_mode="managed",
)
inventory = [item for item in run.get("native_inventory", {}).get("profiles", []) if item.get("agent") == "opencode"][0]
checks = {
"live_binary_backed": run.get("live_evidence", {}).get("binary", {}).get("status") == "found",
"managed_wrapper_artifacts": bool(run.get("managed_run", {}).get("ledger")) and bool(run.get("managed_run", {}).get("proof")) and bool(run.get("managed_run", {}).get("package")),
"plugin_config_inventory": bool(inventory.get("surfaces", {}).get("plugins", {}).get("matches")),
"mcp_config_inventory": bool(inventory.get("surfaces", {}).get("mcp", {}).get("matches")),
"benign_keeps_autonomy": run.get("status") == "passed" and marker.exists(),
"managed_risk_stopped_before_side_effect": risk.get("returncode") == 126 and not risk_marker.exists(),
"l5_workspace_present": run.get("evidence_workspace", {}).get("status") == "pass",
}
return _suite_result(
"v0.9.10-opencode-real-adapter",
checks,
artifacts=run.get("artifacts", {}),
)


def run_terminal_agent_managed_wrappers_benchmark() -> dict[str, object]:
with tempfile.TemporaryDirectory(prefix="invart_v0911_") as tmp:
root = Path(tmp)
(root / ".gemini").mkdir()
(root / ".gemini" / "settings.json").write_text('{"mcpServers":{"fs":{}}}\n', encoding="utf-8")
(root / ".git").mkdir()
(root / ".aider.conf.yml").write_text("auto-commits: false\n", encoding="utf-8")
fake = root / "fake-terminal-agent"
fake.write_text(
"#!/usr/bin/env python3\n"
"import pathlib, sys\n"
"if '--version' in sys.argv:\n"
" raise SystemExit(0)\n"
"if '--write-marker' in sys.argv:\n"
" pathlib.Path(sys.argv[sys.argv.index('--write-marker') + 1]).write_text('ran')\n",
encoding="utf-8",
)
fake.chmod(0o755)
runs: dict[str, dict[str, object]] = {}
approval_counts: dict[str, int] = {}
for agent in ("gemini-cli", "aider"):
marker = root / f"{agent}.txt"
run = run_live_agent_adapter(
agent=agent,
target=root,
out_dir=root / agent,
command=[str(fake), "--write-marker", str(marker)],
binary=str(fake),
require_live=True,
policy_mode="advisory",
)
entries, _warnings = load_ledger_entries(Path(run["managed_run"]["ledger"]))
approval_counts[agent] = sum(
1
for entry in entries
if entry.entry_type == "action" and entry.decision and entry.decision.get("effect") == "require_approval"
)
run["marker_exists"] = marker.exists()
runs[agent] = run
gemini_inventory = [item for item in runs["gemini-cli"]["native_inventory"]["profiles"] if item["agent"] == "gemini-cli"][0]
aider_inventory = [item for item in runs["aider"]["native_inventory"]["profiles"] if item["agent"] == "aider"][0]
checks = {
"gemini_managed_run_passed": runs["gemini-cli"].get("status") == "passed" and runs["gemini-cli"].get("marker_exists") is True,
"aider_managed_run_passed": runs["aider"].get("status") == "passed" and runs["aider"].get("marker_exists") is True,
"gemini_mcp_inventory": bool(gemini_inventory.get("surfaces", {}).get("mcp", {}).get("matches")),
"aider_config_inventory": bool(aider_inventory.get("surfaces", {}).get("config", {}).get("matches")),
"aider_repo_context_inventory": bool(aider_inventory.get("surfaces", {}).get("repo_map", {}).get("matches")),
"approval_noise_zero": all(count == 0 for count in approval_counts.values()),
"artifact_parity_present": all(bool(runs[agent].get("managed_run", {}).get("ledger")) and bool(runs[agent].get("managed_run", {}).get("proof")) for agent in runs),
}
return _suite_result(
"v0.9.11-terminal-agent-managed-wrappers",
checks,
artifacts={agent: runs[agent].get("artifacts", {}).get("report_json") for agent in runs},
)


__all__ = [
"run_agent_adapter_contract_benchmark",
"run_claude_full_live_adapter_benchmark",
"run_claude_reference_adapter_benchmark",
"run_conformance_contract_v2_benchmark",
"run_evidence_workspace_gate_benchmark",
"run_opencode_real_adapter_benchmark",
"run_terminal_agent_managed_wrappers_benchmark",
"run_priority_agent_tracks_benchmark",
"run_layer_runtime_workflow_benchmark",
]
3 changes: 3 additions & 0 deletions src/invart/evaluation/benchmark_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@
{"suite": "v0.9.6-layer-runtime-workflow", "version": "v0.9.6", "category": "runtime", "optional_heavy": False, "claim_scope": "local_layer_runtime_workflow", "evidence_level": "ledger_derived_runtime_fixture"},
{"suite": "v0.9.7-evidence-workspace-gate", "version": "v0.9.7", "category": "evidence", "optional_heavy": False, "claim_scope": "local_l5_evidence_workspace", "evidence_level": "ledger_derived_runtime_fixture"},
{"suite": "v0.9.8-claude-full-live-adapter", "version": "v0.9.8", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_claude_live_adapter_contract", "evidence_level": "binary_backed_fixture_live_path"},
{"suite": "v0.9.9-conformance-contract-v2", "version": "v0.9.9", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "adapter_conformance_claim_gate", "evidence_level": "binary_backed_fixture_plus_vendor_import_negative_control"},
{"suite": "v0.9.10-opencode-real-adapter", "version": "v0.9.10", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_opencode_managed_wrapper", "evidence_level": "binary_backed_fixture_with_plugin_inventory"},
{"suite": "v0.9.11-terminal-agent-managed-wrappers", "version": "v0.9.11", "category": "compatibility", "optional_heavy": False, "claim_scope": "local_terminal_agent_wrappers", "evidence_level": "binary_backed_fixture_with_config_inventory"},
{"suite": "progressive-external-validation", "version": "pre-release", "category": "external-validation", "optional_heavy": False, "claim_scope": "progressive_external_validation", "evidence_level": "external_progressive_sample"},
{"suite": "real-world-agent-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "public_source_mapping", "evidence_level": "public_source_seed_plus_local_demo"},
{"suite": "containerized-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "containerized_local_demo", "evidence_level": "per_case_container_artifact_bundle"},
Expand Down
Loading
Loading