diff --git a/docs/cli-reference.md b/docs/cli-reference.md index fda7ebf..d826c81 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -77,11 +77,14 @@ invart adapter profiles invart adapter profiles --track managed_wrapper invart real-agent check --agent claude-code --out-dir .invart/real-agent invart real-agent run --agent claude-code --require-live --out-dir .invart/live-claude -- +invart real-agent run --agent opencode --require-live --out-dir .invart/live-opencode -- +invart real-agent run --agent gemini-cli --require-live --out-dir .invart/live-gemini -- +invart real-agent run --agent aider --require-live --out-dir .invart/live-aider -- invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html ``` Use `--require-live` when you want missing local agent binaries to fail the run instead of being recorded as blocked evidence. Fixture-backed runs validate the Invart adapter contract; live runs validate the installed product surface. -The plural `adapter profiles` command lists priority agent tracks: reference full adapter, managed wrapper, native bridge, vendor/cloud evidence import, and framework trace import. Vendor import tracks are audit evidence, not Invart mediation. +The plural `adapter profiles` command lists priority agent tracks: reference full adapter, managed wrapper, native bridge, vendor/cloud evidence import, and framework trace import. Vendor import tracks are audit evidence, not Invart mediation. `real-agent check` emits a conformance contract row for each product so imported, discovered, fixture-backed, and live evidence cannot be mixed into a stronger claim. ### Claude Code reference adapter @@ -111,5 +114,8 @@ invart eval benchmark --suite v0.9.5-priority-agent-tracks invart eval benchmark --suite v0.9.6-layer-runtime-workflow invart eval benchmark --suite v0.9.7-evidence-workspace-gate invart eval benchmark --suite v0.9.8-claude-full-live-adapter +invart eval benchmark --suite v0.9.9-conformance-contract-v2 +invart eval benchmark --suite v0.9.10-opencode-real-adapter +invart eval benchmark --suite v0.9.11-terminal-agent-managed-wrappers invart roadmap status --require-full ``` diff --git a/docs/html/cli-reference.html b/docs/html/cli-reference.html index e72a9dd..2a0f3d5 100644 --- a/docs/html/cli-reference.html +++ b/docs/html/cli-reference.html @@ -41,7 +41,10 @@

User intent to command

invart adapter profiles --track managed_wrapper invart real-agent check --agent claude-code --out-dir .invart/real-agent invart real-agent run --agent claude-code --require-live --out-dir .invart/live-claude -- <claude-or-fixture-command> -invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html

Use --require-live when missing local binaries should fail instead of being recorded as blocked evidence. adapter profiles lists priority tracks and keeps vendor/cloud import separate from Invart-mediated control.

+invart real-agent run --agent opencode --require-live --out-dir .invart/live-opencode -- <opencode-or-fixture-command> +invart real-agent run --agent gemini-cli --require-live --out-dir .invart/live-gemini -- <gemini-or-fixture-command> +invart real-agent run --agent aider --require-live --out-dir .invart/live-aider -- <aider-or-fixture-command> +invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html

Use --require-live when missing local binaries should fail instead of being recorded as blocked evidence. adapter profiles lists priority tracks and keeps vendor/cloud import separate from Invart-mediated control. real-agent check emits a conformance contract row so imported, discovered, fixture-backed, and live evidence cannot be mixed into stronger claims.

Claude Code reference adapter

invart adapter claude-code \
   --target . \
   --out-dir .invart/claude-reference \
@@ -58,6 +61,9 @@ 

User intent to command

invart eval benchmark --suite v0.9.6-layer-runtime-workflow invart eval benchmark --suite v0.9.7-evidence-workspace-gate invart eval benchmark --suite v0.9.8-claude-full-live-adapter +invart eval benchmark --suite v0.9.9-conformance-contract-v2 +invart eval benchmark --suite v0.9.10-opencode-real-adapter +invart eval benchmark --suite v0.9.11-terminal-agent-managed-wrappers invart roadmap status --require-full
diff --git a/docs/html/release-history.html b/docs/html/release-history.html index ce55b8f..958c7eb 100644 --- a/docs/html/release-history.html +++ b/docs/html/release-history.html @@ -39,6 +39,9 @@

0.9 Patch Track

v0.9.6ImplementedLedger-derived L1-L5 runtime operation workflow via runtime layers, exporting a stage x layer matrix, layer timeline, operator command guide, and linked proof/replay/path graph/coverage/audit/evidence artifacts. v0.9.7ImplementedL5 evidence workspace and gate hardening via evidence inspect, verifying bundle hashes, required artifacts, layer workflow/adapter package links, and whether proof/audit evidence answers who, what, why, policy, approval, outcome, and coverage. v0.9.8ImplementedClaude Code strict-live adapter: binary probing, missing-binary failure, pre-side-effect managed risk block, full adapter package, layer-runtime workflow, evidence workspace, and explicit degraded process-tree coverage when only portable subprocess supervision is active. + v0.9.9ImplementedLive adapter conformance contract v2: each real-agent row now records evidence level, control position, side-effect timing, required artifacts, claimable coverage, and a claim gate that rejects vendor/import/discovery evidence inflated into Invart-mediated or enforced coverage. + v0.9.10ImplementedOpenCode real adapter track: binary-backed managed wrapper run, plugin/MCP config inventory, L5 artifact export, benign autonomy preservation, and managed risk blocking before side effects without treating plugin-only inventory as mediation. + v0.9.11ImplementedTerminal-agent managed wrappers for Gemini CLI and Aider: binary-backed wrapper runs, Gemini MCP/config inventory, Aider config/repo-context inventory, artifact parity, and low approval-noise checks for benign workflows.
diff --git a/docs/release-history.md b/docs/release-history.md index 8ea93b5..d1aa345 100644 --- a/docs/release-history.md +++ b/docs/release-history.md @@ -38,6 +38,9 @@ This page is a reference, not the first-run learning path. Start with [Product O | v0.9.6 | Implemented | Ledger-derived L1-L5 runtime operation workflow via `runtime layers`, exporting a stage x layer matrix, layer timeline, operator command guide, and linked proof/replay/path graph/coverage/audit/evidence artifacts. | | v0.9.7 | Implemented | L5 evidence workspace and gate hardening via `evidence inspect`, verifying bundle hashes, required artifacts, layer workflow/adapter package links, and whether proof/audit evidence answers who, what, why, policy, approval, outcome, and coverage. | | v0.9.8 | Implemented | Claude Code strict-live adapter: binary probing, missing-binary failure, pre-side-effect managed risk block, full adapter package, layer-runtime workflow, evidence workspace, and explicit degraded process-tree coverage when only portable subprocess supervision is active. | +| v0.9.9 | Implemented | Live adapter conformance contract v2: each real-agent row now records evidence level, control position, side-effect timing, required artifacts, claimable coverage, and a claim gate that rejects vendor/import/discovery evidence inflated into Invart-mediated or enforced coverage. | +| v0.9.10 | Implemented | OpenCode real adapter track: binary-backed managed wrapper run, plugin/MCP config inventory, L5 artifact export, benign autonomy preservation, and managed risk blocking before side effects without treating plugin-only inventory as mediation. | +| v0.9.11 | Implemented | Terminal-agent managed wrappers for Gemini CLI and Aider: binary-backed wrapper runs, Gemini MCP/config inventory, Aider config/repo-context inventory, artifact parity, and low approval-noise checks for benign workflows. | ## Internal History diff --git a/src/invart/benchmarks/registry.py b/src/invart/benchmarks/registry.py index c4cab93..55b2494 100644 --- a/src/invart/benchmarks/registry.py +++ b/src/invart/benchmarks/registry.py @@ -59,9 +59,12 @@ run_agent_adapter_contract_benchmark, run_claude_full_live_adapter_benchmark, run_claude_reference_adapter_benchmark, + run_conformance_contract_v2_benchmark, run_evidence_workspace_gate_benchmark, run_layer_runtime_workflow_benchmark, + run_opencode_real_adapter_benchmark, run_priority_agent_tracks_benchmark, + run_terminal_agent_managed_wrappers_benchmark, ) BenchmarkRunner = Callable[[], dict[str, Any]] @@ -123,6 +126,9 @@ def benchmark_runner_registry() -> dict[str, BenchmarkRunner]: "v0.9.6-layer-runtime-workflow": run_layer_runtime_workflow_benchmark, "v0.9.7-evidence-workspace-gate": run_evidence_workspace_gate_benchmark, "v0.9.8-claude-full-live-adapter": run_claude_full_live_adapter_benchmark, + "v0.9.9-conformance-contract-v2": run_conformance_contract_v2_benchmark, + "v0.9.10-opencode-real-adapter": run_opencode_real_adapter_benchmark, + "v0.9.11-terminal-agent-managed-wrappers": run_terminal_agent_managed_wrappers_benchmark, "progressive-external-validation": run_progressive_external_validation_benchmark, "real-world-agent-risk-demo": run_real_world_risk_benchmark, "containerized-risk-demo": run_container_risk_demo_benchmark, diff --git a/src/invart/benchmarks/releases_v52_v57.py b/src/invart/benchmarks/releases_v52_v57.py index 6f29fcb..6f9bf05 100644 --- a/src/invart/benchmarks/releases_v52_v57.py +++ b/src/invart/benchmarks/releases_v52_v57.py @@ -13,9 +13,10 @@ from invart.control.runtime import close_session, record_action, start_session from invart.evaluation.release_candidate import verify_release_candidate from invart.evaluation.product_control_matrix import run_product_control_matrix -from invart.evaluation.real_agent_conformance import run_real_agent_conformance +from invart.evaluation.real_agent_conformance import run_real_agent_conformance, validate_conformance_contract from invart.surfaces.adapter import run_adapter_command from invart.surfaces.claude_adapter import run_claude_code_adapter +from invart.surfaces.live_adapter import run_live_agent_adapter from invart.surfaces.adapter_profiles import adapter_track_matrix, list_adapter_profiles, validate_adapter_profile_truthfulness @@ -325,11 +326,154 @@ def run_claude_full_live_adapter_benchmark() -> dict[str, object]: ) +def run_conformance_contract_v2_benchmark() -> dict[str, object]: + with tempfile.TemporaryDirectory(prefix="invart_v099_") as tmp: + root = Path(tmp) + fake = root / "fake-agent" + fake.write_text("#!/usr/bin/env python3\nimport sys\nsys.exit(0)\n", encoding="utf-8") + fake.chmod(0o755) + report = run_real_agent_conformance( + out_dir=root / "conformance", + agents=["claude-code", "openclaw"], + binary_overrides={"claude-code": str(fake), "openclaw": str(fake)}, + require_live=True, + ) + by_agent = {row["agent"]: row for row in report["agents"]} + inflated = dict(by_agent["openclaw"]) + inflated["contract"] = {**dict(inflated["contract"]), "claimable_coverage": "managed_wrapper"} + inflated_gate = validate_conformance_contract([inflated]) + checks = { + "contract_schema_present": report.get("conformance_contract", {}).get("schema_version") == "invart.adapter_conformance_contract.v0.9.9", + "managed_wrapper_claim_has_artifacts": by_agent["claude-code"]["contract"]["artifact_completeness"]["status"] == "pass", + "vendor_import_not_mediated": by_agent["openclaw"]["contract"]["claimable_coverage"] == "vendor_import", + "vendor_cannot_claim_pre_side_effect_mediation": "invart_pre_side_effect_mediation" in by_agent["openclaw"]["contract"]["cannot_claim"], + "claim_gate_passes_truthful_rows": report.get("conformance_contract", {}).get("claim_gate", {}).get("status") == "pass", + "claim_gate_fails_inflated_row": inflated_gate.get("status") == "fail", + } + return _suite_result( + "v0.9.9-conformance-contract-v2", + checks, + artifacts=report.get("artifacts", {}), + ) + + +def run_opencode_real_adapter_benchmark() -> dict[str, object]: + with tempfile.TemporaryDirectory(prefix="invart_v0910_") as tmp: + root = Path(tmp) + (root / "opencode.json").write_text('{"plugin":["demo"],"mcp":{"fs":{}}}\n', encoding="utf-8") + fake = root / "fake-opencode" + marker = root / "opencode-marker.txt" + fake.write_text( + "#!/usr/bin/env python3\n" + "import pathlib, sys\n" + "if '--version' in sys.argv:\n" + " raise SystemExit(0)\n" + "if '--write-marker' in sys.argv:\n" + " pathlib.Path(sys.argv[sys.argv.index('--write-marker') + 1]).write_text('ran')\n", + encoding="utf-8", + ) + fake.chmod(0o755) + run = run_live_agent_adapter( + agent="opencode", + target=root, + out_dir=root / "opencode", + command=[str(fake), "--write-marker", str(marker)], + binary=str(fake), + require_live=True, + policy_mode="advisory", + ) + risk_marker = root / "risk-marker.txt" + risk = run_live_agent_adapter( + agent="opencode", + target=root, + out_dir=root / "opencode-risk", + command=[str(fake), "--write-marker", str(risk_marker), "rm -rf ."], + binary=str(fake), + require_live=True, + policy_mode="managed", + ) + inventory = [item for item in run.get("native_inventory", {}).get("profiles", []) if item.get("agent") == "opencode"][0] + checks = { + "live_binary_backed": run.get("live_evidence", {}).get("binary", {}).get("status") == "found", + "managed_wrapper_artifacts": bool(run.get("managed_run", {}).get("ledger")) and bool(run.get("managed_run", {}).get("proof")) and bool(run.get("managed_run", {}).get("package")), + "plugin_config_inventory": bool(inventory.get("surfaces", {}).get("plugins", {}).get("matches")), + "mcp_config_inventory": bool(inventory.get("surfaces", {}).get("mcp", {}).get("matches")), + "benign_keeps_autonomy": run.get("status") == "passed" and marker.exists(), + "managed_risk_stopped_before_side_effect": risk.get("returncode") == 126 and not risk_marker.exists(), + "l5_workspace_present": run.get("evidence_workspace", {}).get("status") == "pass", + } + return _suite_result( + "v0.9.10-opencode-real-adapter", + checks, + artifacts=run.get("artifacts", {}), + ) + + +def run_terminal_agent_managed_wrappers_benchmark() -> dict[str, object]: + with tempfile.TemporaryDirectory(prefix="invart_v0911_") as tmp: + root = Path(tmp) + (root / ".gemini").mkdir() + (root / ".gemini" / "settings.json").write_text('{"mcpServers":{"fs":{}}}\n', encoding="utf-8") + (root / ".git").mkdir() + (root / ".aider.conf.yml").write_text("auto-commits: false\n", encoding="utf-8") + fake = root / "fake-terminal-agent" + fake.write_text( + "#!/usr/bin/env python3\n" + "import pathlib, sys\n" + "if '--version' in sys.argv:\n" + " raise SystemExit(0)\n" + "if '--write-marker' in sys.argv:\n" + " pathlib.Path(sys.argv[sys.argv.index('--write-marker') + 1]).write_text('ran')\n", + encoding="utf-8", + ) + fake.chmod(0o755) + runs: dict[str, dict[str, object]] = {} + approval_counts: dict[str, int] = {} + for agent in ("gemini-cli", "aider"): + marker = root / f"{agent}.txt" + run = run_live_agent_adapter( + agent=agent, + target=root, + out_dir=root / agent, + command=[str(fake), "--write-marker", str(marker)], + binary=str(fake), + require_live=True, + policy_mode="advisory", + ) + entries, _warnings = load_ledger_entries(Path(run["managed_run"]["ledger"])) + approval_counts[agent] = sum( + 1 + for entry in entries + if entry.entry_type == "action" and entry.decision and entry.decision.get("effect") == "require_approval" + ) + run["marker_exists"] = marker.exists() + runs[agent] = run + gemini_inventory = [item for item in runs["gemini-cli"]["native_inventory"]["profiles"] if item["agent"] == "gemini-cli"][0] + aider_inventory = [item for item in runs["aider"]["native_inventory"]["profiles"] if item["agent"] == "aider"][0] + checks = { + "gemini_managed_run_passed": runs["gemini-cli"].get("status") == "passed" and runs["gemini-cli"].get("marker_exists") is True, + "aider_managed_run_passed": runs["aider"].get("status") == "passed" and runs["aider"].get("marker_exists") is True, + "gemini_mcp_inventory": bool(gemini_inventory.get("surfaces", {}).get("mcp", {}).get("matches")), + "aider_config_inventory": bool(aider_inventory.get("surfaces", {}).get("config", {}).get("matches")), + "aider_repo_context_inventory": bool(aider_inventory.get("surfaces", {}).get("repo_map", {}).get("matches")), + "approval_noise_zero": all(count == 0 for count in approval_counts.values()), + "artifact_parity_present": all(bool(runs[agent].get("managed_run", {}).get("ledger")) and bool(runs[agent].get("managed_run", {}).get("proof")) for agent in runs), + } + return _suite_result( + "v0.9.11-terminal-agent-managed-wrappers", + checks, + artifacts={agent: runs[agent].get("artifacts", {}).get("report_json") for agent in runs}, + ) + + __all__ = [ "run_agent_adapter_contract_benchmark", "run_claude_full_live_adapter_benchmark", "run_claude_reference_adapter_benchmark", + "run_conformance_contract_v2_benchmark", "run_evidence_workspace_gate_benchmark", + "run_opencode_real_adapter_benchmark", + "run_terminal_agent_managed_wrappers_benchmark", "run_priority_agent_tracks_benchmark", "run_layer_runtime_workflow_benchmark", ] diff --git a/src/invart/evaluation/benchmark_registry.py b/src/invart/evaluation/benchmark_registry.py index 8e29935..38c315d 100644 --- a/src/invart/evaluation/benchmark_registry.py +++ b/src/invart/evaluation/benchmark_registry.py @@ -37,6 +37,9 @@ {"suite": "v0.9.6-layer-runtime-workflow", "version": "v0.9.6", "category": "runtime", "optional_heavy": False, "claim_scope": "local_layer_runtime_workflow", "evidence_level": "ledger_derived_runtime_fixture"}, {"suite": "v0.9.7-evidence-workspace-gate", "version": "v0.9.7", "category": "evidence", "optional_heavy": False, "claim_scope": "local_l5_evidence_workspace", "evidence_level": "ledger_derived_runtime_fixture"}, {"suite": "v0.9.8-claude-full-live-adapter", "version": "v0.9.8", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_claude_live_adapter_contract", "evidence_level": "binary_backed_fixture_live_path"}, + {"suite": "v0.9.9-conformance-contract-v2", "version": "v0.9.9", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "adapter_conformance_claim_gate", "evidence_level": "binary_backed_fixture_plus_vendor_import_negative_control"}, + {"suite": "v0.9.10-opencode-real-adapter", "version": "v0.9.10", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_opencode_managed_wrapper", "evidence_level": "binary_backed_fixture_with_plugin_inventory"}, + {"suite": "v0.9.11-terminal-agent-managed-wrappers", "version": "v0.9.11", "category": "compatibility", "optional_heavy": False, "claim_scope": "local_terminal_agent_wrappers", "evidence_level": "binary_backed_fixture_with_config_inventory"}, {"suite": "progressive-external-validation", "version": "pre-release", "category": "external-validation", "optional_heavy": False, "claim_scope": "progressive_external_validation", "evidence_level": "external_progressive_sample"}, {"suite": "real-world-agent-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "public_source_mapping", "evidence_level": "public_source_seed_plus_local_demo"}, {"suite": "containerized-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "containerized_local_demo", "evidence_level": "per_case_container_artifact_bundle"}, diff --git a/src/invart/evaluation/real_agent_conformance.py b/src/invart/evaluation/real_agent_conformance.py index 00bfea8..5ef3b83 100644 --- a/src/invart/evaluation/real_agent_conformance.py +++ b/src/invart/evaluation/real_agent_conformance.py @@ -13,7 +13,8 @@ from invart.surfaces.adapter_profiles import get_adapter_profile, list_adapter_profiles, validate_adapter_profile_truthfulness -SCHEMA_VERSION = "invart.real_agent_conformance.v0.9.3" +SCHEMA_VERSION = "invart.real_agent_conformance.v0.9.9" +CONTRACT_SCHEMA_VERSION = "invart.adapter_conformance_contract.v0.9.9" DEFAULT_REQUIRED_AGENTS = ("claude-code", "codex", "hermes", "openclaw") @@ -41,12 +42,13 @@ def run_real_agent_conformance( ) for agent in selected_agents ] + claim_gate = validate_conformance_contract(rows) failed_rows = [ row for row in rows if row["status"] == "failed_run" or (require_live and row["status"] == "blocked_missing_binary") ] - status = "fail" if failed_rows or profile_validation["status"] != "pass" else "pass" + status = "fail" if failed_rows or profile_validation["status"] != "pass" or claim_gate["status"] != "pass" else "pass" report_json = root / "real-agent-conformance.json" report_html = root / "real-agent-conformance.html" report: dict[str, Any] = { @@ -56,16 +58,24 @@ def run_real_agent_conformance( "required_live": require_live, "required_agents": selected_agents, "profile_validation": profile_validation, + "conformance_contract": { + "schema_version": CONTRACT_SCHEMA_VERSION, + "status": claim_gate["status"], + "claim_gate": claim_gate, + "levels": ["live", "binary_backed_fixture", "managed_wrapper", "native_bridge", "vendor_import", "discovery_only", "missing_binary"], + "rule": "Invart-mediated or enforced claims require ledger-backed mediation artifacts; vendor/import/discovery evidence cannot satisfy those claims.", + }, "summary": { "agents": len(rows), "passed_agents": sum(1 for row in rows if row["status"] == "pass"), "blocked_missing_binary": sum(1 for row in rows if row["status"] == "blocked_missing_binary"), "failed_agents": len(failed_rows), + "claim_gate_status": claim_gate["status"], "claim_boundary": "Fixture-backed checks validate Invart's conformance harness. Strict live mode fails when requested real binaries are unavailable or do not produce managed-run evidence.", }, "agents": rows, "artifacts": {"report_json": str(report_json), "report_html": str(report_html)}, - "evidence_hash": stable_json_hash({"agents": rows, "required_live": require_live}), + "evidence_hash": stable_json_hash({"agents": rows, "required_live": require_live, "claim_gate": claim_gate}), } write_json_artifact(report_json, report) write_html_artifact(report_html, render_real_agent_conformance_html(report)) @@ -94,6 +104,7 @@ def render_real_agent_conformance_html(report: dict[str, Any]) -> str: f"{html.escape(str(agent.get('coverage', {}).get('coverage_grade')))}" f"{html.escape(str(agent.get('binary', {}).get('status')))}" f"{html.escape(str(agent.get('managed_run', {}).get('status')))}" + f"{html.escape(str(agent.get('contract', {}).get('claimable_coverage')))}" f"{html.escape(str(agent.get('claim_boundary')))}" "" ) @@ -104,7 +115,7 @@ def render_real_agent_conformance_html(report: dict[str, Any]) -> str: "td,th{border-bottom:1px solid #e5e7eb;padding:8px;text-align:left;vertical-align:top}
" f"

Real Agent Conformance

Status: {html.escape(str(report.get('status')))}

" f"

{html.escape(str(report.get('summary', {}).get('claim_boundary', '')))}

" - "" + "
AgentStatusCoverage GradeBinaryManaged RunClaim Boundary
" f"{''.join(rows)}
AgentStatusCoverage GradeBinaryManaged RunClaimable CoverageClaim Boundary
" ) @@ -132,6 +143,7 @@ def _run_agent_check(*, agent: str, out_dir: Path, target: Path, binary_override "source_urls": profile["source_urls"], } if binary["status"] != "found": + row["contract"] = _row_contract(profile=profile, row=row, binary=binary) return row run = run_adapter_command( @@ -150,12 +162,115 @@ def _run_agent_check(*, agent: str, out_dir: Path, target: Path, binary_override "returncode": run.returncode, "ledger": run.ledger, "proof": run.proof, + "package": run.package, } - row["evidence"] = {"ledger": run.ledger, "proof": run.proof} + row["evidence"] = {"ledger": run.ledger, "proof": run.proof, "package": run.package} row["status"] = "pass" if run.status == "passed" else "failed_run" + row["contract"] = _row_contract(profile=profile, row=row, binary=binary) return row +def validate_conformance_contract(rows: list[dict[str, Any]]) -> dict[str, Any]: + findings: list[dict[str, Any]] = [] + for row in rows: + contract = row.get("contract", {}) if isinstance(row.get("contract"), dict) else {} + profile = row.get("coverage", {}) if isinstance(row.get("coverage"), dict) else {} + control_position = str(contract.get("control_position") or "") + claimable = str(contract.get("claimable_coverage") or "") + required_ok = contract.get("artifact_completeness", {}).get("status") in {"pass", "not_required"} + if claimable in {"full_live_adapter", "managed_wrapper"} and not required_ok: + findings.append(_contract_finding(row, "artifact.missing_for_mediated_claim", "Invart-mediated claims require ledger/proof evidence.")) + if control_position == "vendor_owned_import" and claimable in {"managed_wrapper", "full_live_adapter", "enforced"}: + findings.append(_contract_finding(row, "claim.vendor_import_inflation", "Vendor-owned import evidence cannot satisfy Invart-mediated or enforced coverage.")) + if control_position == "discovery_only" and claimable != "discovery_only": + findings.append(_contract_finding(row, "claim.discovery_inflation", "Discovery-only evidence cannot satisfy runtime mediation claims.")) + if profile.get("coverage_grade") == "vendor_evidence_import" and bool(profile.get("supports_mediation")): + findings.append(_contract_finding(row, "profile.vendor_import_supports_mediation", "Vendor import profiles must not be marked as Invart-mediated.")) + return { + "schema_version": "invart.adapter_conformance_claim_gate.v0.9.9", + "status": "pass" if not findings else "fail", + "summary": {"rows": len(rows), "findings": len(findings)}, + "findings": findings, + } + + +def _row_contract(*, profile: dict[str, Any], row: dict[str, Any], binary: dict[str, Any]) -> dict[str, Any]: + evidence = row.get("evidence", {}) if isinstance(row.get("evidence"), dict) else {} + artifacts = { + "binary": binary.get("status") == "found", + "ledger": bool(evidence.get("ledger") and Path(str(evidence["ledger"])).exists()), + "proof": bool(evidence.get("proof") and Path(str(evidence["proof"])).exists()), + "package": bool(evidence.get("package") and Path(str(evidence["package"])).exists()) if evidence.get("package") else False, + } + artifact_required = _required_for_claim(profile, binary) + missing = [name for name in artifact_required if not artifacts.get(name)] + control_position = str(profile.get("control_position")) + if binary.get("status") != "found": + evidence_level = "missing_binary" + claimable = "missing_binary" + elif control_position == "vendor_owned_import": + evidence_level = "vendor_import" + claimable = "vendor_import" + elif control_position == "bridge_mediated_when_configured": + evidence_level = "native_bridge_fixture" + claimable = "native_bridge" if not missing else "discovery_only" + elif control_position == "invart_mediated" and artifacts["ledger"] and artifacts["proof"]: + evidence_level = "binary_backed_fixture" + claimable = "managed_wrapper" + else: + evidence_level = "discovery_only" + claimable = "discovery_only" + return { + "schema_version": CONTRACT_SCHEMA_VERSION, + "agent": profile.get("agent_id"), + "evidence_level": evidence_level, + "control_position": control_position, + "side_effect_timing": "pre_side_effect" if claimable in {"managed_wrapper", "native_bridge"} else "after_the_fact_or_discovery", + "required_artifacts": artifact_required, + "profile_required_artifacts": list(profile.get("required_artifacts", [])), + "artifact_completeness": { + "status": "pass" if not missing else "fail" if artifact_required else "not_required", + "present": artifacts, + "missing": missing, + }, + "claimable_coverage": claimable, + "cannot_claim": _cannot_claim(profile, claimable), + "source_urls": profile.get("source_urls", []), + "last_reviewed": profile.get("last_reviewed"), + } + + +def _required_for_claim(profile: dict[str, Any], binary: dict[str, Any]) -> list[str]: + if binary.get("status") != "found": + return ["binary"] + if profile.get("control_position") == "invart_mediated": + return ["binary", "ledger", "proof"] + if profile.get("control_position") == "bridge_mediated_when_configured": + return ["binary"] + return [] + + +def _cannot_claim(profile: dict[str, Any], claimable: str) -> list[str]: + claims = [] + if claimable not in {"managed_wrapper", "full_live_adapter"}: + claims.append("invart_mediated") + if claimable != "full_live_adapter": + claims.append("full_live_adapter") + claims.append("invart_enforced_without_enforcement_artifact") + if profile.get("control_position") == "vendor_owned_import": + claims.append("invart_pre_side_effect_mediation") + return sorted(set(claims)) + + +def _contract_finding(row: dict[str, Any], check_id: str, message: str) -> dict[str, Any]: + return { + "check_id": check_id, + "severity": "high", + "agent": row.get("agent"), + "message": message, + } + + def _resolve_binary(profile: dict[str, Any], override: str | None) -> dict[str, Any]: candidates = [override] if override else list(profile.get("binary_candidates", [])) for candidate in candidates: @@ -202,7 +317,9 @@ def _safe_id(value: str) -> str: __all__ = [ "DEFAULT_REQUIRED_AGENTS", "SCHEMA_VERSION", + "CONTRACT_SCHEMA_VERSION", "export_real_agent_report_html", "render_real_agent_conformance_html", "run_real_agent_conformance", + "validate_conformance_contract", ] diff --git a/src/invart/evaluation/release_candidate.py b/src/invart/evaluation/release_candidate.py index d3a37f5..6607d85 100644 --- a/src/invart/evaluation/release_candidate.py +++ b/src/invart/evaluation/release_candidate.py @@ -54,6 +54,9 @@ "v0.9.6-layer-runtime-workflow", "v0.9.7-evidence-workspace-gate", "v0.9.8-claude-full-live-adapter", + "v0.9.9-conformance-contract-v2", + "v0.9.10-opencode-real-adapter", + "v0.9.11-terminal-agent-managed-wrappers", "progressive-external-validation", "pre-v1-control-plane", ) diff --git a/src/invart/evaluation/roadmap.py b/src/invart/evaluation/roadmap.py index 8d9a8e0..dbc5047 100644 --- a/src/invart/evaluation/roadmap.py +++ b/src/invart/evaluation/roadmap.py @@ -857,6 +857,51 @@ def internal_docs_for_capability(docs: list[str]) -> list[str]: external_validation="not_run_optional", next_step="Generalize live evidence contracts across OpenCode, Gemini CLI, Aider, Codex, IDE bridges, gateway/server-agent imports, registration, public benchmark slices, and benign-friction studies.", ), + RoadmapCapability( + version="v0.9.9", + capability_id="live_adapter_conformance_contract_v2", + title="Live adapter conformance contract v2", + target="Make live, fixture, imported, and discovered evidence comparable without inflating Invart-mediated or enforced coverage claims.", + status="implemented", + implementation=["src/invart/evaluation/real_agent_conformance.py", "src/invart/surfaces/adapter_profiles.py", "src/invart/benchmarks/releases_v52_v57.py"], + tests=["test_v099_conformance_contract_v2_blocks_claim_inflation", "test_v099_conformance_contract_cli_and_benchmark_are_registered", "v0.9.9-conformance-contract-v2"], + docs=["docs/cli-reference.md", "docs/html/cli-reference.html", "docs/release-history.md", "docs/html/release-history.html", "docs/roadmap.md", "docs/html/roadmap.html"], + product_boundaries=["Conformance contract v2 compares local binary-backed fixture, live, vendor import, bridge, and discovery rows without upgrading weak evidence.", "Vendor-owned import and discovery-only rows cannot satisfy Invart-mediated or enforced claims.", "Strict live validation remains optional unless explicitly required."], + claim_scope="adapter_conformance_claim_gate", + evidence_level="binary_backed_fixture_plus_vendor_import_negative_control", + external_validation="not_run_optional", + next_step="Use the contract for OpenCode, Gemini CLI, Aider, Codex, IDE bridge, and gateway/server-agent adapter tracks.", + ), + RoadmapCapability( + version="v0.9.10", + capability_id="opencode_real_adapter_track", + title="OpenCode real adapter track", + target="Add OpenCode as the second product-specific local adapter track with binary discovery, managed wrapper execution, plugin/MCP config inventory, L5 artifacts, and managed risk blocking.", + status="implemented", + implementation=["src/invart/surfaces/live_adapter.py", "src/invart/surfaces/adapter_profiles.py", "src/invart/surfaces/native.py", "src/invart/benchmarks/releases_v52_v57.py"], + tests=["test_v0910_opencode_profile_inventory_and_managed_wrapper", "test_v0910_opencode_managed_risk_blocks_before_side_effect_and_cli", "v0.9.10-opencode-real-adapter"], + docs=["docs/cli-reference.md", "docs/html/cli-reference.html", "docs/release-history.md", "docs/html/release-history.html", "docs/roadmap.md", "docs/html/roadmap.html"], + product_boundaries=["OpenCode is Invart-mediated only when launched through the managed wrapper.", "Plugin, agent, MCP, and config inventory are pre-runtime evidence and do not by themselves prove mediation or enforcement.", "Local CI uses binary-shaped fixtures; installed OpenCode validation remains strict-live optional."], + claim_scope="local_opencode_managed_wrapper", + evidence_level="binary_backed_fixture_with_plugin_inventory", + external_validation="not_run_optional", + next_step="Extend the managed wrapper path to Gemini CLI and Aider with benign workflow compatibility metrics.", + ), + RoadmapCapability( + version="v0.9.11", + capability_id="terminal_agent_managed_wrappers", + title="Terminal agent managed wrapper track", + target="Validate Gemini CLI and Aider through Invart managed wrappers with binary discovery, config/context inventory, artifact parity, and low approval noise for benign workflows.", + status="implemented", + implementation=["src/invart/surfaces/live_adapter.py", "src/invart/surfaces/native.py", "src/invart/surfaces/adapter_profiles.py", "src/invart/benchmarks/releases_v52_v57.py"], + tests=["test_v0911_gemini_and_aider_managed_wrappers_emit_inventory_and_low_noise", "test_v0911_terminal_agent_cli_and_benchmark_are_registered", "v0.9.11-terminal-agent-managed-wrappers"], + docs=["docs/cli-reference.md", "docs/html/cli-reference.html", "docs/release-history.md", "docs/html/release-history.html", "docs/roadmap.md", "docs/html/roadmap.html"], + product_boundaries=["Gemini CLI and Aider are mediated only when launched through Invart's managed wrapper.", "Config, MCP, repo-map, and context discovery are observed evidence unless paired with runtime mediation.", "The local suite uses binary-shaped fixtures and measures approval noise on benign workflows."], + claim_scope="local_terminal_agent_wrappers", + evidence_level="binary_backed_fixture_with_config_inventory", + external_validation="not_run_optional", + next_step="Integrate Codex while keeping Codex-native sandbox and approval facts separate from Invart-owned enforcement.", + ), ) diff --git a/src/invart/surfaces/adapter_profiles.py b/src/invart/surfaces/adapter_profiles.py index 75d2789..3d3bdba 100644 --- a/src/invart/surfaces/adapter_profiles.py +++ b/src/invart/surfaces/adapter_profiles.py @@ -112,9 +112,9 @@ def to_dict(self) -> dict[str, Any]: execution_modes=["managed_wrapper", "native_event_bridge", "plugin_inventory"], native_surfaces=["plugins", "agents", "mcp", "config"], event_sources=["wrapper_command", "plugin_config", "native_hook_payload"], - coverage_grade="native_event_bridge", - claim_boundary="OpenCode plugin and agent surfaces are mediated only when their hook payloads enter Invart; plugin-only logs are not full runtime enforcement.", - required_artifacts=["ledger", "proof", "coverage", "audit"], + coverage_grade="managed_wrapper_adapter", + claim_boundary="OpenCode can be mediated when launched through Invart's managed wrapper. Plugin, agent, and MCP config inventory is pre-runtime evidence and must not be treated as mediation unless events enter Invart before side effects.", + required_artifacts=["ledger", "proof", "coverage", "audit", "evidence_bundle"], source_urls=["https://opencode.ai/docs/agents/", "https://opencode.ai/docs/plugins/"], binary_candidates=["opencode"], supports_mediation=True, diff --git a/src/invart/surfaces/native.py b/src/invart/surfaces/native.py index aa9fd15..c6be820 100644 --- a/src/invart/surfaces/native.py +++ b/src/invart/surfaces/native.py @@ -41,6 +41,10 @@ "config": [".hermes", "hermes.json"], "mcp": [".hermes/mcp.json"], }, + "aider": { + "config": [".aider.conf.yml", ".aider.conf.yaml", "aider.conf.yml"], + "repo_map": [".aider.tags.cache.v3", ".git"], + }, } diff --git a/tests/test_integrations.py b/tests/test_integrations.py index 50fba8a..16e0ac4 100644 --- a/tests/test_integrations.py +++ b/tests/test_integrations.py @@ -403,11 +403,14 @@ def test_v093_real_agent_conformance_fixture_and_strict_live_modes(tmp_path: Pat binary_overrides={"claude-code": str(fake), "codex": str(fake)}, require_live=True, ) - assert report["schema_version"] == "invart.real_agent_conformance.v0.9.3" + assert report["schema_version"] == "invart.real_agent_conformance.v0.9.9" assert report["status"] == "pass" + assert report["conformance_contract"]["schema_version"] == "invart.adapter_conformance_contract.v0.9.9" + assert report["conformance_contract"]["status"] == "pass" assert report["summary"]["passed_agents"] == 2 assert all(agent["binary"]["status"] == "found" for agent in report["agents"]) assert all(agent["managed_run"]["status"] == "pass" for agent in report["agents"]) + assert all(agent["contract"]["claimable_coverage"] == "managed_wrapper" for agent in report["agents"]) assert Path(report["artifacts"]["report_json"]).exists() assert Path(report["artifacts"]["report_html"]).exists() @@ -431,6 +434,56 @@ def test_v093_real_agent_conformance_fixture_and_strict_live_modes(tmp_path: Pat assert strict_missing["agents"][0]["status"] == "blocked_missing_binary" +def test_v099_conformance_contract_v2_blocks_claim_inflation(tmp_path: Path) -> None: + from invart.evaluation.real_agent_conformance import run_real_agent_conformance, validate_conformance_contract + + fake = tmp_path / "fake-agent" + fake.write_text("#!/usr/bin/env python3\nimport sys\nprint('fixture'); sys.exit(0)\n", encoding="utf-8") + fake.chmod(0o755) + report = run_real_agent_conformance( + out_dir=tmp_path / "v099", + agents=["claude-code", "openclaw"], + binary_overrides={"claude-code": str(fake), "openclaw": str(fake)}, + require_live=True, + ) + by_agent = {row["agent"]: row for row in report["agents"]} + assert by_agent["claude-code"]["contract"]["claimable_coverage"] == "managed_wrapper" + assert by_agent["claude-code"]["contract"]["side_effect_timing"] == "pre_side_effect" + assert by_agent["openclaw"]["contract"]["control_position"] == "vendor_owned_import" + assert by_agent["openclaw"]["contract"]["claimable_coverage"] == "vendor_import" + assert "invart_pre_side_effect_mediation" in by_agent["openclaw"]["contract"]["cannot_claim"] + inflated = dict(by_agent["openclaw"]) + inflated["contract"] = {**inflated["contract"], "claimable_coverage": "managed_wrapper"} + gate = validate_conformance_contract([inflated]) + assert gate["status"] == "fail" + assert any(finding["check_id"] == "claim.vendor_import_inflation" for finding in gate["findings"]) + + +def test_v099_conformance_contract_cli_and_benchmark_are_registered(tmp_path: Path) -> None: + fake = tmp_path / "fake-agent" + fake.write_text("#!/usr/bin/env python3\nimport sys\nprint('fixture'); sys.exit(0)\n", encoding="utf-8") + fake.chmod(0o755) + out = tmp_path / "v099-cli" + assert main([ + "real-agent", + "check", + "--agent", + "claude-code", + "--agent", + "openclaw", + "--binary", + f"claude-code={fake}", + "--binary", + f"openclaw={fake}", + "--require-live", + "--out-dir", + str(out), + ]) == 0 + report = json.loads((out / "real-agent-conformance.json").read_text(encoding="utf-8")) + assert report["conformance_contract"]["claim_gate"]["status"] == "pass" + assert main(["eval", "benchmark", "--suite", "v0.9.9-conformance-contract-v2"]) == 0 + + def test_v093_real_agent_cli_and_benchmark(tmp_path: Path) -> None: fake = tmp_path / "fake-agent" fake.write_text("#!/usr/bin/env python3\nimport sys\nsys.exit(0)\n", encoding="utf-8") @@ -650,7 +703,7 @@ def test_v095_priority_agent_profiles_emit_tracks_and_control_positions() -> Non "codex": ("managed_wrapper", "invart_mediated"), "gemini-cli": ("managed_wrapper", "invart_mediated"), "cursor": ("native_bridge", "bridge_mediated_when_configured"), - "opencode": ("native_bridge", "bridge_mediated_when_configured"), + "opencode": ("managed_wrapper", "invart_mediated"), "openclaw": ("vendor_evidence_import", "vendor_owned_import"), "hermes": ("vendor_evidence_import", "vendor_owned_import"), "github-copilot-cloud-agent": ("cloud_evidence_import", "vendor_owned_import"), @@ -977,6 +1030,168 @@ def test_v098_claude_live_adapter_cli_real_agent_run_and_benchmark(tmp_path: Pat assert main(["eval", "benchmark", "--suite", "v0.9.8-claude-full-live-adapter"]) == 0 +def test_v0910_opencode_profile_inventory_and_managed_wrapper(tmp_path: Path) -> None: + from invart.surfaces.adapter_profiles import get_adapter_profile + from invart.surfaces.live_adapter import run_live_agent_adapter + + profile = get_adapter_profile("opencode") + assert profile["integration_track"] == "managed_wrapper" + assert profile["control_position"] == "invart_mediated" + assert profile["coverage_grade"] == "managed_wrapper_adapter" + + (tmp_path / "opencode.json").write_text(json.dumps({"plugin": ["lint"], "mcp": {"fs": {}}}), encoding="utf-8") + fake = tmp_path / "fake-opencode" + marker = tmp_path / "opencode-ran.txt" + fake.write_text( + "#!/usr/bin/env python3\n" + "import pathlib, sys\n" + "if '--version' in sys.argv:\n" + " raise SystemExit(0)\n" + "if '--write-marker' in sys.argv:\n" + " pathlib.Path(sys.argv[sys.argv.index('--write-marker') + 1]).write_text('ran')\n", + encoding="utf-8", + ) + fake.chmod(0o755) + result = run_live_agent_adapter( + agent="opencode", + target=tmp_path, + out_dir=tmp_path / "opencode-live", + command=[str(fake), "--write-marker", str(marker)], + binary=str(fake), + require_live=True, + policy_mode="advisory", + ) + assert result["schema_version"] == "invart.live_agent_adapter.v0.9.10" + assert result["status"] == "passed" + assert marker.read_text(encoding="utf-8") == "ran" + assert result["live_evidence"]["binary"]["status"] == "found" + assert result["managed_run"]["package"] + assert result["layer_runtime"]["status"] == "pass" + assert result["evidence_workspace"]["status"] == "pass" + opencode_inventory = [item for item in result["native_inventory"]["profiles"] if item["agent"] == "opencode"][0] + assert opencode_inventory["surfaces"]["plugins"]["matches"] + assert opencode_inventory["surfaces"]["mcp"]["matches"] + + +def test_v0910_opencode_managed_risk_blocks_before_side_effect_and_cli(tmp_path: Path) -> None: + from invart.surfaces.live_adapter import run_live_agent_adapter + + fake = tmp_path / "fake-opencode" + marker = tmp_path / "should-not-exist.txt" + fake.write_text( + "#!/usr/bin/env python3\n" + "import pathlib, sys\n" + "if '--version' in sys.argv:\n" + " raise SystemExit(0)\n" + "if '--write-marker' in sys.argv:\n" + " pathlib.Path(sys.argv[sys.argv.index('--write-marker') + 1]).write_text('ran')\n", + encoding="utf-8", + ) + fake.chmod(0o755) + risky = run_live_agent_adapter( + agent="opencode", + target=tmp_path, + out_dir=tmp_path / "opencode-risk", + command=[str(fake), "--write-marker", str(marker), "rm -rf ."], + binary=str(fake), + require_live=True, + policy_mode="managed", + ) + assert risky["status"] == "blocked" + assert risky["returncode"] == 126 + assert marker.exists() is False + + assert main([ + "real-agent", + "run", + "--agent", + "opencode", + "--binary", + str(fake), + "--require-live", + "--target", + str(tmp_path), + "--out-dir", + str(tmp_path / "opencode-cli"), + "--", + str(fake), + "--version", + ]) == 0 + assert main(["eval", "benchmark", "--suite", "v0.9.10-opencode-real-adapter"]) == 0 + + +def test_v0911_gemini_and_aider_managed_wrappers_emit_inventory_and_low_noise(tmp_path: Path) -> None: + from invart.surfaces.live_adapter import run_live_agent_adapter + + (tmp_path / ".gemini").mkdir() + (tmp_path / ".gemini" / "settings.json").write_text(json.dumps({"mcpServers": {"fs": {}}}), encoding="utf-8") + (tmp_path / ".git").mkdir() + (tmp_path / ".aider.conf.yml").write_text("auto-commits: false\n", encoding="utf-8") + fake = tmp_path / "fake-terminal-agent" + fake.write_text( + "#!/usr/bin/env python3\n" + "import pathlib, sys\n" + "if '--version' in sys.argv:\n" + " raise SystemExit(0)\n" + "if '--write-marker' in sys.argv:\n" + " pathlib.Path(sys.argv[sys.argv.index('--write-marker') + 1]).write_text('ran')\n", + encoding="utf-8", + ) + fake.chmod(0o755) + results = {} + for agent in ("gemini-cli", "aider"): + marker = tmp_path / f"{agent}.txt" + results[agent] = run_live_agent_adapter( + agent=agent, + target=tmp_path, + out_dir=tmp_path / agent, + command=[str(fake), "--write-marker", str(marker)], + binary=str(fake), + require_live=True, + policy_mode="advisory", + ) + assert results[agent]["status"] == "passed" + assert marker.read_text(encoding="utf-8") == "ran" + assert results[agent]["managed_run"]["returncode"] == 0 + assert results[agent]["evidence_workspace"]["status"] == "pass" + entries, _warnings = load_ledger_entries(Path(results[agent]["managed_run"]["ledger"])) + approval_effects = [ + entry.decision.get("effect") + for entry in entries + if entry.decision and entry.entry_type == "action" + ] + assert approval_effects.count("require_approval") == 0 + gemini_inventory = [item for item in results["gemini-cli"]["native_inventory"]["profiles"] if item["agent"] == "gemini-cli"][0] + aider_inventory = [item for item in results["aider"]["native_inventory"]["profiles"] if item["agent"] == "aider"][0] + assert gemini_inventory["surfaces"]["mcp"]["matches"] + assert aider_inventory["surfaces"]["config"]["matches"] + assert aider_inventory["surfaces"]["repo_map"]["matches"] + + +def test_v0911_terminal_agent_cli_and_benchmark_are_registered(tmp_path: Path) -> None: + fake = tmp_path / "fake-terminal-agent" + fake.write_text("#!/usr/bin/env python3\nimport sys\nsys.exit(0)\n", encoding="utf-8") + fake.chmod(0o755) + for agent in ("gemini-cli", "aider"): + assert main([ + "real-agent", + "run", + "--agent", + agent, + "--binary", + str(fake), + "--require-live", + "--target", + str(tmp_path), + "--out-dir", + str(tmp_path / f"{agent}-cli"), + "--", + str(fake), + "--version", + ]) == 0 + assert main(["eval", "benchmark", "--suite", "v0.9.11-terminal-agent-managed-wrappers"]) == 0 + + def test_v09_swe_bench_lite_runner_skips_cleanly_without_dependencies(tmp_path: Path) -> None: out = tmp_path / "swebench-report.json" assert main([