Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,25 @@ invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/r

Use `--require-live` when you want missing local agent binaries to fail the run instead of being recorded as blocked evidence. Fixture-backed runs validate the Invart adapter contract; live runs validate the installed product surface.

### Claude Code reference adapter

```bash
invart adapter claude-code \
--target . \
--out-dir .invart/claude-reference \
--hook-events .invart/claude-hooks.jsonl \
--policy-mode managed \
-- <claude-or-harness-command>
```

This reference adapter records Claude-style hook events, mediates the child command, and exports an adapter package containing ledger, proof, replay, path graph, coverage, audit, and evidence manifest. In managed/ci mode, deterministic risky actions pause or block before the child command is launched; advisory mode preserves autonomy and records evidence. Portable subprocess supervision is explicitly marked as degraded process-tree coverage unless native supervision is enabled.

### Evaluation

```bash
invart eval list
invart eval benchmark --suite full-product-readiness
invart eval benchmark --suite v0.9.3-agent-adapter-contract
invart eval benchmark --suite v0.9.4-claude-reference-adapter
invart roadmap status --require-full
```
7 changes: 7 additions & 0 deletions docs/html/cli-reference.html
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,16 @@
<div class="card"><h3>Real agent conformance</h3><pre>invart adapter profile --kind claude-code
invart real-agent check --agent claude-code --out-dir .invart/real-agent
invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html</pre><p>Use <code>--require-live</code> when missing local binaries should fail instead of being recorded as blocked evidence.</p></div>
<div class="card"><h3>Claude Code reference adapter</h3><pre>invart adapter claude-code \
--target . \
--out-dir .invart/claude-reference \
--hook-events .invart/claude-hooks.jsonl \
--policy-mode managed \
-- &lt;claude-or-harness-command&gt;</pre><p>Records Claude-style hook events, mediates the child command, and exports ledger, proof, replay, path graph, coverage, audit, and evidence manifest. Portable subprocess supervision is marked as degraded process-tree coverage unless native supervision is enabled.</p></div>
<div class="card"><h3>Evaluation</h3><pre>invart eval list
invart eval benchmark --suite full-product-readiness
invart eval benchmark --suite v0.9.3-agent-adapter-contract
invart eval benchmark --suite v0.9.4-claude-reference-adapter
invart roadmap status --require-full</pre></div>
</section>
</main>
Expand Down
1 change: 1 addition & 0 deletions docs/html/release-history.html
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ <h2>0.9 Patch Track</h2>
<table>
<tr><th>Version</th><th>Status</th><th>Focus</th></tr>
<tr><td>v0.9.3</td><td>Implemented</td><td>Agent adapter contract registry and fixture-backed real-agent conformance foundation. The live mode can be strict, but missing local agent binaries are not reported as successful live validation.</td></tr>
<tr><td>v0.9.4</td><td>Implemented</td><td>Claude Code reference full adapter: hook events and child command mediation, managed risk pause/block before launch, full L5 evidence package, and truthful degraded process-tree coverage when only portable subprocess supervision is active.</td></tr>
</table>
</section>
<section>
Expand Down
1 change: 1 addition & 0 deletions docs/release-history.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Invart has a long local implementation history. The public docs summarize capabi
| Version | Status | Focus |
| --- | --- | --- |
| v0.9.3 | Implemented | Agent adapter contract registry and fixture-backed real-agent conformance foundation. The live mode can be strict, but missing local agent binaries are not reported as successful live validation. |
| v0.9.4 | Implemented | Claude Code reference full adapter: hook events and child command mediation, managed risk pause/block before launch, full L5 evidence package, and truthful degraded process-tree coverage when only portable subprocess supervision is active. |

## Internal History

Expand Down
3 changes: 2 additions & 1 deletion src/invart/benchmarks/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@
run_product_control_matrix_benchmark,
run_reviewer_ablation_cost_benchmark,
)
from .releases_v52_v57 import run_agent_adapter_contract_benchmark
from .releases_v52_v57 import run_agent_adapter_contract_benchmark, run_claude_reference_adapter_benchmark

BenchmarkRunner = Callable[[], dict[str, Any]]

Expand Down Expand Up @@ -111,6 +111,7 @@ def benchmark_runner_registry() -> dict[str, BenchmarkRunner]:
"v0.50-product-control-matrix": run_product_control_matrix_benchmark,
"v0.51-pre-1.0-research-ready-gate": run_pre_1_0_research_ready_gate_benchmark,
"v0.9.3-agent-adapter-contract": run_agent_adapter_contract_benchmark,
"v0.9.4-claude-reference-adapter": run_claude_reference_adapter_benchmark,
"progressive-external-validation": run_progressive_external_validation_benchmark,
"real-world-agent-risk-demo": run_real_world_risk_benchmark,
"containerized-risk-demo": run_container_risk_demo_benchmark,
Expand Down
74 changes: 73 additions & 1 deletion src/invart/benchmarks/releases_v52_v57.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
from __future__ import annotations

import tempfile
import sys
from pathlib import Path

from .common import _suite_result
from invart.assurance.evidence_bundle import verify_evidence_bundle
from invart.core.ledger import load_ledger_entries
from invart.evaluation.real_agent_conformance import run_real_agent_conformance
from invart.surfaces.claude_adapter import run_claude_code_adapter
from invart.surfaces.adapter_profiles import list_adapter_profiles, validate_adapter_profile_truthfulness


Expand Down Expand Up @@ -38,4 +42,72 @@ def run_agent_adapter_contract_benchmark() -> dict[str, object]:
)


__all__ = ["run_agent_adapter_contract_benchmark"]
def run_claude_reference_adapter_benchmark() -> dict[str, object]:
with tempfile.TemporaryDirectory(prefix="invart_v094_") as tmp:
root = Path(tmp)
hooks = root / "hooks.jsonl"
hooks.write_text(
'{"type":"file_read","path":".env","metadata":{"source":"claude_code_hook"}}\n',
encoding="utf-8",
)
package_run = run_claude_code_adapter(
target=root,
command=[sys.executable, "-c", "pass"],
hook_events=hooks,
out_dir=root / "package",
session_id="ks_v094_benchmark_package",
policy_mode="advisory",
)
managed_marker = root / "managed_should_not_exist.txt"
managed_run = run_claude_code_adapter(
target=root,
command=["sh", "-c", f"touch {managed_marker}; rm -rf ."],
out_dir=root / "managed",
session_id="ks_v094_benchmark_managed",
policy_mode="managed",
)
benign_marker = root / "benign.txt"
benign_run = run_claude_code_adapter(
target=root,
command=[sys.executable, "-c", f"from pathlib import Path; Path({str(benign_marker)!r}).write_text('ok')"],
out_dir=root / "benign",
session_id="ks_v094_benchmark_benign",
policy_mode="advisory",
)
entries, _warnings = load_ledger_entries(Path(package_run["ledger"]))
mediation_surfaces = {
entry.result.get("request", {}).get("surface")
for entry in entries
if entry.entry_type == "mediation" and isinstance(entry.result, dict)
}
verification = verify_evidence_bundle(Path(package_run["adapter_package"]["manifest_path"]))
checks = {
"hook_event_mediated": "file" in mediation_surfaces,
"adapter_package_verified": verification.get("status") == "pass",
"l5_artifacts_present": {
"ledger",
"proof",
"replay",
"path_graph_json",
"path_graph_html",
"coverage",
"audit_html",
}.issubset(set(package_run["adapter_package"]["artifacts"])),
"managed_risk_stopped_before_side_effect": managed_run.get("returncode") == 126 and not managed_marker.exists(),
"managed_status_is_explicit": managed_run.get("status") in {"blocked", "requires_approval"},
"benign_advisory_kept_autonomy": benign_run.get("status") == "passed" and benign_marker.exists(),
"supervision_truthful_degraded": package_run.get("supervision", {}).get("coverage_grade") == "mediated_without_process_tree",
}
return _suite_result(
"v0.9.4-claude-reference-adapter",
checks,
artifacts={
"package_manifest": package_run["adapter_package"]["manifest_path"],
"package_ledger": package_run["ledger"],
"managed_ledger": managed_run["ledger"],
"benign_ledger": benign_run["ledger"],
},
)


__all__ = ["run_agent_adapter_contract_benchmark", "run_claude_reference_adapter_benchmark"]
1 change: 1 addition & 0 deletions src/invart/commands/integrations.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ def handle_adapter(args: argparse.Namespace) -> int:
session_id=args.session_id,
create_preflight=not args.no_preflight,
enforcement=args.enforcement,
policy_mode=args.policy_mode,
)
print(json.dumps({"claude_code_adapter": result}, ensure_ascii=False, indent=2, sort_keys=True))
return int(result.get("returncode", 1))
Expand Down
1 change: 1 addition & 0 deletions src/invart/commands/parser_integrations.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ def register_integration_commands(subparsers: argparse._SubParsersAction[argpars
adapter_claude.add_argument("--hook-events", default=None)
adapter_claude.add_argument("--session-id", default=None)
adapter_claude.add_argument("--no-preflight", action="store_true")
adapter_claude.add_argument("--policy-mode", choices=("audit", "advisory", "managed", "ci"), default="advisory")
adapter_claude.add_argument("--enforcement", choices=("off", "file-write"), default="off")
adapter_claude.add_argument("cmd", nargs=argparse.REMAINDER)

Expand Down
1 change: 1 addition & 0 deletions src/invart/evaluation/benchmark_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
{"suite": "v0.50-product-control-matrix", "version": "v0.50", "category": "product-boundary", "optional_heavy": False, "claim_scope": "product_comparison", "evidence_level": "documented_capability_matrix"},
{"suite": "v0.51-pre-1.0-research-ready-gate", "version": "v0.51", "category": "release", "optional_heavy": False, "claim_scope": "research_gate", "evidence_level": "local_research_artifact_bundle"},
{"suite": "v0.9.3-agent-adapter-contract", "version": "v0.9.3", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_agent_adapter_contract", "evidence_level": "fixture_backed_conformance"},
{"suite": "v0.9.4-claude-reference-adapter", "version": "v0.9.4", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_claude_reference_adapter", "evidence_level": "local_runtime_fixture"},
{"suite": "progressive-external-validation", "version": "pre-release", "category": "external-validation", "optional_heavy": False, "claim_scope": "progressive_external_validation", "evidence_level": "external_progressive_sample"},
{"suite": "real-world-agent-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "public_source_mapping", "evidence_level": "public_source_seed_plus_local_demo"},
{"suite": "containerized-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "containerized_local_demo", "evidence_level": "per_case_container_artifact_bundle"},
Expand Down
1 change: 1 addition & 0 deletions src/invart/evaluation/release_candidate.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
"v0.50-product-control-matrix",
"v0.51-pre-1.0-research-ready-gate",
"v0.9.3-agent-adapter-contract",
"v0.9.4-claude-reference-adapter",
"progressive-external-validation",
"pre-v1-control-plane",
)
Expand Down
15 changes: 15 additions & 0 deletions src/invart/evaluation/roadmap.py
Original file line number Diff line number Diff line change
Expand Up @@ -782,6 +782,21 @@ def internal_docs_for_capability(docs: list[str]) -> list[str]:
external_validation="not_run_optional",
next_step="Harden Claude Code as the first full reference adapter and attach live conformance traces for each priority product.",
),
RoadmapCapability(
version="v0.9.4",
capability_id="claude_reference_full_adapter",
title="Claude Code reference full adapter",
target="Run Claude Code style wrapper and hook events through unified mediation, managed pause/block behavior, truthful degraded supervision evidence, and a full L5 artifact package.",
status="implemented",
implementation=["src/invart/surfaces/claude_adapter.py", "src/invart/control/mediation.py", "src/invart/assurance/evidence_bundle.py", "src/invart/commands/parser_integrations.py", "src/invart/commands/integrations.py"],
tests=["test_v094_claude_adapter_exports_full_package_and_mediates_hooks", "test_v094_claude_adapter_managed_risk_pauses_before_side_effect", "test_v094_claude_adapter_cli_and_benchmark_are_registered", "v0.9.4-claude-reference-adapter"],
docs=["docs/cli-reference.md", "docs/html/cli-reference.html", "docs/release-history.md", "docs/html/release-history.html"],
product_boundaries=["Local fixture-backed validation proves the Claude reference adapter contract, artifact package, and managed pause/block behavior; it is not a claim that every live Claude Code installation has been exercised.", "The reference adapter uses a portable subprocess wrapper plus Claude hook JSONL ingestion; process-tree coverage is explicitly degraded unless native supervision is enabled.", "Managed mode pauses or blocks risky actions before launching the child command, while advisory mode preserves autonomy and records evidence."],
claim_scope="local_claude_reference_adapter",
evidence_level="local_runtime_fixture",
external_validation="not_run_optional",
next_step="Add priority agent tracks and product-specific adapter coverage claims for Codex, Gemini CLI, Cursor, OpenClaw, Hermes, OpenCode, and related agent products.",
),
)


Expand Down
Loading
Loading