diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 7da405a..eff420e 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -23,8 +23,11 @@ invart run --target . --agent codex --goal "..." -- ```bash invart runtime analyze-event --event '{"type":"shell","command":"rm -rf ."}' invart runtime shell --session demo --ledger .invart/demo.jsonl -- +invart runtime layers --ledger .invart/demo.jsonl --out-dir .invart/layers ``` +`runtime layers` exports a L1-L5 operation workflow for an existing ledger. It writes JSON and HTML that link proof, replay, path graph, coverage, audit, and evidence manifest artifacts. + ### Proof and gate ```bash @@ -44,11 +47,14 @@ invart audit report --ledger ledger.jsonl --out-dir .invart/audit ```bash invart adapter profile --kind claude-code +invart adapter profiles +invart adapter profiles --track managed_wrapper invart real-agent check --agent claude-code --out-dir .invart/real-agent invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html ``` Use `--require-live` when you want missing local agent binaries to fail the run instead of being recorded as blocked evidence. Fixture-backed runs validate the Invart adapter contract; live runs validate the installed product surface. +The plural `adapter profiles` command lists priority agent tracks: reference full adapter, managed wrapper, native bridge, vendor/cloud evidence import, and framework trace import. Vendor import tracks are audit evidence, not Invart mediation. ### Claude Code reference adapter @@ -70,5 +76,7 @@ invart eval list invart eval benchmark --suite full-product-readiness invart eval benchmark --suite v0.9.3-agent-adapter-contract invart eval benchmark --suite v0.9.4-claude-reference-adapter +invart eval benchmark --suite v0.9.5-priority-agent-tracks +invart eval benchmark --suite v0.9.6-layer-runtime-workflow invart roadmap status --require-full ``` diff --git a/docs/html/cli-reference.html b/docs/html/cli-reference.html index 9749ece..2637491 100644 --- a/docs/html/cli-reference.html +++ b/docs/html/cli-reference.html @@ -9,15 +9,18 @@

Managed session

invart session start --target . --agent codex --goal "..."
 invart run --target . --agent codex --goal "..." -- <command>

Runtime event

invart runtime analyze-event --event '{"type":"shell","command":"rm -rf ."}'
-invart runtime shell --session demo --ledger .invart/demo.jsonl -- <command>
+invart runtime shell --session demo --ledger .invart/demo.jsonl -- <command> +invart runtime layers --ledger .invart/demo.jsonl --out-dir .invart/layers

runtime layers exports a L1-L5 operation workflow with proof, replay, path graph, coverage, audit, and evidence manifest links.

Proof and gate

invart proof export --ledger ledger.jsonl --out proof.json
 invart proof verify --proof proof.json --ledger ledger.jsonl
 invart gate verify --proof proof.json --ledger ledger.jsonl --mode ci

Replay and audit

invart replay export --ledger ledger.jsonl --out replay.html
 invart audit report --ledger ledger.jsonl --out-dir .invart/audit

Real agent conformance

invart adapter profile --kind claude-code
+invart adapter profiles
+invart adapter profiles --track managed_wrapper
 invart real-agent check --agent claude-code --out-dir .invart/real-agent
-invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html

Use --require-live when missing local binaries should fail instead of being recorded as blocked evidence.

+invart real-agent report --run-dir .invart/real-agent --out .invart/real-agent/report.html

Use --require-live when missing local binaries should fail instead of being recorded as blocked evidence. adapter profiles lists priority tracks and keeps vendor/cloud import separate from Invart-mediated control.

Claude Code reference adapter

invart adapter claude-code \
   --target . \
   --out-dir .invart/claude-reference \
@@ -28,6 +31,8 @@
 invart eval benchmark --suite full-product-readiness
 invart eval benchmark --suite v0.9.3-agent-adapter-contract
 invart eval benchmark --suite v0.9.4-claude-reference-adapter
+invart eval benchmark --suite v0.9.5-priority-agent-tracks
+invart eval benchmark --suite v0.9.6-layer-runtime-workflow
 invart roadmap status --require-full
diff --git a/docs/html/release-history.html b/docs/html/release-history.html index 2222d51..77af34f 100644 --- a/docs/html/release-history.html +++ b/docs/html/release-history.html @@ -32,6 +32,8 @@

0.9 Patch Track

VersionStatusFocus v0.9.3ImplementedAgent adapter contract registry and fixture-backed real-agent conformance foundation. The live mode can be strict, but missing local agent binaries are not reported as successful live validation. v0.9.4ImplementedClaude Code reference full adapter: hook events and child command mediation, managed risk pause/block before launch, full L5 evidence package, and truthful degraded process-tree coverage when only portable subprocess supervision is active. + v0.9.5ImplementedPriority agent tracks for Claude Code, Codex, Gemini CLI, Cursor, OpenCode, OpenClaw, Hermes, cloud agents, and frameworks, with profile-derived product matrix rows and explicit vendor/import versus Invart-mediated control positions. + v0.9.6ImplementedLedger-derived L1-L5 runtime operation workflow via runtime layers, exporting a stage x layer matrix, layer timeline, operator command guide, and linked proof/replay/path graph/coverage/audit/evidence artifacts.
diff --git a/docs/html/runtime-effect-demo.html b/docs/html/runtime-effect-demo.html index 69d624a..bfef672 100644 --- a/docs/html/runtime-effect-demo.html +++ b/docs/html/runtime-effect-demo.html @@ -25,6 +25,11 @@

Run

--out-dir .invart/pre-1-demo-check scripts/container-demo.sh all .invart/container-risk-demo +

For any existing Invart ledger, generate a layer operation report:

+ 
PYTHONPATH=src python -m invart.cli runtime layers \
+  --ledger .invart/session.jsonl \
+  --out-dir .invart/layer-workflow
+

Open .invart/layer-workflow/layer-runtime-workflow.html. The report is derived from the ledger and links proof, replay, path graph, coverage, audit, and evidence manifest artifacts.

Runtime Effect Matrix

@@ -58,6 +63,18 @@

Artifact Map

path-graph.htmlL3 causal chainTracing risky paths such as secret read to network sink. coverage.htmlL4/L5 truthfulnessChecking observed, mediated, enforced, and fail-open claims. audit-report.htmlL5 reviewer reportPresenting the enterprise security review view. + layer-runtime-workflow.htmlL1-L5 operation workflowOperating a single ledger through before/during/after runtime questions. + +
+
+

Layer Operation Flow

+ + + + + + +
LayerPractical commandWhat the user gets
L1 Execution Surfaceinvart pre-runtime --target . --saveSurface inventory and unmanaged coverage gaps.
L2 Runtime Fact Modelinvart runtime record-event --ledger ledger.jsonl --event '{...}'Normalized invocation, resource, taint, identity, and outcome facts.
L3 Decision Planeinvart policy check-path --ledger ledger.jsonl --out path-policy.jsonDeterministic and path-aware reasons for allow, approval, or deny.
L4 Mediation Planeinvart mediation inspect --ledger ledger.jsonlPause, block, fail-open, approval, and mediation outcome state.
L5 Evidence Planeinvart runtime layers --ledger ledger.jsonl --out-dir .invart/layersA reviewable stage x layer matrix with proof/replay/graph/coverage/audit links.
diff --git a/docs/release-history.md b/docs/release-history.md index 8e33184..b71885e 100644 --- a/docs/release-history.md +++ b/docs/release-history.md @@ -32,6 +32,8 @@ Invart has a long local implementation history. The public docs summarize capabi | --- | --- | --- | | v0.9.3 | Implemented | Agent adapter contract registry and fixture-backed real-agent conformance foundation. The live mode can be strict, but missing local agent binaries are not reported as successful live validation. | | v0.9.4 | Implemented | Claude Code reference full adapter: hook events and child command mediation, managed risk pause/block before launch, full L5 evidence package, and truthful degraded process-tree coverage when only portable subprocess supervision is active. | +| v0.9.5 | Implemented | Priority agent tracks for Claude Code, Codex, Gemini CLI, Cursor, OpenCode, OpenClaw, Hermes, cloud agents, and frameworks, with profile-derived product matrix rows and explicit vendor/import versus Invart-mediated control positions. | +| v0.9.6 | Implemented | Ledger-derived L1-L5 runtime operation workflow via `runtime layers`, exporting a stage x layer matrix, layer timeline, operator command guide, and linked proof/replay/path graph/coverage/audit/evidence artifacts. | ## Internal History diff --git a/docs/runtime-effect-demo.md b/docs/runtime-effect-demo.md index 904ef65..d074f75 100644 --- a/docs/runtime-effect-demo.md +++ b/docs/runtime-effect-demo.md @@ -19,6 +19,16 @@ For isolated per-risk-case runs: scripts/container-demo.sh all .invart/container-risk-demo ``` +For any existing Invart ledger: + +```bash +PYTHONPATH=src python -m invart.cli runtime layers \ + --ledger .invart/session.jsonl \ + --out-dir .invart/layer-workflow +``` + +Open `.invart/layer-workflow/layer-runtime-workflow.html`. This report is derived from the ledger and links the proof, replay, path graph, coverage, audit, and evidence manifest produced for that run. + ## What To Look For | View | What it proves | Main artifact | @@ -28,6 +38,7 @@ scripts/container-demo.sh all .invart/container-risk-demo | Path Graph | Risk chains such as secret read to external network sink can be traced. | `path-graph.html` | | Coverage Report | Observed, mediated, and enforced are reported as different claims. | `coverage.html` | | Audit Report | A reviewer can answer who, what, why, policy, approval, outcome, and coverage. | `audit-report.html` | +| Layer Runtime Workflow | A command-level way to operate L1-L5 for a single ledger. | `layer-runtime-workflow.html` | ## Three Runtime Stages @@ -47,6 +58,16 @@ scripts/container-demo.sh all .invart/container-risk-demo | L4 Mediation Plane | Shows allow, audit, require approval, deny, enforced block, and fail-open alert semantics. | | L5 Evidence Plane | Shows proof, replay, graph, coverage, audit, evidence bundle, and release gate outputs. | +## Layer Operation Flow + +| Layer | Practical command | What the user gets | +| --- | --- | --- | +| L1 Execution Surface | `invart pre-runtime --target . --save` | Surface inventory and unmanaged coverage gaps. | +| L2 Runtime Fact Model | `invart runtime record-event --ledger ledger.jsonl --event '{...}'` | Normalized invocation, resource, taint, identity, and outcome facts. | +| L3 Decision Plane | `invart policy check-path --ledger ledger.jsonl --out path-policy.json` | Deterministic and path-aware reasons for allow, approval, or deny. | +| L4 Mediation Plane | `invart mediation inspect --ledger ledger.jsonl` | Pause, block, fail-open, approval, and mediation outcome state. | +| L5 Evidence Plane | `invart runtime layers --ledger ledger.jsonl --out-dir .invart/layers` | A reviewable stage x layer matrix with proof/replay/graph/coverage/audit links. | + ## Boundaries The container demo uses safe equivalent trajectories. It does not install hostile packages or replay private incidents. diff --git a/src/invart/assurance/layer_runtime.py b/src/invart/assurance/layer_runtime.py new file mode 100644 index 0000000..8086b86 --- /dev/null +++ b/src/invart/assurance/layer_runtime.py @@ -0,0 +1,206 @@ +from __future__ import annotations + +import html +from pathlib import Path +from typing import Any + +from invart.assurance.evidence_bundle import export_evidence_bundle +from invart.core.artifacts import relative_href, write_html_artifact, write_json_artifact +from invart.core.ledger import load_ledger_entries +from invart.core.models import LedgerEntry, utc_now + + +SCHEMA_VERSION = "invart.layer_runtime_workflow.v0.9.6" +LAYERS = ( + ("L1", "Execution Surface"), + ("L2", "Runtime Fact Model"), + ("L3", "Decision Plane"), + ("L4", "Mediation Plane"), + ("L5", "Evidence Plane"), +) +STAGES = ("before-runtime", "during-runtime", "after-runtime") + + +def export_layer_runtime_workflow(ledger_path: Path, out_dir: Path, *, profile: dict[str, Any] | None = None) -> dict[str, Any]: + ledger_path = ledger_path.expanduser().resolve() + out_dir = out_dir.expanduser().resolve() + out_dir.mkdir(parents=True, exist_ok=True) + entries, warnings = load_ledger_entries(ledger_path) + bundle = export_evidence_bundle(ledger_path, out_dir / "evidence", profile=profile or {"name": "layer-runtime-workflow", "mode": "managed"}) + artifacts = { + **bundle["artifacts"], + "evidence_manifest": bundle["manifest_path"], + "workflow_json": str(out_dir / "layer-runtime-workflow.json"), + "workflow_html": str(out_dir / "layer-runtime-workflow.html"), + } + matrix = _runtime_effect_matrix(artifacts) + timeline = _layer_timeline(entries, artifacts) + operations = _operation_guide(artifacts) + report = { + "schema_version": SCHEMA_VERSION, + "status": "pass", + "generated_at": utc_now(), + "ledger": str(ledger_path), + "warnings": warnings, + "runtime_effect_matrix": matrix, + "layer_timeline": timeline, + "operations": operations, + "artifacts": artifacts, + "summary": { + "entries": len(entries), + "actions": sum(1 for entry in entries if entry.entry_type == "action"), + "decisions": sum(1 for entry in entries if entry.decision), + "outcomes": sum(1 for entry in entries if entry.outcome), + "layers": len(LAYERS), + "stages": len(STAGES), + }, + "claim_boundary": "Layer workflow is derived from the ledger and generated artifacts; it does not add facts outside the ledger fact source.", + } + write_json_artifact(Path(artifacts["workflow_json"]), report) + write_html_artifact(Path(artifacts["workflow_html"]), _render_workflow_html(report)) + return report + + +def _runtime_effect_matrix(artifacts: dict[str, str]) -> list[dict[str, Any]]: + rows = { + ("before-runtime", "L1"): ("Execution Surface", "Identify launchers, hooks, wrapper, MCP, skill, command, file, and network surfaces.", artifacts["path_graph_html"]), + ("before-runtime", "L2"): ("Runtime Fact Model", "Confirm the ledger path, session facts, identity/grant facts, resources, and credential boundary.", artifacts["ledger"]), + ("before-runtime", "L3"): ("Decision Plane", "Inspect policy profile and path-aware rules before interpreting actions.", artifacts["path_policy"]), + ("before-runtime", "L4"): ("Mediation Plane", "Check whether the run was only observed, mediated, or enforced before side effects.", artifacts["coverage"]), + ("before-runtime", "L5"): ("Evidence Plane", "Record the evidence bundle manifest and hash contract.", artifacts["evidence_manifest"]), + ("during-runtime", "L1"): ("Execution Surface", "Map each action to its surface: shell, file, network, tool, skill, MCP, hook, or wrapper.", artifacts["path_graph_json"]), + ("during-runtime", "L2"): ("Runtime Fact Model", "Read invocations, taint state, resources, decisions, approvals, and outcomes as ledger entries.", artifacts["proof"]), + ("during-runtime", "L3"): ("Decision Plane", "Explain why policy allowed, requested approval, denied, or marked a path critical.", artifacts["path_policy"]), + ("during-runtime", "L4"): ("Mediation Plane", "Inspect allow/audit/paused/blocked/enforced/fail-open outcomes without mixing coverage grades.", artifacts["replay"]), + ("during-runtime", "L5"): ("Evidence Plane", "Use replay and coverage artifacts to show what Invart observed or controlled during execution.", artifacts["coverage"]), + ("after-runtime", "L1"): ("Execution Surface", "Review uncovered or degraded execution surfaces as follow-up control gaps.", artifacts["coverage"]), + ("after-runtime", "L2"): ("Runtime Fact Model", "Keep the ledger as the fact source for reconstruction and dispute resolution.", artifacts["ledger"]), + ("after-runtime", "L3"): ("Decision Plane", "Use path graph and policy report to explain why the run was safe, paused, or blocked.", artifacts["path_graph_html"]), + ("after-runtime", "L4"): ("Mediation Plane", "Verify approvals, outcomes, and gate findings after the run.", artifacts["audit_json"]), + ("after-runtime", "L5"): ("Evidence Plane", "Share the portable proof, audit report, and manifest with reviewers.", artifacts["audit_html"]), + } + return [ + { + "stage": stage, + "layer": layer, + "layer_name": layer_name, + "effect": effect, + "artifact": artifact, + } + for stage in STAGES + for layer, _label in LAYERS + for layer_name, effect, artifact in [rows[(stage, layer)]] + ] + + +def _layer_timeline(entries: list[LedgerEntry], artifacts: dict[str, str]) -> list[dict[str, Any]]: + actions = [entry for entry in entries if entry.entry_type == "action"] + decisions = [entry for entry in entries if entry.decision] + approvals = [entry for entry in entries if entry.entry_type == "approval"] + outcomes = [entry for entry in entries if entry.outcome] + sessions = [entry for entry in entries if entry.entry_type == "session"] + first_action = _first_action_summary(actions) + return [ + { + "stage": "before-runtime", + "layer": "L1", + "layer_name": "Execution Surface", + "operator_question": "Which runtime surface will the agent use?", + "invart_observation": f"{len(actions)} action surface(s) are reconstructable from the ledger.", + "artifact": artifacts["path_graph_html"], + }, + { + "stage": "during-runtime", + "layer": "L2", + "layer_name": "Runtime Fact Model", + "operator_question": "What happened, and what resource did it touch?", + "invart_observation": first_action, + "artifact": artifacts["proof"], + }, + { + "stage": "during-runtime", + "layer": "L3", + "layer_name": "Decision Plane", + "operator_question": "Why was it allowed, paused, or blocked?", + "invart_observation": f"{len(decisions)} decision record(s) and {len(approvals)} approval record(s) are available.", + "artifact": artifacts["path_policy"], + }, + { + "stage": "during-runtime", + "layer": "L4", + "layer_name": "Mediation Plane", + "operator_question": "Did Invart only observe, mediate, or enforce?", + "invart_observation": f"{len(outcomes)} outcome record(s) exist; coverage report preserves observed/mediated/enforced distinctions.", + "artifact": artifacts["coverage"], + }, + { + "stage": "after-runtime", + "layer": "L5", + "layer_name": "Evidence Plane", + "operator_question": "What should a reviewer open after the run?", + "invart_observation": f"{len(sessions)} session boundary record(s) plus proof, replay, graph, audit, and manifest are exported.", + "artifact": artifacts["audit_html"], + }, + ] + + +def _operation_guide(artifacts: dict[str, str]) -> list[dict[str, str]]: + return [ + {"layer": "L1", "command": "invart pre-runtime --target . --save", "artifact": artifacts["path_graph_html"], "purpose": "Inventory surfaces before runtime."}, + {"layer": "L2", "command": "invart runtime record-event --ledger ledger.jsonl --event '{...}'", "artifact": artifacts["ledger"], "purpose": "Persist normalized runtime facts."}, + {"layer": "L3", "command": "invart policy check-path --ledger ledger.jsonl --out path-policy.json", "artifact": artifacts["path_policy"], "purpose": "Explain path-aware policy decisions."}, + {"layer": "L4", "command": "invart mediation inspect --ledger ledger.jsonl", "artifact": artifacts["replay"], "purpose": "Inspect mediation outcomes and approval state."}, + {"layer": "L5", "command": "invart runtime layers --ledger ledger.jsonl --out-dir .invart/layers", "artifact": artifacts["evidence_manifest"], "purpose": "Export reviewable evidence package."}, + ] + + +def _first_action_summary(actions: list[LedgerEntry]) -> str: + if not actions: + return "No action entries were found; only session/evidence boundaries can be reported." + event = actions[0].event or {} + touched = event.get("command") or event.get("path") or event.get("url") or event.get("tool") or event.get("skill") or event.get("action_type") + return f"First action {event.get('invocation_id') or event.get('event_id')} touched {touched}." + + +def _render_workflow_html(report: dict[str, Any]) -> str: + base = Path(report["artifacts"]["workflow_html"]).parent + matrix_rows = _render_matrix_rows(report["runtime_effect_matrix"], base) + timeline_rows = "".join( + "" + f"{html.escape(item['layer'])} {html.escape(item['layer_name'])}" + f"{html.escape(item['stage'])}" + f"{html.escape(item['operator_question'])}" + f"{html.escape(item['invart_observation'])}" + f"{html.escape(Path(item['artifact']).name)}" + "" + for item in report["layer_timeline"] + ) + operation_rows = "".join( + "" + f"{html.escape(item['layer'])}" + f"{html.escape(item['command'])}" + f"{html.escape(item['purpose'])}" + f"{html.escape(Path(item['artifact']).name)}" + "" + for item in report["operations"] + ) + return f"""Invart Layer Runtime Workflow

Invart Layer Runtime Workflow

Runtime operation view for before-runtime, during-runtime, after-runtime across L1-L5.

Runtime Effect Matrix

Observed, mediated, and enforced are not interchangeable.

{matrix_rows}
LayerBefore runtimeDuring runtimeAfter runtime

Layer Timeline

{timeline_rows}
LayerStageQuestionInvart observationArtifact

Operator Commands

{operation_rows}
LayerCommandPurposeArtifact

Claim Boundary

{html.escape(report['claim_boundary'])}

""" + + +def _render_matrix_rows(matrix: list[dict[str, Any]], base: Path) -> str: + by_layer: dict[str, dict[str, dict[str, Any]]] = {} + for item in matrix: + by_layer.setdefault(item["layer"], {})[item["stage"]] = item + rows = [] + for layer, layer_name in LAYERS: + cells = [] + for stage in STAGES: + item = by_layer[layer][stage] + cells.append( + f"{html.escape(item['effect'])}
{html.escape(Path(item['artifact']).name)}" + ) + rows.append(f"{html.escape(layer)} {html.escape(layer_name)}{''.join(cells)}") + return "".join(rows) + + +__all__ = ["export_layer_runtime_workflow"] diff --git a/src/invart/benchmarks/registry.py b/src/invart/benchmarks/registry.py index 3488436..52e63be 100644 --- a/src/invart/benchmarks/registry.py +++ b/src/invart/benchmarks/registry.py @@ -55,7 +55,7 @@ run_product_control_matrix_benchmark, run_reviewer_ablation_cost_benchmark, ) -from .releases_v52_v57 import run_agent_adapter_contract_benchmark, run_claude_reference_adapter_benchmark +from .releases_v52_v57 import run_agent_adapter_contract_benchmark, run_claude_reference_adapter_benchmark, run_layer_runtime_workflow_benchmark, run_priority_agent_tracks_benchmark BenchmarkRunner = Callable[[], dict[str, Any]] @@ -112,6 +112,8 @@ def benchmark_runner_registry() -> dict[str, BenchmarkRunner]: "v0.51-pre-1.0-research-ready-gate": run_pre_1_0_research_ready_gate_benchmark, "v0.9.3-agent-adapter-contract": run_agent_adapter_contract_benchmark, "v0.9.4-claude-reference-adapter": run_claude_reference_adapter_benchmark, + "v0.9.5-priority-agent-tracks": run_priority_agent_tracks_benchmark, + "v0.9.6-layer-runtime-workflow": run_layer_runtime_workflow_benchmark, "progressive-external-validation": run_progressive_external_validation_benchmark, "real-world-agent-risk-demo": run_real_world_risk_benchmark, "containerized-risk-demo": run_container_risk_demo_benchmark, diff --git a/src/invart/benchmarks/releases_v52_v57.py b/src/invart/benchmarks/releases_v52_v57.py index 49b793c..ebec099 100644 --- a/src/invart/benchmarks/releases_v52_v57.py +++ b/src/invart/benchmarks/releases_v52_v57.py @@ -6,10 +6,15 @@ from .common import _suite_result from invart.assurance.evidence_bundle import verify_evidence_bundle +from invart.assurance.layer_runtime import export_layer_runtime_workflow from invart.core.ledger import load_ledger_entries +from invart.core.models import RuntimeEvent +from invart.control.runtime import close_session, record_action, start_session +from invart.evaluation.product_control_matrix import run_product_control_matrix from invart.evaluation.real_agent_conformance import run_real_agent_conformance +from invart.surfaces.adapter import run_adapter_command from invart.surfaces.claude_adapter import run_claude_code_adapter -from invart.surfaces.adapter_profiles import list_adapter_profiles, validate_adapter_profile_truthfulness +from invart.surfaces.adapter_profiles import adapter_track_matrix, list_adapter_profiles, validate_adapter_profile_truthfulness def run_agent_adapter_contract_benchmark() -> dict[str, object]: @@ -110,4 +115,92 @@ def run_claude_reference_adapter_benchmark() -> dict[str, object]: ) -__all__ = ["run_agent_adapter_contract_benchmark", "run_claude_reference_adapter_benchmark"] +def run_priority_agent_tracks_benchmark() -> dict[str, object]: + with tempfile.TemporaryDirectory(prefix="invart_v095_") as tmp: + root = Path(tmp) + profiles = list_adapter_profiles() + by_agent = {profile["agent_id"]: profile for profile in profiles} + validation = validate_adapter_profile_truthfulness(profiles) + track_matrix = adapter_track_matrix() + claude = run_claude_code_adapter( + target=root, + command=[sys.executable, "-c", "pass"], + out_dir=root / "claude", + session_id="ks_v095_benchmark_claude", + policy_mode="advisory", + ) + codex = run_adapter_command( + target=root, + command=[sys.executable, "-c", "pass"], + agent="codex", + goal="v0.9.5 priority track benchmark", + session_id="ks_v095_benchmark_codex", + out_dir=root / "codex", + capabilities="audit", + gate_mode="audit", + create_preflight=False, + ) + matrix = run_product_control_matrix(out_dir=root / "matrix") + profile_rows = {row["agent_id"]: row for row in matrix["rows"] if row.get("source_kind") == "invart_adapter_profile"} + checks = { + "profiles_validate": validation.get("status") == "pass", + "track_matrix_passes": track_matrix.get("status") == "pass", + "priority_agents_have_tracks": all( + by_agent[agent].get("integration_track") and by_agent[agent].get("control_position") + for agent in ["claude-code", "codex", "gemini-cli", "cursor", "opencode", "openclaw", "hermes"] + ), + "vendor_import_not_mediated": all( + profile.get("supports_mediation") is False and profile.get("control_position") == "vendor_owned_import" + for profile in profiles + if profile.get("integration_track") in {"vendor_evidence_import", "cloud_evidence_import", "framework_trace_import"} + ), + "claude_fixture_package_verifies": verify_evidence_bundle(Path(claude["adapter_package"]["manifest_path"])).get("status") == "pass", + "codex_wrapper_package_exists": codex.package is not None and Path(codex.package).exists(), + "product_matrix_uses_profile_rows": matrix.get("checks", {}).get("profile_rows_match_track_vocabulary") is True and "claude-code" in profile_rows, + "profile_matrix_agrees_with_coverage_vocabulary": profile_rows.get("github-copilot-cloud-agent", {}).get("coverage_grade") == "vendor_owned", + } + return _suite_result( + "v0.9.5-priority-agent-tracks", + checks, + artifacts={ + "claude_package": claude["adapter_package"]["manifest_path"], + "codex_package": codex.package, + "product_matrix": matrix["artifacts"]["matrix_json"], + }, + ) + + +def run_layer_runtime_workflow_benchmark() -> dict[str, object]: + with tempfile.TemporaryDirectory(prefix="invart_v096_") as tmp: + root = Path(tmp) + ledger = root / "ledger.jsonl" + session = start_session(root, ledger, agent="claude-code", goal="v0.9.6 layer runtime benchmark", create_preflight=False) + record_action(RuntimeEvent(type="file_read", session_id=session.session_id, path=str(root / ".env"), metadata={"coverage_layer": "native_hook"}), ledger) + record_action(RuntimeEvent(type="network", session_id=session.session_id, url="https://example.com/upload", metadata={"coverage_layer": "native_hook"}), ledger) + close_session(ledger) + workflow = export_layer_runtime_workflow(ledger, root / "layers") + checks = { + "workflow_passes": workflow.get("status") == "pass", + "matrix_has_all_stages": {item["stage"] for item in workflow["runtime_effect_matrix"]} == {"before-runtime", "during-runtime", "after-runtime"}, + "matrix_has_all_layers": {item["layer"] for item in workflow["runtime_effect_matrix"]} == {"L1", "L2", "L3", "L4", "L5"}, + "timeline_has_all_layers": {item["layer"] for item in workflow["layer_timeline"]} == {"L1", "L2", "L3", "L4", "L5"}, + "l5_artifacts_exist": all(Path(workflow["artifacts"][key]).exists() for key in ["proof", "replay", "path_graph_json", "path_graph_html", "coverage", "audit_html", "evidence_manifest", "workflow_json", "workflow_html"]), + "operation_guide_has_cli": any("runtime layers" in item["command"] for item in workflow["operations"]), + } + return _suite_result( + "v0.9.6-layer-runtime-workflow", + checks, + artifacts={ + "workflow_json": workflow["artifacts"]["workflow_json"], + "workflow_html": workflow["artifacts"]["workflow_html"], + "evidence_manifest": workflow["artifacts"]["evidence_manifest"], + }, + ) + + +__all__ = [ + "run_agent_adapter_contract_benchmark", + "run_claude_reference_adapter_benchmark", + "run_priority_agent_tracks_benchmark", + "run_layer_runtime_workflow_benchmark", +] diff --git a/src/invart/commands/foundation.py b/src/invart/commands/foundation.py index 57fff7f..0f6452c 100644 --- a/src/invart/commands/foundation.py +++ b/src/invart/commands/foundation.py @@ -8,6 +8,7 @@ from pathlib import Path from invart.assurance.postruntime import export_proof_report, summarize_session, verify_proof_report +from invart.assurance.layer_runtime import export_layer_runtime_workflow from invart.control.runtime import ( analyze_event_payload, close_session, @@ -191,6 +192,10 @@ def handle_runtime(args: argparse.Namespace) -> int: ) print(json.dumps({"outcome": outcome.to_dict()}, ensure_ascii=False, indent=2, sort_keys=True)) return 0 + if args.runtime_command == "layers": + report = export_layer_runtime_workflow(Path(args.ledger), Path(args.out_dir)) + print(json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True)) + return 0 if report.get("status") == "pass" else 1 return 2 def handle_approval(args: argparse.Namespace) -> int: diff --git a/src/invart/commands/integrations.py b/src/invart/commands/integrations.py index 1f396b2..9e5617b 100644 --- a/src/invart/commands/integrations.py +++ b/src/invart/commands/integrations.py @@ -19,7 +19,7 @@ run_swe_bench_lite_check, ) from invart.evaluation.real_agent_conformance import export_real_agent_report_html, run_real_agent_conformance -from invart.surfaces.adapter_profiles import build_adapter_profile +from invart.surfaces.adapter_profiles import adapter_track_matrix, build_adapter_profile from invart.surfaces.enforcement import check_enforcement, run_enforced_command, run_file_write_intercepted, rust_build_check, rust_shim_decision, rust_shim_spec from invart.surfaces.native import install_native_integration, inventory_native_integrations, native_capability_matrix, native_conformance_report, unmanaged_agent_inventory from invart.surfaces.launcher import install_managed_launcher, preview_managed_launcher, verify_managed_launcher @@ -45,6 +45,10 @@ def handle_adapter(args: argparse.Namespace) -> int: if args.adapter_command == "profile": print(json.dumps(build_adapter_profile(args.kind), ensure_ascii=False, indent=2, sort_keys=True)) return 0 + if args.adapter_command == "profiles": + result = adapter_track_matrix(track=args.track) + print(json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True)) + return 0 if result.get("status") == "pass" else 1 if args.adapter_command == "claude-code-check": result = check_claude_code_environment(binary=args.binary) print(json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True)) diff --git a/src/invart/commands/parser_foundation.py b/src/invart/commands/parser_foundation.py index 4953423..eb1b33a 100644 --- a/src/invart/commands/parser_foundation.py +++ b/src/invart/commands/parser_foundation.py @@ -99,6 +99,9 @@ def register_foundation_commands(subparsers: argparse._SubParsersAction[argparse outcome.add_argument("--status", choices=("executed", "blocked", "skipped", "overridden", "failed"), required=True) outcome.add_argument("--actor", default=None) outcome.add_argument("--reason", default=None) + layers = runtime_sub.add_parser("layers", help="Export the L1-L5 runtime operation workflow for a ledger.") + layers.add_argument("--ledger", required=True) + layers.add_argument("--out-dir", required=True) approval = subparsers.add_parser("approval", help="List and resolve approval inbox items.") approval.set_defaults(handler=handle_approval) @@ -177,4 +180,3 @@ def register_foundation_commands(subparsers: argparse._SubParsersAction[argparse post = subparsers.add_parser("post-runtime", help="Summarize a JSONL runtime audit log.") post.set_defaults(handler=handle_post_runtime) post.add_argument("--events", default=".invart/session.jsonl", help="JSONL event log path.") - diff --git a/src/invart/commands/parser_integrations.py b/src/invart/commands/parser_integrations.py index 8355ee2..548681b 100644 --- a/src/invart/commands/parser_integrations.py +++ b/src/invart/commands/parser_integrations.py @@ -32,6 +32,8 @@ def register_integration_commands(subparsers: argparse._SubParsersAction[argpars adapter_package.add_argument("--out-dir", required=True) adapter_profile = adapter_sub.add_parser("profile", help="Inspect a hardened adapter profile.") adapter_profile.add_argument("--kind", choices=adapter_profile_ids(), default="claude-code") + adapter_profiles = adapter_sub.add_parser("profiles", help="List priority agent adapter tracks.") + adapter_profiles.add_argument("--track", default=None) adapter_claude_check = adapter_sub.add_parser("claude-code-check", help="Check a real Claude Code binary environment when available.") adapter_claude_check.add_argument("--binary", default="claude") adapter_claude = adapter_sub.add_parser("claude-code", help="Run a command through the Claude Code wrapper/hook bridge.") diff --git a/src/invart/evaluation/benchmark_registry.py b/src/invart/evaluation/benchmark_registry.py index 4a7ffa2..588db90 100644 --- a/src/invart/evaluation/benchmark_registry.py +++ b/src/invart/evaluation/benchmark_registry.py @@ -33,6 +33,8 @@ {"suite": "v0.51-pre-1.0-research-ready-gate", "version": "v0.51", "category": "release", "optional_heavy": False, "claim_scope": "research_gate", "evidence_level": "local_research_artifact_bundle"}, {"suite": "v0.9.3-agent-adapter-contract", "version": "v0.9.3", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_agent_adapter_contract", "evidence_level": "fixture_backed_conformance"}, {"suite": "v0.9.4-claude-reference-adapter", "version": "v0.9.4", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_claude_reference_adapter", "evidence_level": "local_runtime_fixture"}, + {"suite": "v0.9.5-priority-agent-tracks", "version": "v0.9.5", "category": "agent-adapter", "optional_heavy": False, "claim_scope": "local_agent_track_matrix", "evidence_level": "fixture_backed_profile_matrix"}, + {"suite": "v0.9.6-layer-runtime-workflow", "version": "v0.9.6", "category": "runtime", "optional_heavy": False, "claim_scope": "local_layer_runtime_workflow", "evidence_level": "ledger_derived_runtime_fixture"}, {"suite": "progressive-external-validation", "version": "pre-release", "category": "external-validation", "optional_heavy": False, "claim_scope": "progressive_external_validation", "evidence_level": "external_progressive_sample"}, {"suite": "real-world-agent-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "public_source_mapping", "evidence_level": "public_source_seed_plus_local_demo"}, {"suite": "containerized-risk-demo", "version": "pre-release", "category": "demo", "optional_heavy": False, "claim_scope": "containerized_local_demo", "evidence_level": "per_case_container_artifact_bundle"}, diff --git a/src/invart/evaluation/product_control_matrix.py b/src/invart/evaluation/product_control_matrix.py index 018b95b..bb5f2ad 100644 --- a/src/invart/evaluation/product_control_matrix.py +++ b/src/invart/evaluation/product_control_matrix.py @@ -7,6 +7,7 @@ from invart.core.artifacts import stable_json_hash, write_html_artifact, write_json_artifact from invart.core.models import utc_now +from invart.surfaces.adapter_profiles import list_adapter_profiles SCHEMA_VERSION = "invart.product_control_matrix.v0.50" @@ -15,7 +16,7 @@ def run_product_control_matrix(*, out_dir: Path | None = None) -> dict[str, Any]: root = (out_dir or Path(tempfile.mkdtemp(prefix="invart_product_control_matrix_"))).expanduser().resolve() root.mkdir(parents=True, exist_ok=True) - rows = _product_rows() + rows = _product_rows() + _profile_rows() baselines = _baseline_rows() checks = { "products_covered": len({row["product"] for row in rows}) >= 4, @@ -29,6 +30,7 @@ def run_product_control_matrix(*, out_dir: Path | None = None) -> dict[str, Any] for row in baselines ), "required_fields_present": all(_required_fields_present(row) for row in rows), + "profile_rows_match_track_vocabulary": _profile_rows_match_track_vocabulary(rows), } matrix_json = root / "product-control-matrix.json" matrix_html = root / "product-control-matrix.html" @@ -139,6 +141,56 @@ def _product_rows() -> list[dict[str, Any]]: ] +def _profile_rows() -> list[dict[str, Any]]: + rows = [] + for profile in list_adapter_profiles(): + coverage_grade = _profile_coverage_grade(profile) + rows.append( + { + "product": str(profile["display_name"]), + "agent_id": str(profile["agent_id"]), + "surface": ", ".join(profile.get("native_surfaces", [])), + "source": f"invart_profile:{profile['agent_id']}", + "source_kind": "invart_adapter_profile", + "source_urls": list(profile.get("source_urls", [])), + "native_control": ", ".join(profile.get("execution_modes", [])), + "invart_layer": str(profile.get("integration_track")), + "integration_track": str(profile.get("integration_track")), + "track_status": str(profile.get("track_status")), + "adapter_family": str(profile.get("adapter_family")), + "control_position": str(profile.get("control_position")), + "coverage_grade": coverage_grade, + "supports_mediation": bool(profile.get("supports_mediation")), + "limitation": str(profile.get("claim_boundary")), + "reference_meaning": "Profile-derived track row used to keep product comparison aligned with adapter coverage claims.", + } + ) + return rows + + +def _profile_rows_match_track_vocabulary(rows: list[dict[str, Any]]) -> bool: + valid_coverage = {"mediated", "vendor_owned", "observed"} + for row in rows: + if row.get("source_kind") != "invart_adapter_profile": + continue + if row.get("coverage_grade") not in valid_coverage: + return False + if row.get("control_position") == "vendor_owned_import" and (row.get("supports_mediation") or row.get("coverage_grade") != "vendor_owned"): + return False + if row.get("control_position") == "invart_mediated" and row.get("coverage_grade") != "mediated": + return False + return True + + +def _profile_coverage_grade(profile: dict[str, Any]) -> str: + control_position = str(profile.get("control_position", "")) + if control_position == "vendor_owned_import": + return "vendor_owned" + if control_position in {"invart_mediated", "bridge_mediated_when_configured"}: + return "mediated" + return "observed" + + def _baseline_rows() -> list[dict[str, Any]]: return [ { diff --git a/src/invart/evaluation/release_candidate.py b/src/invart/evaluation/release_candidate.py index 60fc1c5..1c23113 100644 --- a/src/invart/evaluation/release_candidate.py +++ b/src/invart/evaluation/release_candidate.py @@ -49,6 +49,8 @@ "v0.51-pre-1.0-research-ready-gate", "v0.9.3-agent-adapter-contract", "v0.9.4-claude-reference-adapter", + "v0.9.5-priority-agent-tracks", + "v0.9.6-layer-runtime-workflow", "progressive-external-validation", "pre-v1-control-plane", ) diff --git a/src/invart/evaluation/roadmap.py b/src/invart/evaluation/roadmap.py index 92f3830..451f94e 100644 --- a/src/invart/evaluation/roadmap.py +++ b/src/invart/evaluation/roadmap.py @@ -797,6 +797,36 @@ def internal_docs_for_capability(docs: list[str]) -> list[str]: external_validation="not_run_optional", next_step="Add priority agent tracks and product-specific adapter coverage claims for Codex, Gemini CLI, Cursor, OpenClaw, Hermes, OpenCode, and related agent products.", ), + RoadmapCapability( + version="v0.9.5", + capability_id="priority_agent_tracks", + title="Priority agent profile tracks", + target="Classify priority agent products into truthful adapter tracks with control position, adapter family, coverage grade, and product-matrix alignment.", + status="implemented", + implementation=["src/invart/surfaces/adapter_profiles.py", "src/invart/evaluation/product_control_matrix.py", "src/invart/commands/parser_integrations.py", "src/invart/commands/integrations.py", "src/invart/benchmarks/releases_v52_v57.py"], + tests=["test_v095_priority_agent_profiles_emit_tracks_and_control_positions", "test_v095_managed_local_tracks_produce_fixture_evidence", "test_v095_product_matrix_uses_profile_track_vocabulary", "test_v095_cli_and_benchmark_are_registered", "v0.9.5-priority-agent-tracks"], + docs=["docs/cli-reference.md", "docs/html/cli-reference.html", "docs/release-history.md", "docs/html/release-history.html"], + product_boundaries=["Local fixture-backed profile matrix validates Invart's track vocabulary and artifact expectations; it does not claim every live vendor product has full runtime mediation.", "Vendor evidence import, cloud import, and framework trace import tracks are explicitly not treated as Invart-mediated or enforced control positions.", "Managed wrapper and reference full adapter tracks require ledger/proof/evidence artifacts before they can satisfy product claims."], + claim_scope="local_agent_track_matrix", + evidence_level="fixture_backed_profile_matrix", + external_validation="not_run_optional", + next_step="Add L1-L5 runtime operation commands and evidence views so users can operate each layer explicitly.", + ), + RoadmapCapability( + version="v0.9.6", + capability_id="layer_runtime_workflow", + title="L1-L5 runtime operation workflow", + target="Generate a ledger-derived runtime workflow that maps before-runtime, during-runtime, and after-runtime operations across L1-L5 with linked proof, replay, graph, coverage, audit, and evidence artifacts.", + status="implemented", + implementation=["src/invart/assurance/layer_runtime.py", "src/invart/commands/parser_foundation.py", "src/invart/commands/foundation.py", "src/invart/benchmarks/releases_v52_v57.py"], + tests=["test_v096_layer_runtime_workflow_exports_stage_layer_artifacts", "test_v096_runtime_layers_cli_and_benchmark_are_registered", "v0.9.6-layer-runtime-workflow"], + docs=["docs/runtime-effect-demo.md", "docs/html/runtime-effect-demo.html", "docs/cli-reference.md", "docs/html/cli-reference.html", "docs/release-history.md", "docs/html/release-history.html"], + product_boundaries=["Local ledger-derived runtime fixture validates the L1-L5 operation workflow; it does not create facts outside the ledger source of truth.", "The workflow distinguishes observed, mediated, and enforced coverage and does not upgrade weak coverage labels.", "The command is an operator/reporting surface, not a hosted UI."], + claim_scope="local_layer_runtime_workflow", + evidence_level="ledger_derived_runtime_fixture", + external_validation="not_run_optional", + next_step="Harden L5 evidence workspace and release gate checks so missing or inconsistent artifacts fail clearly.", + ), ) diff --git a/src/invart/surfaces/adapter.py b/src/invart/surfaces/adapter.py index ba6c7e5..3da77b1 100644 --- a/src/invart/surfaces/adapter.py +++ b/src/invart/surfaces/adapter.py @@ -9,6 +9,7 @@ from typing import Any from invart.core.artifacts import sha256_file, write_html_artifact, write_json_artifact +from invart.assurance.evidence_bundle import export_evidence_bundle from invart.assurance.coverage import export_coverage_html_report from invart.control.daemon import RuntimeAuthority from invart.control.gate import verify_gate @@ -34,6 +35,7 @@ class AdapterRunResult: proof: str gate_report: str | None = None gate_status: str | None = None + package: str | None = None capability_registration: dict[str, Any] | None = None started_at: str | None = None ended_at: str | None = None @@ -113,6 +115,7 @@ def run_adapter_command( authority.transition_session(registry.session_id, "stopped", reason="adapter command blocked") export_proof_report(ledger, proof) gate_status = None + package = _export_command_adapter_package(ledger, resolved_out_dir, policy_mode=policy_mode, agent=agent) if gate_report_path: gate = verify_gate(ledger_path=ledger, proof_path=proof, mode=gate_mode, output_path=gate_report_path) gate_status = str(gate.get("status")) @@ -125,6 +128,7 @@ def run_adapter_command( proof=str(proof), gate_report=str(gate_report_path) if gate_report_path else None, gate_status=gate_status, + package=package, capability_registration=capability_registration, started_at=started_at, ended_at=utc_now(), @@ -146,6 +150,7 @@ def run_adapter_command( if gate_report_path: gate = verify_gate(ledger_path=ledger, proof_path=proof, mode=gate_mode, output_path=gate_report_path) gate_status = str(gate.get("status")) + package = _export_command_adapter_package(ledger, resolved_out_dir, policy_mode=policy_mode, agent=agent) status = "blocked" if child_status in {"blocked", "requires_approval"} else "passed" if returncode == 0 and gate_status not in {"fail"} else "failed" return AdapterRunResult( session_id=registry.session_id, @@ -156,6 +161,7 @@ def run_adapter_command( proof=str(proof), gate_report=str(gate_report_path) if gate_report_path else None, gate_status=gate_status, + package=package, capability_registration=capability_registration, started_at=started_at, ended_at=utc_now(), @@ -376,4 +382,15 @@ def _write_adapter_package(package_path: Path, *, adapter: dict[str, Any], artif return package_path +def _export_command_adapter_package(ledger: Path, out_dir: Path | None, *, policy_mode: str, agent: str | None) -> str | None: + if out_dir is None: + return None + bundle = export_evidence_bundle( + ledger, + out_dir / "adapter-package", + profile={"name": "adapter-run", "mode": policy_mode, "agent": agent or "generic"}, + ) + return str(bundle["manifest_path"]) + + __all__ = ["AdapterRunResult", "inspect_adapter_package", "run_adapter_command", "run_adapter_runtime"] diff --git a/src/invart/surfaces/adapter_profiles.py b/src/invart/surfaces/adapter_profiles.py index bd8a9e0..75d2789 100644 --- a/src/invart/surfaces/adapter_profiles.py +++ b/src/invart/surfaces/adapter_profiles.py @@ -26,9 +26,18 @@ class AgentAdapterProfile: supports_mediation: bool = False can_block: bool = False can_pause_resume: bool = False + integration_track: str | None = None + track_status: str | None = None + adapter_family: str | None = None + control_position: str | None = None def to_dict(self) -> dict[str, Any]: - return asdict(self) + payload = asdict(self) + derived = _derive_track(payload) + for key, value in derived.items(): + if not payload.get(key): + payload[key] = value + return payload _PROFILES: tuple[AgentAdapterProfile, ...] = ( @@ -335,6 +344,51 @@ def adapter_profile_registry() -> dict[str, Any]: } +def adapter_track_matrix(track: str | None = None) -> dict[str, Any]: + profiles = list_adapter_profiles() + if track: + profiles = [profile for profile in profiles if profile.get("integration_track") == track] + rows = [ + { + "agent_id": profile["agent_id"], + "display_name": profile["display_name"], + "priority": profile["priority"], + "integration_track": profile["integration_track"], + "track_status": profile["track_status"], + "adapter_family": profile["adapter_family"], + "control_position": profile["control_position"], + "coverage_grade": profile["coverage_grade"], + "supports_mediation": profile["supports_mediation"], + "can_block": profile["can_block"], + "can_pause_resume": profile["can_pause_resume"], + "required_artifacts": profile["required_artifacts"], + "claim_boundary": profile["claim_boundary"], + } + for profile in profiles + ] + validation = validate_adapter_profile_truthfulness(profiles) + checks = { + "profiles_validate": validation.get("status") == "pass", + "vendor_import_not_mediated": all(row["control_position"] == "vendor_owned_import" and not row["supports_mediation"] for row in rows if row["integration_track"] in {"vendor_evidence_import", "cloud_evidence_import", "framework_trace_import"}), + "managed_tracks_have_blocking_path": all(row["supports_mediation"] and row["can_block"] for row in rows if row["integration_track"] in {"reference_full_adapter", "managed_wrapper"}), + "tracks_have_claim_boundaries": all(bool(row["claim_boundary"]) for row in rows), + } + return { + "schema_version": "invart.adapter_track_matrix.v0.9.5", + "status": "pass" if all(checks.values()) else "fail", + "track_filter": track, + "rows": rows, + "checks": checks, + "validation": validation, + "summary": { + "profiles": len(rows), + "tracks": _count_by(rows, "integration_track"), + "adapter_families": _count_by(rows, "adapter_family"), + "control_positions": _count_by(rows, "control_position"), + }, + } + + def validate_adapter_profile_truthfulness(profiles: list[dict[str, Any]] | None = None) -> dict[str, Any]: profiles = profiles or list_adapter_profiles() required_fields = { @@ -349,6 +403,10 @@ def validate_adapter_profile_truthfulness(profiles: list[dict[str, Any]] | None "required_artifacts", "source_urls", "last_reviewed", + "integration_track", + "track_status", + "adapter_family", + "control_position", } valid_grades = { "full_managed_adapter", @@ -357,9 +415,26 @@ def validate_adapter_profile_truthfulness(profiles: list[dict[str, Any]] | None "vendor_evidence_import", "discovery_only", } + valid_tracks = { + "reference_full_adapter", + "managed_wrapper", + "native_bridge", + "vendor_evidence_import", + "cloud_evidence_import", + "framework_trace_import", + "discovery_only", + } + valid_control_positions = { + "invart_mediated", + "bridge_mediated_when_configured", + "vendor_owned_import", + "discovery_only", + } checks = { "required_fields_present": all(required_fields.issubset(profile) and all(profile.get(field) for field in required_fields) for profile in profiles), "coverage_grades_known": all(profile.get("coverage_grade") in valid_grades for profile in profiles), + "track_fields_present": all(all(profile.get(field) for field in ("integration_track", "track_status", "adapter_family", "control_position")) for profile in profiles), + "tracks_known": all(profile.get("integration_track") in valid_tracks and profile.get("control_position") in valid_control_positions for profile in profiles), "source_urls_https": all(str(url).startswith("https://") for profile in profiles for url in profile.get("source_urls", [])), "claim_boundaries_present": all(bool(profile.get("claim_boundary")) for profile in profiles), "full_managed_requires_artifacts": all( @@ -380,6 +455,21 @@ def validate_adapter_profile_truthfulness(profiles: list[dict[str, Any]] | None for profile in profiles if profile.get("coverage_grade") == "discovery_only" ), + "vendor_import_track_not_mediated": all( + profile.get("supports_mediation") is False + and profile.get("can_block") is False + and profile.get("control_position") == "vendor_owned_import" + for profile in profiles + if profile.get("integration_track") in {"vendor_evidence_import", "cloud_evidence_import", "framework_trace_import"} + ), + "managed_track_has_artifact_boundary": all( + {"ledger", "proof"}.issubset(set(profile.get("required_artifacts", []))) + and profile.get("supports_mediation") is True + and profile.get("can_block") is True + and profile.get("control_position") == "invart_mediated" + for profile in profiles + if profile.get("integration_track") in {"reference_full_adapter", "managed_wrapper"} + ), } findings = [ {"check_id": check_id, "status": "fail"} @@ -426,10 +516,65 @@ def _count_by(items: list[dict[str, Any]], field_name: str) -> dict[str, int]: return counts +def _derive_track(profile: dict[str, Any]) -> dict[str, str]: + agent_id = str(profile.get("agent_id", "")) + coverage_grade = str(profile.get("coverage_grade", "")) + execution_modes = set(profile.get("execution_modes", [])) + if agent_id == "claude-code": + return { + "integration_track": "reference_full_adapter", + "track_status": "implemented", + "adapter_family": "product_cli", + "control_position": "invart_mediated", + } + if agent_id == "github-copilot-cloud-agent": + return { + "integration_track": "cloud_evidence_import", + "track_status": "planned_import", + "adapter_family": "cloud_agent", + "control_position": "vendor_owned_import", + } + if coverage_grade == "managed_wrapper_adapter": + return { + "integration_track": "managed_wrapper", + "track_status": "fixture_validated", + "adapter_family": "product_cli", + "control_position": "invart_mediated", + } + if coverage_grade == "native_event_bridge": + return { + "integration_track": "native_bridge", + "track_status": "fixture_validated" if profile.get("supports_mediation") else "planned_import", + "adapter_family": "ide_agent" if "ide_extension" in execution_modes or "ide_extension" in set(profile.get("native_surfaces", [])) else "product_cli", + "control_position": "bridge_mediated_when_configured", + } + if coverage_grade == "vendor_evidence_import" and ("framework_trace_import" in execution_modes or agent_id in {"openai-agents-sdk", "langgraph", "crewai"}): + return { + "integration_track": "framework_trace_import", + "track_status": "planned_import", + "adapter_family": "framework", + "control_position": "vendor_owned_import", + } + if coverage_grade == "vendor_evidence_import": + return { + "integration_track": "vendor_evidence_import", + "track_status": "planned_import", + "adapter_family": "product_cli", + "control_position": "vendor_owned_import", + } + return { + "integration_track": "discovery_only", + "track_status": "planned_import", + "adapter_family": "unknown", + "control_position": "discovery_only", + } + + __all__ = [ "AgentAdapterProfile", "adapter_profile_ids", "adapter_profile_registry", + "adapter_track_matrix", "build_adapter_profile", "get_adapter_profile", "list_adapter_profiles", diff --git a/tests/test_integrations.py b/tests/test_integrations.py index 890ebb0..16cffd3 100644 --- a/tests/test_integrations.py +++ b/tests/test_integrations.py @@ -640,6 +640,124 @@ def test_v094_claude_adapter_cli_and_benchmark_are_registered(tmp_path: Path) -> assert main(["eval", "benchmark", "--suite", "v0.9.4-claude-reference-adapter"]) == 0 +def test_v095_priority_agent_profiles_emit_tracks_and_control_positions() -> None: + from invart.surfaces.adapter_profiles import adapter_track_matrix, list_adapter_profiles, validate_adapter_profile_truthfulness + + profiles = list_adapter_profiles() + by_agent = {profile["agent_id"]: profile for profile in profiles} + expected_tracks = { + "claude-code": ("reference_full_adapter", "invart_mediated"), + "codex": ("managed_wrapper", "invart_mediated"), + "gemini-cli": ("managed_wrapper", "invart_mediated"), + "cursor": ("native_bridge", "bridge_mediated_when_configured"), + "opencode": ("native_bridge", "bridge_mediated_when_configured"), + "openclaw": ("vendor_evidence_import", "vendor_owned_import"), + "hermes": ("vendor_evidence_import", "vendor_owned_import"), + "github-copilot-cloud-agent": ("cloud_evidence_import", "vendor_owned_import"), + } + for agent_id, (track, control_position) in expected_tracks.items(): + profile = by_agent[agent_id] + assert profile["integration_track"] == track + assert profile["control_position"] == control_position + assert profile["adapter_family"] + assert profile["track_status"] in {"implemented", "fixture_validated", "planned_import"} + + validation = validate_adapter_profile_truthfulness(profiles) + assert validation["status"] == "pass" + assert validation["checks"]["track_fields_present"] is True + assert validation["checks"]["vendor_import_track_not_mediated"] is True + + matrix = adapter_track_matrix() + assert matrix["schema_version"] == "invart.adapter_track_matrix.v0.9.5" + assert matrix["status"] == "pass" + assert matrix["summary"]["tracks"]["reference_full_adapter"] >= 1 + + +def test_v095_managed_local_tracks_produce_fixture_evidence(tmp_path: Path) -> None: + from invart.assurance.evidence_bundle import verify_evidence_bundle + + claude = run_claude_code_adapter( + target=tmp_path, + command=[sys.executable, "-c", "pass"], + out_dir=tmp_path / "claude", + session_id="ks_v095_claude", + policy_mode="advisory", + ) + assert claude["status"] == "passed" + assert verify_evidence_bundle(Path(claude["adapter_package"]["manifest_path"]))["status"] == "pass" + + codex = run_adapter_command( + target=tmp_path, + command=[sys.executable, "-c", "print('codex track')"], + agent="codex", + goal="v0.9.5 track fixture", + session_id="ks_v095_codex", + out_dir=tmp_path / "codex", + capabilities="audit", + gate_mode="audit", + create_preflight=False, + ) + assert codex.status == "passed" + assert Path(codex.ledger).exists() + assert Path(codex.proof).exists() + assert codex.package is not None and Path(codex.package).exists() + + +def test_v095_product_matrix_uses_profile_track_vocabulary(tmp_path: Path) -> None: + from invart.evaluation.product_control_matrix import run_product_control_matrix + + matrix = run_product_control_matrix(out_dir=tmp_path / "matrix") + rows = [row for row in matrix["rows"] if row.get("source_kind") == "invart_adapter_profile"] + by_agent = {row["agent_id"]: row for row in rows} + assert by_agent["claude-code"]["integration_track"] == "reference_full_adapter" + assert by_agent["claude-code"]["coverage_grade"] == "mediated" + assert by_agent["github-copilot-cloud-agent"]["coverage_grade"] == "vendor_owned" + assert by_agent["github-copilot-cloud-agent"]["supports_mediation"] is False + assert matrix["checks"]["profile_rows_match_track_vocabulary"] is True + + +def test_v095_cli_and_benchmark_are_registered() -> None: + assert main(["adapter", "profiles"]) == 0 + assert main(["adapter", "profiles", "--track", "managed_wrapper"]) == 0 + assert main(["eval", "benchmark", "--suite", "v0.9.5-priority-agent-tracks"]) == 0 + + +def test_v096_layer_runtime_workflow_exports_stage_layer_artifacts(tmp_path: Path) -> None: + from invart.assurance.layer_runtime import export_layer_runtime_workflow + + ledger = tmp_path / "ledger.jsonl" + session = start_session(tmp_path, ledger, agent="claude-code", goal="v0.9.6 layer workflow", create_preflight=False) + record_action(RuntimeEvent(type="file_read", session_id=session.session_id, path=str(tmp_path / ".env"), metadata={"coverage_layer": "native_hook"}), ledger) + record_action(RuntimeEvent(type="network", session_id=session.session_id, url="https://example.com/upload", metadata={"coverage_layer": "native_hook"}), ledger) + close_session(ledger) + + report = export_layer_runtime_workflow(ledger, tmp_path / "layers") + assert report["schema_version"] == "invart.layer_runtime_workflow.v0.9.6" + assert report["status"] == "pass" + assert {item["stage"] for item in report["runtime_effect_matrix"]} == {"before-runtime", "during-runtime", "after-runtime"} + assert {item["layer"] for item in report["runtime_effect_matrix"]} == {"L1", "L2", "L3", "L4", "L5"} + assert {item["layer"] for item in report["layer_timeline"]} == {"L1", "L2", "L3", "L4", "L5"} + for artifact in ("proof", "replay", "path_graph_json", "path_graph_html", "coverage", "audit_html", "evidence_manifest", "workflow_json", "workflow_html"): + assert Path(report["artifacts"][artifact]).exists() + html = Path(report["artifacts"]["workflow_html"]).read_text(encoding="utf-8") + assert "L1 Execution Surface" in html + assert "before-runtime" in html + assert "Observed, mediated, and enforced are not interchangeable" in html + + +def test_v096_runtime_layers_cli_and_benchmark_are_registered(tmp_path: Path) -> None: + ledger = tmp_path / "ledger.jsonl" + session = start_session(tmp_path, ledger, agent="codex", goal="v0.9.6 cli", create_preflight=False) + record_action(RuntimeEvent(type="shell", session_id=session.session_id, command="echo ok", metadata={"coverage_layer": "shell_wrapper"}), ledger) + close_session(ledger) + out_dir = tmp_path / "cli-layers" + + assert main(["runtime", "layers", "--ledger", str(ledger), "--out-dir", str(out_dir)]) == 0 + assert (out_dir / "layer-runtime-workflow.json").exists() + assert (out_dir / "layer-runtime-workflow.html").exists() + assert main(["eval", "benchmark", "--suite", "v0.9.6-layer-runtime-workflow"]) == 0 + + def test_v09_swe_bench_lite_runner_skips_cleanly_without_dependencies(tmp_path: Path) -> None: out = tmp_path / "swebench-report.json" assert main([