Merge pull request #71 from Taz33m/codex/qualification-evidence-scaffold

Add captured two-run qualification evidence

Author: Taz33m

CHANGELOG.md

@@ -6,6 +6,12 @@ The project follows [Keep a Changelog](https://keepachangelog.com/) conventions.
 
 ## Unreleased
 
+- Added a captured two-run qualification workflow. Candidate adapters can now
+  report optional revision and snapshot metadata, candidate-facing commands can
+  enforce a task-pinned identity before the first event, and `evidence-init` /
+  `evidence-verify` prepare clean roots and emit a grading-ready manifest only
+  for unchanged, complete, canonical, byte-identical qualification bundles.
+
 ## 0.6.0 - 2026-07-27
 
 - Fixed conformance CLI exit classification across single runs, suites,

README.md

@@ -189,6 +189,15 @@ these are separate version boundaries. `fifo-limit-v1` qualification does not
 run STP or pro-rata cases, so unsupported features stay explicit without
 becoming false failures.
 
+For release evidence, do not rely on one successful workspace. Start a captured
+pair with `tracebook-conformance evidence-init`, run the canonical qualification
+once in each generated clean root with the plan's pinned candidate name,
+revision, and snapshot, then run `tracebook-conformance evidence-verify`. The
+verifier emits a compact `evidence-manifest.json` only when both source trees
+remain unchanged and both qualification bundles agree on terminal result,
+candidate metadata, deterministic IDs, counts, coverage, and every artifact
+byte. See the [captured evidence workflow](https://github.com/Taz33m/tracebook/blob/main/docs/conformance.md#captured-two-run-evidence).
+
 [Read the research-grounded roadmap and adoption experiment](https://github.com/Taz33m/tracebook/blob/main/docs/research-roadmap.md).
 
 ## Architecture

docs/api-stability.md

@@ -30,6 +30,11 @@
   normalization, or hashing requires an explicit protocol/schema version
   decision. `suite_hash` binds case configuration and fixture identity; an
   intentional suite edit must update it.
+- Protocol-v1 engine metadata may add the optional `revision` and `snapshot_id`
+  identity fields. Ordinary adapters may omit them; a command using the three
+  task-pinned `--candidate-*` flags requires an exact match before any event is
+  sent. Evidence plan schema v1 and evidence manifest schema v1 are public
+  artifacts for the canonical captured two-run qualification workflow.
 - Campaign artifact schema version 1 is public. Campaign generator version 2,
   the built-in versioned profile definitions, seed derivation, and trace hashes
   are reproducibility contracts. An intentional generation change requires a

docs/commands.md

@@ -132,6 +132,27 @@ automatically minimized failure. Use this command for a first integration or a
 profile-level CI claim; use `suite` when intentionally comparing every broader
 fixed semantic surface.
 
+Prepare and verify a release-grade two-run evidence pair:
+
+```bash
+tracebook-conformance evidence-init /path/to/candidate \
+  --workspace .tracebook/release-evidence \
+  --candidate-name owner/repository \
+  --candidate-revision REVISION
+
+# Run the canonical qualification from evidence-plan.json in both generated
+# roots, with all three --candidate-* identity flags, then:
+tracebook-conformance evidence-verify \
+  .tracebook/release-evidence/evidence-plan.json
+```
+
+`evidence-init` refuses an existing workspace and creates independent candidate,
+adapter, build, cache, and qualification paths for `run-1` and `run-2`.
+`evidence-verify` writes `evidence-manifest.json` only after both canonical
+bundles are qualified, byte-identical, and bound to the unchanged pinned
+candidate. See [Captured Two-Run Evidence](conformance.md#captured-two-run-evidence)
+for the exact qualification command and trust boundary.
+
 Copy and run the standard suite:
 
 ```bash

docs/conformance.md

@@ -135,6 +135,64 @@ not exchange certification and does not imply support for unselected suite
 cases. Selection version 1 is part of the artifact identity and will not change
 silently.
 
+## Captured Two-Run Evidence
+
+Use the evidence workflow when a qualification result will support a release,
+benchmark, or external claim. It pins the candidate before adapter readiness and
+requires the same canonical qualification from two prepared roots.
+
+```bash
+REVISION=$(git -C /path/to/candidate rev-parse HEAD)
+WORKSPACE=$PWD/.tracebook/release-evidence
+
+tracebook-conformance evidence-init /path/to/candidate \
+  --workspace "$WORKSPACE" \
+  --candidate-name owner/repository \
+  --candidate-revision "$REVISION"
+```
+
+`evidence-init` strips `.git` from the captured source identity, copies that
+exact tree into `runs/run-1/candidate` and `runs/run-2/candidate`, and creates a
+separate empty `adapter`, `build`, and `cache` directory for each run. It writes
+`evidence-plan.json` last and refuses an existing or source-nested workspace.
+Use each run's paths for that run's adapter and build; do not share generated
+files between the two runs.
+
+The command prints the candidate snapshot ID. The adapter's `ready` frame must
+report that ID and the task-pinned name and revision. Run the plan's canonical
+contract in each root, changing only the run number and adapter command:
+
+```bash
+SNAPSHOT=$(python -c \
+  'import json,sys; print(json.load(open(sys.argv[1]))["candidate"]["snapshot_id"])' \
+  "$WORKSPACE/evidence-plan.json")
+
+tracebook-conformance qualify \
+  --profile fifo-limit-v1 --suite-version v2 \
+  --seed 42 --traces 25 --events-per-trace 200 --max-minimize-runs 100 \
+  --candidate-name owner/repository \
+  --candidate-revision "$REVISION" \
+  --candidate-snapshot "$SNAPSHOT" \
+  --candidate-cmd "$WORKSPACE/runs/run-1/adapter/launch" \
+  --output-dir "$WORKSPACE/runs/run-1/qualification"
+```
+
+Repeat for `run-2`, using only its candidate, adapter, build, cache, and
+qualification paths. Then verify the pair:
+
+```bash
+tracebook-conformance evidence-verify "$WORKSPACE/evidence-plan.json"
+```
+
+Verification fails closed if either captured tree changed; either bundle is
+missing, noncanonical, incomplete, or unqualified; candidate identity differs;
+or terminal result, deterministic IDs, counts, semantic coverage, or artifact
+bytes differ between runs. Success writes one exclusive
+`evidence-manifest.json` at the workspace root. The workflow prepares and checks
+the filesystem contract, but it does not sandbox a compiler or prove that an
+adapter avoided undeclared external caches; CI or a disposable host remains the
+strongest execution boundary.
+
 ## Differential Campaigns
 
 Campaigns generate stateful traces, compare them one at a time, and stop at the
@@ -346,9 +404,15 @@ Host to candidate, once:
 Candidate to host:
 
 ```json
-{"type":"ready","protocol":"tracebook.conformance","protocol_version":1,"engine":{"name":"my-engine","version":"1.4.2","language":"Rust"}}
+{"type":"ready","protocol":"tracebook.conformance","protocol_version":1,"engine":{"name":"owner/repository","version":"1.4.2","language":"Rust","revision":"8f31c2a","snapshot_id":"sha256:..."}}
 ```
 
+`revision` and `snapshot_id` are optional for ordinary comparisons. When
+`--candidate-name`, `--candidate-revision`, and `--candidate-snapshot` are
+supplied, all three flags are required together and the `ready` metadata must
+match before Tracebook sends the first event. Captured two-run evidence always
+uses the pinned form.
+
 Host to candidate for each event:
 
 ```json
@@ -429,7 +493,13 @@ from tracebook.conformance import EngineMetadata, serve_stdio
 
 class MyAdapter:
     def __init__(self, config):
-        self.metadata = EngineMetadata("my-engine", "1.0", "Python")
+        self.metadata = EngineMetadata(
+            "owner/repository",
+            "1.0",
+            "Python",
+            revision="8f31c2a",
+            snapshot_id="sha256:...",
+        )
         self.config = config
 
     def apply(self, event, index):
@@ -447,6 +517,10 @@ class MyAdapter:
 raise SystemExit(serve_stdio(MyAdapter))
 ```
 
+The last two fields are needed only when the invoking command pins candidate
+identity. In a captured evidence run, populate them from the immutable task or
+`evidence-plan.json`, not by inspecting the host after the adapter starts.
+
 [`examples/conformance_adapter.py`](../examples/conformance_adapter.py) is a
 runnable reference. Non-Python adapters implement the same frames directly.
 

src/tracebook/conformance/cli.py

@@ -20,11 +20,12 @@
     write_campaign_corpus,
 )
 from .compare import run_conformance
+from .evidence import prepare_evidence_workspace, write_evidence_manifest
 from .external import AdapterProtocolError, ExternalProcessAdapterFactory
 from .exit_codes import exit_code_for_artifact
 from .junit import write_junit
 from .minimize import minimize_failing_trace
-from .model import ConformanceConfig, ConformanceError
+from .model import ConformanceConfig, ConformanceError, PinnedCandidateIdentity
 from .qualification import _QualificationOutputReservation, run_qualification
 from .reproduce import (
     discover_failure_metadata,
@@ -54,6 +55,18 @@ def _add_config_arguments(parser: argparse.ArgumentParser) -> None:
 
 def _add_candidate_arguments(parser: argparse.ArgumentParser) -> None:
     parser.add_argument("--timeout", type=float, default=5.0)
+    parser.add_argument(
+        "--candidate-name",
+        help="Task-pinned candidate name expected in the ready frame.",
+    )
+    parser.add_argument(
+        "--candidate-revision",
+        help="Task-pinned revision expected in the ready frame.",
+    )
+    parser.add_argument(
+        "--candidate-snapshot",
+        help="Task-pinned snapshot ID expected in the ready frame.",
+    )
     candidate = parser.add_mutually_exclusive_group(required=True)
     candidate.add_argument(
         "--candidate-cmd",
@@ -163,6 +176,23 @@ def _build_parser() -> argparse.ArgumentParser:
     reproduce.add_argument("--junit-output")
     _add_config_arguments(reproduce)
     _add_candidate_arguments(reproduce)
+
+    evidence_init = commands.add_parser(
+        "evidence-init",
+        help="Create two clean roots for a captured qualification pair.",
+    )
+    evidence_init.add_argument("candidate_source")
+    evidence_init.add_argument("--workspace", required=True)
+    evidence_init.add_argument("--candidate-name", required=True)
+    evidence_init.add_argument("--candidate-revision", required=True)
+    evidence_init.add_argument("--expected-snapshot")
+
+    evidence_verify = commands.add_parser(
+        "evidence-verify",
+        help="Verify two captured qualification bundles and write one manifest.",
+    )
+    evidence_verify.add_argument("plan")
+    evidence_verify.add_argument("--output")
     return parser
 
 
@@ -174,7 +204,32 @@ def _candidate_factory(args) -> ExternalProcessAdapterFactory:
         command = command[1:]
     if not command:
         raise ConformanceError("--candidate requires a command")
-    return ExternalProcessAdapterFactory(command, timeout_seconds=args.timeout)
+    identity_values = (
+        args.candidate_name,
+        args.candidate_revision,
+        args.candidate_snapshot,
+    )
+    if any(value is not None for value in identity_values) and not all(
+        value is not None for value in identity_values
+    ):
+        raise ConformanceError(
+            "--candidate-name, --candidate-revision, and --candidate-snapshot "
+            "must be provided together"
+        )
+    expected_identity = (
+        PinnedCandidateIdentity(
+            name=args.candidate_name,
+            revision=args.candidate_revision,
+            snapshot_id=args.candidate_snapshot,
+        )
+        if all(value is not None for value in identity_values)
+        else None
+    )
+    return ExternalProcessAdapterFactory(
+        command,
+        timeout_seconds=args.timeout,
+        expected_identity=expected_identity,
+    )
 
 
 def _config(args) -> ConformanceConfig:
@@ -253,6 +308,31 @@ def main(argv: Optional[List[str]] = None) -> int:
             print(f"Suite id: {suite.suite_id}")
             print(f"Cases: {len(suite.cases)}")
             return 0
+        if args.command == "evidence-init":
+            plan_path = prepare_evidence_workspace(
+                args.candidate_source,
+                args.workspace,
+                candidate_name=args.candidate_name,
+                candidate_revision=args.candidate_revision,
+                expected_snapshot=args.expected_snapshot,
+            )
+            plan = json.loads(plan_path.read_text(encoding="utf-8"))
+            print(f"Evidence plan written: {plan_path}")
+            print(f"Candidate snapshot: {plan['candidate']['snapshot_id']}")
+            for run in plan["runs"]:
+                print(
+                    f"{run['run_id']}: candidate={run['candidate_root']} "
+                    f"qualification={run['qualification_dir']}"
+                )
+            return 0
+        if args.command == "evidence-verify":
+            manifest_path = write_evidence_manifest(args.plan, args.output)
+            manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+            print(f"Evidence manifest written: {manifest_path}")
+            print(f"Manifest: {manifest['manifest_id']}")
+            print(f"Qualification: {manifest['runs'][0]['qualification_id']}")
+            print("Evidence pair: PASS")
+            return 0
         if args.command == "run":
             _require_distinct_paths(args.events, args.output, args.junit_output)
             single_report = run_conformance(