Validation
Behavioral subtyping
Substitution contracts rather than inheritance-shaped code reuse.
Reference corpus · 17 calibrated cases
+Reference corpus · 21 calibrated cases
Cases designed to expose weak evidence.
Every case carries observable requirements, an objective verifier, explicit resource bounds, and known-good, known-bad, and adversarial variants. The corpus demonstrates the contract; it does not define Skivolve’s domain ceiling.
Cases designed to expose weak evidence.
Composition
-Ten train cases. Seven validation cases. Two disciplines.
-The split is fixed in the suite manifest. Engineering contributes five train and five validation cases; testing contributes five train and two validation cases.
+Ten train cases. Eleven validation cases. Two disciplines.
+The split is fixed in the suite manifest. Engineering contributes five train and nine validation cases; testing contributes five train and two validation cases.
-
10engineering cases
+ 14engineering cases
7testing cases
10train cases
- 7validation cases
+ 11validation cases
Ten train cases. Seven validation cases. Two discipli
Validation
Knowledge boundary
Authority, locality, and duplication decisions under change pressure.
Validation
Domain simplicity
Direct design and earned complexity for domain behavior.
Validation
Compatibility decision
Public-contract conflicts, migration ownership, and safe ask-before-change restraint.
Validation
Root-cause diagnosis
Concrete causal evidence, reproducible checks, and honest remaining gaps.
Validation
Surgical plan
Proportional implementation scope, precise boundaries, and native verification.
Validation
Evidence gap
Source-bounded decisions, material counter-evidence, and resolving next checks.
Evolve agent skills with evidence.
- 17calibrated cases
+ 21calibrated cases
3artifact contracts
3 × 3comparison arms and repetitions
MITopen-source license
@@ -213,15 +213,15 @@ Evaluate the output you actually care about.
Reference corpus
- Seventeen cases built to break weak evidence.
- Ten engineering and seven testing cases span train and validation splits. The corpus is a reference implementation of the case contract, not the evaluator’s domain limit.
+ Twenty-one cases built to break weak evidence.
+ Fourteen engineering and seven testing cases span train and validation splits. The corpus is a reference implementation of the case contract, not the evaluator’s domain limit.
Included reference corpus by track and split
Track Train Validation Focus Engineering 5 5 Correctness, compatibility, security, concurrency, performance, simplicity Engineering 5 9 Correctness, compatibility, diagnosis, planning, research, security, concurrency, performance, simplicity Testing 5 2 Oracle sensitivity, boundary fidelity, state models, flake control, idempotency
diff --git a/skivolve/codex_app_server.py b/skivolve/codex_app_server.py
index d9a82f0..d46678d 100644
--- a/skivolve/codex_app_server.py
+++ b/skivolve/codex_app_server.py
@@ -4400,7 +4400,7 @@ def _static_config(root: Path, model: str, reasoning_effort: str) -> bytes:
"",
"[shell_environment_policy]",
'inherit = "none"',
- f'set = {{ PATH = {_toml_string(f"{tools}/bin:{tools}/codex-path:{tools}/codex-resources:{tools}/codex-resources/zsh/bin:{tools}/required:/usr/bin:/bin")}, HOME = {_toml_string(str(home_directory))}, LANG = "C.UTF-8", TMPDIR = {_toml_string(str(temp_directory))}, XDG_CACHE_HOME = {_toml_string(str(cache_directory))} }}',
+ f'set = {{ PATH = {_toml_string(f"{tools}/bin:{tools}/codex-path:{tools}/codex-resources:{tools}/codex-resources/zsh/bin:{tools}/required:/usr/bin:/bin")}, HOME = {_toml_string(str(home_directory))}, LANG = "C.UTF-8", PYTHONDONTWRITEBYTECODE = "1", TMPDIR = {_toml_string(str(temp_directory))}, XDG_CACHE_HOME = {_toml_string(str(cache_directory))} }}',
"",
"[permissions.eval]",
'description = "Isolated Skivolve execution"',
diff --git a/skivolve/comparator-profile-authority.json b/skivolve/comparator-profile-authority.json
index 7a8aefc..100ebd9 100644
--- a/skivolve/comparator-profile-authority.json
+++ b/skivolve/comparator-profile-authority.json
@@ -4,8 +4,8 @@
{
"id": "plain-language-revision-v1",
"descriptor_sha256": "60806e964d6c415d1ebb007976acedd1286a2ec6dd788442e58f345cbbd26b30",
- "production_release_sha256": "94be3ed9ceb4ffe4316beef141cf8d3fa38e9aec01ae4a6ec47be3d73d444739",
- "test_release_sha256": "fea628d77f9b3509dfd9c8a5b113d9bf93fed4be911aa92cdee81e948b338863",
+ "production_release_sha256": "da012be22caeb1c76b37b181537a8f4e061419b35c520bde6d4dce484a8c9c40",
+ "test_release_sha256": "fdf61ed4ff33650a28c4431b98e8553205fb0c3aafc457844efcaad6f4c0958b",
"certification_contract_sha256": "48acdc90136e871ab9cea000c76f2ec94c517ed4466d6cba5beb57f5b4a9c265",
"requires_live_certification": true,
"authority_scope": "test"
@@ -13,8 +13,8 @@
{
"id": "software-engineering-v1",
"descriptor_sha256": "6a166ed3a313ee145d00519382c93c9fb56223aaf9b80f29806428f53f505e35",
- "production_release_sha256": "c07e78f8f5d3bd64f806ca924424c58d94753b521550c8b39bf0c51bd22efc89",
- "test_release_sha256": "866e2fe4a3a1bfd53ec3c020d6583d92b746a97c4a833a0a580a5ff5ecf7808c",
+ "production_release_sha256": "177f65fe98ea11ca11e9f5998070ff4c5d9a5233ba362d97e43356422c6f01f0",
+ "test_release_sha256": "ba789cdad72490699914d2046565e8a2fc4132252e31b2d8c5e4e279ffd2554e",
"certification_contract_sha256": "48acdc90136e871ab9cea000c76f2ec94c517ed4466d6cba5beb57f5b4a9c265",
"requires_live_certification": true,
"authority_scope": "production"
diff --git a/skivolve/comparator_calibration/release.json b/skivolve/comparator_calibration/release.json
index 3954158..6c4fe27 100644
--- a/skivolve/comparator_calibration/release.json
+++ b/skivolve/comparator_calibration/release.json
@@ -178,9 +178,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/comparator_calibration/tests/test-release.json b/skivolve/comparator_calibration/tests/test-release.json
index c3078b1..026eb1b 100644
--- a/skivolve/comparator_calibration/tests/test-release.json
+++ b/skivolve/comparator_calibration/tests/test-release.json
@@ -160,9 +160,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/plain_language_calibration/release.json b/skivolve/plain_language_calibration/release.json
index ca153cf..451d65f 100644
--- a/skivolve/plain_language_calibration/release.json
+++ b/skivolve/plain_language_calibration/release.json
@@ -160,9 +160,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/plain_language_calibration/tests/test-release.json b/skivolve/plain_language_calibration/tests/test-release.json
index fece2ef..1065a10 100644
--- a/skivolve/plain_language_calibration/tests/test-release.json
+++ b/skivolve/plain_language_calibration/tests/test-release.json
@@ -142,9 +142,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/providers.py b/skivolve/providers.py
index d40c9ef..5c15c6a 100644
--- a/skivolve/providers.py
+++ b/skivolve/providers.py
@@ -1512,6 +1512,7 @@ def _inner_prefix(
"SHELL=/bin/bash",
"TERM=dumb",
"CI=1",
+ "PYTHONDONTWRITEBYTECODE=1",
"CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1",
self._unshare,
"--user",
diff --git a/skivolve/runner.py b/skivolve/runner.py
index db46ff0..25ca2fb 100644
--- a/skivolve/runner.py
+++ b/skivolve/runner.py
@@ -3,6 +3,7 @@
from __future__ import annotations
import concurrent.futures
+import ctypes
import datetime as dt
import difflib
import fcntl
@@ -14,6 +15,7 @@
import select
import shutil
import stat
+import struct
import subprocess
import tempfile
import threading
@@ -133,6 +135,110 @@ class _GeneratorDispatchJournalError(RunnerError):
"""Raised when generator dispatch accounting cannot remain trustworthy."""
+class _WorkspaceMutationMonitor:
+ """Observe successful write operations even when final bytes are restored."""
+
+ _MASK = 0x00000002 | 0x00000004 | 0x00000008 | 0x00000040 | 0x00000080
+ _MASK |= 0x00000100 | 0x00000200 | 0x00000400 | 0x00000800
+ _EVENT = struct.Struct("=iIII")
+ _PROVIDER_RUNTIME_ENTRIES = frozenset(
+ {".skill-eval-cache", ".skill-eval-home", ".skill-eval-tmp"}
+ )
+
+ def __init__(self, root: Path) -> None:
+ libc = ctypes.CDLL(None, use_errno=True)
+ initialize = libc.inotify_init1
+ initialize.argtypes = (ctypes.c_int,)
+ initialize.restype = ctypes.c_int
+ add_watch = libc.inotify_add_watch
+ add_watch.argtypes = (ctypes.c_int, ctypes.c_char_p, ctypes.c_uint32)
+ add_watch.restype = ctypes.c_int
+ descriptor = initialize(os.O_CLOEXEC | os.O_NONBLOCK)
+ if descriptor < 0:
+ error = ctypes.get_errno()
+ raise RunnerError(
+ f"cannot monitor agent workspace writes: {os.strerror(error)}"
+ )
+ self._descriptor = descriptor
+ self._watched_directories: dict[int, Path] = {}
+ try:
+ for current, directories, _files in os.walk(root, followlinks=False):
+ current_path = Path(current)
+ watch_descriptor = add_watch(
+ descriptor,
+ ctypes.c_char_p(os.fsencode(current_path)),
+ self._MASK,
+ )
+ if watch_descriptor < 0:
+ error = ctypes.get_errno()
+ raise RunnerError(
+ "cannot monitor agent workspace directory "
+ f"{current_path}: {os.strerror(error)}"
+ )
+ self._watched_directories[watch_descriptor] = current_path.relative_to(
+ root
+ )
+ for name in directories:
+ path = current_path / name
+ if path.is_symlink() or not path.is_dir():
+ raise RunnerError(
+ f"agent workspace contains unsafe directory: {path}"
+ )
+ except BaseException:
+ self.close()
+ raise
+
+ def discard_pending(self) -> None:
+ self._drain()
+
+ def observed(self) -> bool:
+ return self._drain()
+
+ def _drain(self) -> bool:
+ observed = False
+ while True:
+ try:
+ chunk = os.read(self._descriptor, 64 * 1024)
+ except BlockingIOError:
+ return observed
+ except InterruptedError:
+ continue
+ except OSError as exc:
+ raise RunnerError(
+ f"cannot read agent workspace mutation evidence: {exc}"
+ ) from exc
+ if not chunk:
+ raise RunnerError(
+ "agent workspace mutation monitor closed unexpectedly"
+ )
+ offset = 0
+ while offset < len(chunk):
+ if len(chunk) - offset < self._EVENT.size:
+ raise RunnerError("agent workspace mutation event was truncated")
+ watch, _mask, _cookie, name_bytes = self._EVENT.unpack_from(
+ chunk, offset
+ )
+ offset += self._EVENT.size
+ end = offset + name_bytes
+ if end > len(chunk):
+ raise RunnerError("agent workspace mutation name was truncated")
+ name = os.fsdecode(chunk[offset:end].split(b"\0", 1)[0])
+ offset = end
+ relative = self._watched_directories.get(watch)
+ provider_runtime_event = (
+ relative is not None
+ and not relative.parts
+ and name in self._PROVIDER_RUNTIME_ENTRIES
+ )
+ if not provider_runtime_event:
+ observed = True
+
+ def close(self) -> None:
+ if self._descriptor >= 0:
+ os.close(self._descriptor)
+ self._descriptor = -1
+
+
_GENERATOR_DISPATCH_JOURNAL = "generator-dispatch.jsonl"
_GENERATOR_DISPATCH_LOCK = "generator-dispatch.lock"
_GENERATOR_DISPATCH_SCHEMA_VERSION = 1
@@ -1817,7 +1923,11 @@ def _preflight(
selection, allow_unsealed_holdout=allow_unsealed_holdout
)
self._assert_injected_fake_generator_admissible(selection, cases)
- if runtime is None and self.suite.evaluation_mode != "objective_only":
+ if (
+ runtime is None
+ and self.suite.evaluation_mode != "objective_only"
+ and not selection.verifier_only
+ ):
runtime = self._load_comparator_runtime()
generator_artifact_kinds = set(
capabilities_for(
@@ -1836,7 +1946,7 @@ def _preflight(
"generator adapter does not support selected artifact kinds: "
+ ", ".join(unsupported_generator_artifacts)
)
- if runtime is not None:
+ if runtime is not None and not selection.verifier_only:
unsupported_profile_artifacts = sorted(
{
case.artifact_contract.kind
@@ -2090,7 +2200,7 @@ def run(
)
return dry_run_result
runtime: ComparatorRuntime | None = None
- if self.suite.evaluation_mode == "judged":
+ if self.suite.evaluation_mode == "judged" and not selection.verifier_only:
runtime = self._load_comparator_runtime()
if runtime is not None and not selection.verifier_only:
assert self.suite.comparator is not None
@@ -3444,12 +3554,15 @@ def _run_arm(
provider_journal_state: str | None = None
provider_entered = False
synchronization_complete = False
+ workspace_mutation_observed = False
+ workspace_mutation_monitor: _WorkspaceMutationMonitor | None = None
normalized_artifact: NormalizedArtifact | None = None
stage = "fixture_copy"
try:
self._assert_case_integrity(case)
_copy_tree(case.fixture_dir, workspace, ignore_generated_caches=True)
before = _read_tree(workspace, ignore_generated_caches=True)
+ before_permissions = _read_tree_permissions(workspace)
hashes["fixture_before_sha256"] = _states_hash(before)
stage = "source_materialization"
source = self._materialize_source(variant, case, temp_root / "source")
@@ -3457,7 +3570,7 @@ def _run_arm(
hashes["context_sha256"] = source.context_hash
system_context = _system_context(case, source)
- def account_dispatch() -> None:
+ def account_dispatch(*, provider_callback: bool = True) -> None:
if provider_attempt_id is None:
raise _GeneratorDispatchJournalError(
"generator dispatch callback preceded durable planning"
@@ -3467,6 +3580,8 @@ def account_dispatch() -> None:
raise _GeneratorDispatchJournalError(
"generator dispatch ledger was not initialized"
)
+ if provider_callback and workspace_mutation_monitor is not None:
+ workspace_mutation_monitor.discard_pending()
ledger.mark_dispatched(provider_attempt_id)
provider_dispatched.set()
@@ -3518,11 +3633,13 @@ def account_dispatch() -> None:
provider_window["started"] = time.monotonic()
synchronization_complete = True
stage = "agent"
+ if case.artifact_contract.kind != "workspace_diff":
+ workspace_mutation_monitor = _WorkspaceMutationMonitor(workspace)
try:
provider_entered = True
provider_result = self.agent_provider.run_agent(request)
if not provider_dispatched.is_set():
- account_dispatch()
+ account_dispatch(provider_callback=False)
provider_journal_state = "dispatched"
if case.artifact_contract.kind != "workspace_diff":
stage = "artifact_normalization"
@@ -3547,6 +3664,13 @@ def account_dispatch() -> None:
provider_journal_state = "completed"
finally:
provider_window["finished"] = time.monotonic()
+ if workspace_mutation_monitor is not None:
+ try:
+ workspace_mutation_observed = (
+ workspace_mutation_monitor.observed()
+ )
+ finally:
+ workspace_mutation_monitor.close()
if source is not None and source.snapshot is not None:
_make_tree_writable(source.snapshot)
hashes["agent_output_sha256"] = _sha256(
@@ -3557,6 +3681,13 @@ def account_dispatch() -> None:
)
stage = "agent_workspace_scan"
after_agent = _read_tree(workspace, ignore_generated_caches=True)
+ after_agent_mutation = _read_tree(workspace)
+ after_agent_permissions = _read_tree_permissions(workspace)
+ agent_workspace_mutated = (
+ before != after_agent_mutation
+ or before_permissions != after_agent_permissions
+ or workspace_mutation_observed
+ )
hashes["workspace_after_agent_sha256"] = _states_hash(after_agent)
diff_text = _diff_states(before, after_agent)
hashes["diff_sha256"] = _sha256(diff_text.encode("utf-8"))
@@ -3588,6 +3719,7 @@ def account_dispatch() -> None:
normalized_artifact,
result_root,
workspace_read_only=final_output_artifact,
+ agent_workspace_mutated=agent_workspace_mutated,
)
verifier_after_hash = _tree_hash(verifier_workspace)
verifier_json["workspace_before_sha256"] = verifier_before_hash
@@ -3798,6 +3930,7 @@ def _run_verifier(
result_root: Path,
*,
workspace_read_only: bool,
+ agent_workspace_mutated: bool,
) -> dict[str, Any]:
command = self._verifier_commands.get(case.id)
if command is None:
@@ -3945,6 +4078,7 @@ def _run_verifier(
f"EVAL_ARTIFACT_PATH={mounted_artifact}",
f"EVAL_ARTIFACT_KIND={artifact.kind}",
f"EVAL_ARTIFACT_SHA256={artifact.sha256}",
+ f"EVAL_AGENT_WORKSPACE_MUTATED={int(agent_workspace_mutated)}",
f"EVAL_CASE_ROOT={mounted_case}",
*(
[f"EVAL_SHARED_ROOT={mounted_shared}"]
@@ -4001,6 +4135,7 @@ def _run_verifier(
"read_only": True,
},
"workspace_read_only": workspace_read_only,
+ "agent_workspace_mutated": agent_workspace_mutated,
"go_root": str(go_root) if go_root is not None else None,
"gcc_exec_prefix": (
str(gcc_exec_prefix) if gcc_exec_prefix is not None else None
@@ -6104,6 +6239,7 @@ def _scan_tree(
*,
ignore_generated_caches: bool = False,
ignore_empty_directories: bool = False,
+ include_directories: bool = False,
) -> list[Path]:
if not root.is_dir() or root.is_symlink():
raise RunnerError(f"tree root must be a regular directory: {root}")
@@ -6111,7 +6247,7 @@ def _scan_tree(
return _scan_normalized_worktree(
root, ignore_generated_caches=ignore_generated_caches
)
- files: list[Path] = []
+ paths: list[Path] = []
total = 0
entries = 0
for current, directories, filenames in os.walk(root, followlinks=False):
@@ -6132,6 +6268,8 @@ def _scan_tree(
f"tree exceeds maximum entries {MAX_TREE_ENTRIES}: {root}"
)
retained_directories.append(name)
+ if include_directories:
+ paths.append(path)
directories[:] = retained_directories
for name in filenames:
path = current_path / name
@@ -6151,8 +6289,8 @@ def _scan_tree(
total += size
if total > MAX_TREE_BYTES:
raise RunnerError(f"tree exceeds {MAX_TREE_BYTES} bytes: {root}")
- files.append(path)
- return sorted(files, key=lambda path: path.relative_to(root).as_posix())
+ paths.append(path)
+ return sorted(paths, key=lambda path: path.relative_to(root).as_posix())
def _read_tree(
@@ -6173,6 +6311,23 @@ def _read_tree(
return states
+def _read_tree_permissions(
+ root: Path,
+ *,
+ ignore_generated_caches: bool = False,
+) -> dict[str, int]:
+ permissions = {".": stat.S_IMODE(root.stat().st_mode)}
+ for path in _scan_tree(
+ root,
+ ignore_generated_caches=ignore_generated_caches,
+ include_directories=True,
+ ):
+ permissions[path.relative_to(root).as_posix()] = stat.S_IMODE(
+ path.stat().st_mode
+ )
+ return permissions
+
+
def _copy_tree(
source: Path,
destination: Path,
diff --git a/suite.json b/suite.json
index 2744bc3..94553c2 100644
--- a/suite.json
+++ b/suite.json
@@ -665,6 +665,226 @@
"qualitative_bases": {}
}
},
+ {
+ "id": "software-compatibility-decision",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/compatibility-decision/prompt.md",
+ "fixture_dir": "cases/software/compatibility-decision/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/compatibility-decision/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "pointed-public-choice",
+ "safe-default-recommendation",
+ "consumer-conflict-grounding",
+ "ask-and-stop-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "pointed-public-choice",
+ "kind": "required_behavior",
+ "text": "Ask one concrete either/or question that contrasts preserving the public v3 field names with authorizing a legacy-name migration."
+ },
+ {
+ "id": "safe-default-recommendation",
+ "kind": "required_behavior",
+ "text": "Recommend preserving the current v3 names until an owner decides the migration."
+ },
+ {
+ "id": "consumer-conflict-grounding",
+ "kind": "required_behavior",
+ "text": "Ground the block in North's deployed v3 dependency, South's legacy-name dependency, the public wire contract, and the missing migration owner or date."
+ },
+ {
+ "id": "ask-and-stop-restraint",
+ "kind": "hard_constraint",
+ "text": "Return exactly status, question, recommendation, reason, and actions_taken; record that no change was made and leave the workspace unchanged."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
+ {
+ "id": "software-root-cause-diagnosis",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/root-cause-diagnosis/prompt.md",
+ "fixture_dir": "cases/software/root-cause-diagnosis/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/root-cause-diagnosis/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "concrete-root-cause",
+ "minimal-line-evidence",
+ "reproducible-verification-check",
+ "diagnostic-report-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "concrete-root-cause",
+ "kind": "required_behavior",
+ "text": "Identify that the inclusive nonnegative condition classifies attempts=0 as ok instead of retry."
+ },
+ {
+ "id": "minimal-line-evidence",
+ "kind": "required_behavior",
+ "text": "Cite the failing observation at failure.log:2 and the causal condition at worker.py:2 using two or three bounded evidence entries."
+ },
+ {
+ "id": "reproducible-verification-check",
+ "kind": "required_behavior",
+ "text": "State one reproducible local check with its expected observation, grounded by the supplied fixture, and name a real remaining gap such as the absent broader specification or test suite."
+ },
+ {
+ "id": "diagnostic-report-restraint",
+ "kind": "hard_constraint",
+ "text": "Return exactly verdict, root_cause, evidence, verification, and unverified; do not patch, document, redesign, or otherwise mutate the workspace."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
+ {
+ "id": "software-surgical-plan",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/surgical-plan/prompt.md",
+ "fixture_dir": "cases/software/surgical-plan/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/surgical-plan/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "precise-production-step",
+ "focused-test-step",
+ "native-verification-command",
+ "proportional-plan-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "precise-production-step",
+ "kind": "required_behavior",
+ "text": "Plan the exact policy.py MAX_RETRIES change from 3 to 4 while preserving may_retry and its signature."
+ },
+ {
+ "id": "focused-test-step",
+ "kind": "required_behavior",
+ "text": "Plan the existing test_policy.py boundary update so attempt 3 is accepted and attempt 4 is rejected."
+ },
+ {
+ "id": "native-verification-command",
+ "kind": "required_behavior",
+ "text": "Name an executable Python unittest command targeting test_policy.py."
+ },
+ {
+ "id": "proportional-plan-restraint",
+ "kind": "hard_constraint",
+ "text": "Return exactly level, steps, verification, and non_goals at surgical scope; add no files, dependencies, unrelated work, or workspace edits."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
+ {
+ "id": "software-evidence-gap",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/evidence-gap/prompt.md",
+ "fixture_dir": "cases/software/evidence-gap/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/evidence-gap/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "balanced-source-facts",
+ "material-uncertainty",
+ "decision-resolving-check",
+ "research-output-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "balanced-source-facts",
+ "kind": "required_behavior",
+ "text": "Represent source-and-claim facts covering Q-17's 18% median result, its missing tail measurement, and the earlier build's 9% p99 regression."
+ },
+ {
+ "id": "material-uncertainty",
+ "kind": "required_behavior",
+ "text": "Keep the exact release identity and production-shaped p99 evidence as explicit unknowns."
+ },
+ {
+ "id": "decision-resolving-check",
+ "kind": "required_behavior",
+ "text": "Propose measuring p99 for the exact release candidate on the production-shaped workload, optionally against the current engine."
+ },
+ {
+ "id": "research-output-restraint",
+ "kind": "hard_constraint",
+ "text": "Do not recommend replacement yet; return exactly decision, supported_facts, unknowns, and next_check using only supplied sources, no confidence score, and no workspace edits."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
{
"id": "testing-oracle-sensitivity",
"skill": "testing",
diff --git a/tests/run_known_good_smoke.py b/tests/run_known_good_smoke.py
index d78bb24..6593854 100755
--- a/tests/run_known_good_smoke.py
+++ b/tests/run_known_good_smoke.py
@@ -28,11 +28,14 @@
}
-def _apply_known_good(case_id: str, workspace: Path) -> None:
+def _apply_known_good(case_id: str, workspace: Path) -> str:
if case_id.startswith("software-"):
case_directory = (
SUITE_ROOT / "cases" / "software" / case_id.removeprefix("software-")
)
+ artifact = case_directory / "calibration" / "good" / "artifact.json"
+ if artifact.is_file():
+ return artifact.read_text(encoding="utf-8")
apply_script = case_directory / "calibration" / "good" / "apply.py"
completed = subprocess.run(
[sys.executable, str(apply_script), str(workspace)],
@@ -47,13 +50,14 @@ def _apply_known_good(case_id: str, workspace: Path) -> None:
raise RuntimeError(
f"{case_id} known-good patch failed: {completed.stderr.strip()}"
)
- return
+ return "Applied the checked-in known-good calibration."
case_directory = SUITE_ROOT / "cases" / "testing" / case_id.removeprefix("testing-")
target = TEST_TARGETS[case_id]
shutil.copyfile(
case_directory / "calibration" / "good" / target,
workspace / target,
)
+ return "Applied the checked-in known-good calibration."
def main() -> int:
@@ -68,9 +72,9 @@ def main() -> int:
suite = load_suite(SUITE_ROOT / "suite.json")
def agent(request):
- _apply_known_good(request.case_id, request.workspace)
+ final_output = _apply_known_good(request.case_id, request.workspace)
return {
- "final_output": "Applied the checked-in known-good calibration.",
+ "final_output": final_output,
"actual_models": [request.model],
"cost_usd": 0.0,
"tokens": {"input_tokens": 0, "output_tokens": 0},
diff --git a/tests/test_calibrators.py b/tests/test_calibrators.py
index 69d9432..3605ab5 100644
--- a/tests/test_calibrators.py
+++ b/tests/test_calibrators.py
@@ -200,6 +200,61 @@ def test_good_variant_discovery_requires_executable_fixtures(self) -> None:
with self.assertRaisesRegex(AssertionError, "lack apply.py"):
SOFTWARE.discover_good_variants(root)
+ def test_final_output_variant_discovery_requires_artifacts(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ for name in ("adversarial/verbose", "bad", "good", "good-shaped"):
+ root.joinpath(name).mkdir(parents=True)
+ root.joinpath(name, "artifact.json").write_text("{}", encoding="utf-8")
+ self.assertEqual(
+ SOFTWARE.discover_good_variants(root, "final_output_json"),
+ ("good", "good-shaped"),
+ )
+ self.assertEqual(
+ SOFTWARE.discover_variants(root, "final_output_json"),
+ ("adversarial/verbose", "bad", "good", "good-shaped"),
+ )
+ root.joinpath("good-shaped", "artifact.json").unlink()
+ with self.assertRaisesRegex(AssertionError, "lack artifact.json"):
+ SOFTWARE.discover_good_variants(root, "final_output_json")
+
+ def test_workspace_fingerprint_detects_mode_and_content_changes(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary:
+ workspace = Path(temporary)
+ target = workspace / "value.txt"
+ target.write_text("first\n", encoding="utf-8")
+ initial = SOFTWARE.workspace_fingerprint(workspace)
+ workspace.chmod(workspace.stat().st_mode | stat.S_IWGRP)
+ after_root_mode = SOFTWARE.workspace_fingerprint(workspace)
+ target.chmod(target.stat().st_mode | stat.S_IXUSR)
+ after_mode = SOFTWARE.workspace_fingerprint(workspace)
+ target.write_text("second\n", encoding="utf-8")
+ after_content = SOFTWARE.workspace_fingerprint(workspace)
+ self.assertNotEqual(initial, after_root_mode)
+ self.assertNotEqual(after_root_mode, after_mode)
+ self.assertNotEqual(after_mode, after_content)
+
+ def test_workspace_fingerprint_frames_tree_entries(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ left = root / "left"
+ right = root / "right"
+ left.mkdir()
+ right.mkdir()
+ left_file = left / "a"
+ right_file = right / "a"
+ appended_file = right / "b"
+ right_file.write_bytes(b"")
+ appended_file.write_bytes(b"")
+ left_file.write_bytes(
+ b"b" + appended_file.lstat().st_mode.to_bytes(4, "big") + b"file\0"
+ )
+
+ self.assertNotEqual(
+ SOFTWARE.workspace_fingerprint(left),
+ SOFTWARE.workspace_fingerprint(right),
+ )
+
def test_expectation_opt_in_requires_every_variant(self) -> None:
variants = ("good", "bad", "adversarial/reflection")
with tempfile.TemporaryDirectory() as temporary:
@@ -222,11 +277,8 @@ def test_checked_in_expectation_corpora_are_complete(self) -> None:
if case["skill"] != "engineering":
continue
calibration_root = (SUITE_ROOT / case["prompt_file"]).parent / "calibration"
- variants = tuple(
- sorted(
- path.parent.relative_to(calibration_root).as_posix()
- for path in calibration_root.rglob("apply.py")
- )
+ variants = SOFTWARE.discover_variants(
+ calibration_root, case["artifact_contract"]["kind"]
)
if any(
calibration_root.joinpath(variant, "expect.json").is_file()
@@ -239,10 +291,59 @@ def test_checked_in_expectation_corpora_are_complete(self) -> None:
{
"software-behavioral-subtyping",
"software-domain-simplicity",
+ "software-evidence-gap",
"software-knowledge-boundary",
+ "software-compatibility-decision",
+ "software-root-cause-diagnosis",
"software-secure-archive-restore",
+ "software-surgical-plan",
+ },
+ )
+
+ def test_final_output_corpora_have_shape_and_mutation_sensitivity(self) -> None:
+ manifest = json.loads((SUITE_ROOT / "suite.json").read_text(encoding="utf-8"))
+ cases = {
+ case["id"]: case
+ for case in manifest["cases"]
+ if case["skill"] == "engineering"
+ and case["artifact_contract"]["kind"] == "final_output_json"
+ }
+ self.assertEqual(
+ set(cases),
+ {
+ "software-compatibility-decision",
+ "software-evidence-gap",
+ "software-root-cause-diagnosis",
+ "software-surgical-plan",
},
)
+ for case_id, case in cases.items():
+ with self.subTest(case_id=case_id):
+ calibration_root = (
+ SUITE_ROOT / case["prompt_file"]
+ ).parent / "calibration"
+ self.assertEqual(
+ len(
+ SOFTWARE.discover_good_variants(
+ calibration_root, "final_output_json"
+ )
+ ),
+ 7,
+ )
+ isolated_failures = set()
+ for variant in SOFTWARE.discover_variants(
+ calibration_root, "final_output_json"
+ ):
+ expectation = SOFTWARE.load_expectation(
+ calibration_root / variant / "expect.json"
+ )
+ self.assertIsNotNone(expectation, variant)
+ assert expectation is not None
+ if len(expectation["must_fail"]) == 1:
+ isolated_failures.update(expectation["must_fail"])
+ self.assertEqual(isolated_failures, set(case["critical_expectations"]))
+ mutation = calibration_root / "adversarial/workspace-edit"
+ self.assertTrue(mutation.joinpath("apply.py").is_file())
def test_secure_archive_expectations_cover_every_critical_assertion(self) -> None:
manifest = json.loads((SUITE_ROOT / "suite.json").read_text(encoding="utf-8"))
diff --git a/tests/test_codex_app_server.py b/tests/test_codex_app_server.py
index 1b8cf7a..ac5b4ab 100644
--- a/tests/test_codex_app_server.py
+++ b/tests/test_codex_app_server.py
@@ -5010,6 +5010,10 @@ def test_static_config_denies_credentials_and_command_network(self) -> None:
self.assertIn(f'"{self.runtime / "codex-home" / "auth.json"}" = "deny"', text)
self.assertIn(f'"{self.runtime / "codex-home" / "config.toml"}" = "deny"', text)
self.assertIn(f'HOME = "{self.runtime / "work" / ".skill-eval-home"}"', text)
+ self.assertEqual(
+ config["shell_environment_policy"]["set"]["PYTHONDONTWRITEBYTECODE"],
+ "1",
+ )
self.assertIn(f'"{self.runtime / "work"}" = "write"', text)
self.assertIn(f'"{self.runtime / "work" / ".skill-eval-tmp"}" = "write"', text)
self.assertIn(
diff --git a/tests/test_providers.py b/tests/test_providers.py
index a231263..f7d417e 100644
--- a/tests/test_providers.py
+++ b/tests/test_providers.py
@@ -714,6 +714,7 @@ def test_agent_uses_safe_stateless_budgeted_command_and_captures_exact_usage(
self.assertEqual(run.call_args_list[0].kwargs["timeout_seconds"], 12)
inner = command[command.index("--") + 1 :]
self.assertIn("CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1", inner)
+ self.assertIn("PYTHONDONTWRITEBYTECODE=1", inner)
def test_comparator_delegates_canonical_bytes_to_shared_runtime(self) -> None:
provider = ClaudeCliProvider(self.config())
diff --git a/tests/test_runner.py b/tests/test_runner.py
index 9bcc9c9..88c8aa0 100644
--- a/tests/test_runner.py
+++ b/tests/test_runner.py
@@ -2903,6 +2903,28 @@ def test_verifier_runs_in_disposable_copy_and_cannot_contaminate_agent_diff(
self.assertTrue(arm["verifier"]["workspace_mutated"])
self.assertNotIn("verifier-created.txt", arm["diff"])
+ def test_workspace_diff_excludes_generated_agent_caches(self) -> None:
+ provider = self.fixture.provider()
+ original_handler = provider._agent_handler
+
+ def create_cache(request):
+ result = original_handler(request)
+ cache = request.workspace / "__pycache__"
+ cache.mkdir()
+ (cache / "note.pyc").write_bytes(b"generated cache")
+ return result
+
+ provider._agent_handler = create_cache
+ result = self.runner(provider).run(
+ RunSelection(comparison_ids=("without-current",)),
+ output_dir=self.output("agent-cache-diff"),
+ )
+
+ self.assertTrue(result["passed"], result)
+ for arm in result["pairs"][0]["arms"].values():
+ self.assertTrue(arm["verifier"]["sandbox"]["agent_workspace_mutated"])
+ self.assertNotIn("__pycache__", arm["diff"])
+
def test_run_when_artifact_is_final_text_keeps_workspace_read_only(self) -> None:
# Arrange
self.fixture.use_objective()
@@ -3038,6 +3060,7 @@ def test_run_when_artifact_is_final_json_completes_without_comparator(self) -> N
passed = (
content == b'{"a":1,"b":2}'
and os.environ["EVAL_ARTIFACT_KIND"] == "final_output_json"
+ and os.environ["EVAL_AGENT_WORKSPACE_MUTATED"] == "1"
and os.environ["EVAL_ARTIFACT_SHA256"] == hashlib.sha256(content).hexdigest()
and (workspace / "input.txt").read_text(encoding="utf-8") == "original\\n"
and not (workspace / "poison.txt").exists()
@@ -3085,8 +3108,117 @@ def json_output(request):
)
self.assertTrue(arm["verifier"]["artifact"]["read_only"])
self.assertTrue(arm["verifier"]["sandbox"]["workspace_read_only"])
+ self.assertTrue(arm["verifier"]["sandbox"]["agent_workspace_mutated"])
self.assertFalse(arm["verifier"]["workspace_mutated"])
+ def test_final_output_mutation_signal_covers_all_workspace_entries(
+ self,
+ ) -> None:
+ self.fixture.use_objective(artifact_kind="final_output_json")
+ self.fixture.set_verifier(
+ """import json
+import os
+
+passed = os.environ["EVAL_AGENT_WORKSPACE_MUTATED"] == "1"
+print(json.dumps({
+ "passed": passed,
+ "assertions": [{
+ "id": "answer-present",
+ "passed": passed,
+ "evidence": "agent workspace mutation signal",
+ }],
+ "metrics": {},
+}))
+"""
+ )
+ self.fixture.save_manifest()
+
+ def create_empty_directory(workspace: Path) -> None:
+ (workspace / "empty").mkdir()
+
+ def remove_owner_write_permission(workspace: Path) -> None:
+ (workspace / "input.txt").chmod(0o444)
+
+ def create_cache_file(workspace: Path) -> None:
+ cache = workspace / "__pycache__"
+ cache.mkdir()
+ (cache / "note.txt").write_text("agent-created\n", encoding="utf-8")
+
+ def mutate_then_restore_file(workspace: Path) -> None:
+ target = workspace / "input.txt"
+ original = target.read_bytes()
+ target.write_bytes(b"transient mutation\n")
+ target.write_bytes(original)
+
+ mutations = (
+ ("empty-directory", create_empty_directory),
+ ("file-permission", remove_owner_write_permission),
+ ("cache-file", create_cache_file),
+ ("transient-write", mutate_then_restore_file),
+ )
+ for name, mutate in mutations:
+ with self.subTest(name=name):
+
+ def json_output(request):
+ mutate(request.workspace)
+ return '{"a": 1}'
+
+ result = EvalRunner(
+ self.load(), FakeProvider(agent_handler=json_output)
+ ).run(
+ RunSelection(comparison_ids=("without-current",)),
+ output_dir=self.output(f"final-json-{name}"),
+ )
+
+ self.assertTrue(result["passed"], result)
+ for arm in result["pairs"][0]["arms"].values():
+ self.assertTrue(
+ arm["verifier"]["sandbox"]["agent_workspace_mutated"]
+ )
+
+ def test_final_output_without_workspace_mutation_reports_false(self) -> None:
+ self.fixture.use_objective(artifact_kind="final_output_json")
+ self.fixture.set_verifier(
+ """import json
+import os
+
+passed = os.environ["EVAL_AGENT_WORKSPACE_MUTATED"] == "0"
+print(json.dumps({
+ "passed": passed,
+ "assertions": [{
+ "id": "answer-present",
+ "passed": passed,
+ "evidence": "agent workspace remained untouched",
+ }],
+ "metrics": {},
+}))
+"""
+ )
+ self.fixture.save_manifest()
+
+ def use_provider_runtime_scratch(request):
+ for name in (
+ ".skill-eval-cache",
+ ".skill-eval-home",
+ ".skill-eval-tmp",
+ ):
+ scratch = request.workspace / name
+ scratch.mkdir()
+ (scratch / "runtime").write_text("transient\n", encoding="utf-8")
+ shutil.rmtree(scratch)
+ return '{"a": 1}'
+
+ result = EvalRunner(
+ self.load(), FakeProvider(agent_handler=use_provider_runtime_scratch)
+ ).run(
+ RunSelection(comparison_ids=("without-current",)),
+ output_dir=self.output("final-json-no-mutation"),
+ )
+
+ self.assertTrue(result["passed"], result)
+ for arm in result["pairs"][0]["arms"].values():
+ self.assertFalse(arm["verifier"]["sandbox"]["agent_workspace_mutated"])
+
def test_preflight_when_final_output_is_judged_requires_calibrated_profile(
self,
) -> None:
@@ -3104,6 +3236,15 @@ def test_preflight_when_final_output_is_judged_requires_calibrated_profile(
self.assertEqual(provider.agent_requests, [])
self.assertEqual(provider.comparator_requests, [])
+ verifier_only = runner.preflight(
+ RunSelection(
+ comparison_ids=("without-current",),
+ verifier_only=True,
+ )
+ )
+ self.assertEqual(verifier_only["execution_mode"], "verifier_only")
+ self.assertIsNone(verifier_only["comparator"])
+
# Arrange
for case in self.fixture.manifest["cases"]:
case["artifact_contract"] = {"kind": "workspace_diff"}
@@ -3381,6 +3522,7 @@ def hidden(path):
"EVAL_ARTIFACT_PATH",
"EVAL_ARTIFACT_KIND",
"EVAL_ARTIFACT_SHA256",
+ "EVAL_AGENT_WORKSPACE_MUTATED",
"EVAL_CASE_ROOT",
"EVAL_TOOL_BIN",
"EVAL_RESULT_ROOT",
@@ -8051,9 +8193,9 @@ def test_gcc_attestation_includes_derived_driver_closure(self) -> None:
def test_manifest_is_loadable_and_public_cases_are_not_holdouts(self) -> None:
suite = load_suite(HARNESS_ROOT / "suite.json")
splits = [case.split for case in suite.cases]
- self.assertEqual(len(suite.cases), 17)
+ self.assertEqual(len(suite.cases), 21)
self.assertEqual(splits.count("train"), 10)
- self.assertEqual(splits.count("validation"), 7)
+ self.assertEqual(splits.count("validation"), 11)
self.assertNotIn("holdout", splits)
def test_models_and_frozen_original_are_pinned(self) -> None:
Validation
Knowledge boundary
Authority, locality, and duplication decisions under change pressure.
Validation
Domain simplicity
Direct design and earned complexity for domain behavior.
Validation
Compatibility decision
Public-contract conflicts, migration ownership, and safe ask-before-change restraint.
Validation
Root-cause diagnosis
Concrete causal evidence, reproducible checks, and honest remaining gaps.
Validation
Surgical plan
Proportional implementation scope, precise boundaries, and native verification.
Validation
Evidence gap
Source-bounded decisions, material counter-evidence, and resolving next checks.
-
diff --git a/skivolve/codex_app_server.py b/skivolve/codex_app_server.py
index d9a82f0..d46678d 100644
--- a/skivolve/codex_app_server.py
+++ b/skivolve/codex_app_server.py
@@ -4400,7 +4400,7 @@ def _static_config(root: Path, model: str, reasoning_effort: str) -> bytes:
"",
"[shell_environment_policy]",
'inherit = "none"',
- f'set = {{ PATH = {_toml_string(f"{tools}/bin:{tools}/codex-path:{tools}/codex-resources:{tools}/codex-resources/zsh/bin:{tools}/required:/usr/bin:/bin")}, HOME = {_toml_string(str(home_directory))}, LANG = "C.UTF-8", TMPDIR = {_toml_string(str(temp_directory))}, XDG_CACHE_HOME = {_toml_string(str(cache_directory))} }}',
+ f'set = {{ PATH = {_toml_string(f"{tools}/bin:{tools}/codex-path:{tools}/codex-resources:{tools}/codex-resources/zsh/bin:{tools}/required:/usr/bin:/bin")}, HOME = {_toml_string(str(home_directory))}, LANG = "C.UTF-8", PYTHONDONTWRITEBYTECODE = "1", TMPDIR = {_toml_string(str(temp_directory))}, XDG_CACHE_HOME = {_toml_string(str(cache_directory))} }}',
"",
"[permissions.eval]",
'description = "Isolated Skivolve execution"',
diff --git a/skivolve/comparator-profile-authority.json b/skivolve/comparator-profile-authority.json
index 7a8aefc..100ebd9 100644
--- a/skivolve/comparator-profile-authority.json
+++ b/skivolve/comparator-profile-authority.json
@@ -4,8 +4,8 @@
{
"id": "plain-language-revision-v1",
"descriptor_sha256": "60806e964d6c415d1ebb007976acedd1286a2ec6dd788442e58f345cbbd26b30",
- "production_release_sha256": "94be3ed9ceb4ffe4316beef141cf8d3fa38e9aec01ae4a6ec47be3d73d444739",
- "test_release_sha256": "fea628d77f9b3509dfd9c8a5b113d9bf93fed4be911aa92cdee81e948b338863",
+ "production_release_sha256": "da012be22caeb1c76b37b181537a8f4e061419b35c520bde6d4dce484a8c9c40",
+ "test_release_sha256": "fdf61ed4ff33650a28c4431b98e8553205fb0c3aafc457844efcaad6f4c0958b",
"certification_contract_sha256": "48acdc90136e871ab9cea000c76f2ec94c517ed4466d6cba5beb57f5b4a9c265",
"requires_live_certification": true,
"authority_scope": "test"
@@ -13,8 +13,8 @@
{
"id": "software-engineering-v1",
"descriptor_sha256": "6a166ed3a313ee145d00519382c93c9fb56223aaf9b80f29806428f53f505e35",
- "production_release_sha256": "c07e78f8f5d3bd64f806ca924424c58d94753b521550c8b39bf0c51bd22efc89",
- "test_release_sha256": "866e2fe4a3a1bfd53ec3c020d6583d92b746a97c4a833a0a580a5ff5ecf7808c",
+ "production_release_sha256": "177f65fe98ea11ca11e9f5998070ff4c5d9a5233ba362d97e43356422c6f01f0",
+ "test_release_sha256": "ba789cdad72490699914d2046565e8a2fc4132252e31b2d8c5e4e279ffd2554e",
"certification_contract_sha256": "48acdc90136e871ab9cea000c76f2ec94c517ed4466d6cba5beb57f5b4a9c265",
"requires_live_certification": true,
"authority_scope": "production"
diff --git a/skivolve/comparator_calibration/release.json b/skivolve/comparator_calibration/release.json
index 3954158..6c4fe27 100644
--- a/skivolve/comparator_calibration/release.json
+++ b/skivolve/comparator_calibration/release.json
@@ -178,9 +178,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/comparator_calibration/tests/test-release.json b/skivolve/comparator_calibration/tests/test-release.json
index c3078b1..026eb1b 100644
--- a/skivolve/comparator_calibration/tests/test-release.json
+++ b/skivolve/comparator_calibration/tests/test-release.json
@@ -160,9 +160,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/plain_language_calibration/release.json b/skivolve/plain_language_calibration/release.json
index ca153cf..451d65f 100644
--- a/skivolve/plain_language_calibration/release.json
+++ b/skivolve/plain_language_calibration/release.json
@@ -160,9 +160,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/plain_language_calibration/tests/test-release.json b/skivolve/plain_language_calibration/tests/test-release.json
index fece2ef..1065a10 100644
--- a/skivolve/plain_language_calibration/tests/test-release.json
+++ b/skivolve/plain_language_calibration/tests/test-release.json
@@ -142,9 +142,9 @@
"runtime_adapter": {
"id": "shared-harness-claude-cli-v1",
"source_sha256": "2ace70a6a941a4078a83aa198d05b3344f5f2add72f99278182fe394de1f575c",
- "harness_runner_source_sha256": "62edb2c1bb7163cf871e77bdbcccd66e8f9b771a8959d123db9ae54e628961c6",
+ "harness_runner_source_sha256": "17bfea40e8079f12c5e919a992edc2bc7588289d87db6036db7926ea18393aa1",
"artifact_normalizer_source_sha256": "03d1895393718f4882f17b8a01347e45723b4ae634116319f746e8786ea8d688",
- "provider_source_sha256": "7e79122bb2b68ec62b3fbf9239ffe0526e4f6870802fa695c3b04cee72a43729",
+ "provider_source_sha256": "6eae3836780ec6bd2abfeb16e04555999879253f2aa82b8319a484ed60159900",
"profile_registry_source_sha256": "1582fa2eaebdf3c645f16918e2d5a62a279c176f55c1451d81aa48072b27648a",
"provider_capability_registry_source_sha256": "90871a063abcd1061c4d336589667837dd2d07ecfabcb69aa642ed5db283c07e",
"harness_manifest_source_sha256": "2d2924e7168a789cfa37ba513a30b4ceef28135d830127e833c7d3e1dc5c0569",
diff --git a/skivolve/providers.py b/skivolve/providers.py
index d40c9ef..5c15c6a 100644
--- a/skivolve/providers.py
+++ b/skivolve/providers.py
@@ -1512,6 +1512,7 @@ def _inner_prefix(
"SHELL=/bin/bash",
"TERM=dumb",
"CI=1",
+ "PYTHONDONTWRITEBYTECODE=1",
"CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1",
self._unshare,
"--user",
diff --git a/skivolve/runner.py b/skivolve/runner.py
index db46ff0..25ca2fb 100644
--- a/skivolve/runner.py
+++ b/skivolve/runner.py
@@ -3,6 +3,7 @@
from __future__ import annotations
import concurrent.futures
+import ctypes
import datetime as dt
import difflib
import fcntl
@@ -14,6 +15,7 @@
import select
import shutil
import stat
+import struct
import subprocess
import tempfile
import threading
@@ -133,6 +135,110 @@ class _GeneratorDispatchJournalError(RunnerError):
"""Raised when generator dispatch accounting cannot remain trustworthy."""
+class _WorkspaceMutationMonitor:
+ """Observe successful write operations even when final bytes are restored."""
+
+ _MASK = 0x00000002 | 0x00000004 | 0x00000008 | 0x00000040 | 0x00000080
+ _MASK |= 0x00000100 | 0x00000200 | 0x00000400 | 0x00000800
+ _EVENT = struct.Struct("=iIII")
+ _PROVIDER_RUNTIME_ENTRIES = frozenset(
+ {".skill-eval-cache", ".skill-eval-home", ".skill-eval-tmp"}
+ )
+
+ def __init__(self, root: Path) -> None:
+ libc = ctypes.CDLL(None, use_errno=True)
+ initialize = libc.inotify_init1
+ initialize.argtypes = (ctypes.c_int,)
+ initialize.restype = ctypes.c_int
+ add_watch = libc.inotify_add_watch
+ add_watch.argtypes = (ctypes.c_int, ctypes.c_char_p, ctypes.c_uint32)
+ add_watch.restype = ctypes.c_int
+ descriptor = initialize(os.O_CLOEXEC | os.O_NONBLOCK)
+ if descriptor < 0:
+ error = ctypes.get_errno()
+ raise RunnerError(
+ f"cannot monitor agent workspace writes: {os.strerror(error)}"
+ )
+ self._descriptor = descriptor
+ self._watched_directories: dict[int, Path] = {}
+ try:
+ for current, directories, _files in os.walk(root, followlinks=False):
+ current_path = Path(current)
+ watch_descriptor = add_watch(
+ descriptor,
+ ctypes.c_char_p(os.fsencode(current_path)),
+ self._MASK,
+ )
+ if watch_descriptor < 0:
+ error = ctypes.get_errno()
+ raise RunnerError(
+ "cannot monitor agent workspace directory "
+ f"{current_path}: {os.strerror(error)}"
+ )
+ self._watched_directories[watch_descriptor] = current_path.relative_to(
+ root
+ )
+ for name in directories:
+ path = current_path / name
+ if path.is_symlink() or not path.is_dir():
+ raise RunnerError(
+ f"agent workspace contains unsafe directory: {path}"
+ )
+ except BaseException:
+ self.close()
+ raise
+
+ def discard_pending(self) -> None:
+ self._drain()
+
+ def observed(self) -> bool:
+ return self._drain()
+
+ def _drain(self) -> bool:
+ observed = False
+ while True:
+ try:
+ chunk = os.read(self._descriptor, 64 * 1024)
+ except BlockingIOError:
+ return observed
+ except InterruptedError:
+ continue
+ except OSError as exc:
+ raise RunnerError(
+ f"cannot read agent workspace mutation evidence: {exc}"
+ ) from exc
+ if not chunk:
+ raise RunnerError(
+ "agent workspace mutation monitor closed unexpectedly"
+ )
+ offset = 0
+ while offset < len(chunk):
+ if len(chunk) - offset < self._EVENT.size:
+ raise RunnerError("agent workspace mutation event was truncated")
+ watch, _mask, _cookie, name_bytes = self._EVENT.unpack_from(
+ chunk, offset
+ )
+ offset += self._EVENT.size
+ end = offset + name_bytes
+ if end > len(chunk):
+ raise RunnerError("agent workspace mutation name was truncated")
+ name = os.fsdecode(chunk[offset:end].split(b"\0", 1)[0])
+ offset = end
+ relative = self._watched_directories.get(watch)
+ provider_runtime_event = (
+ relative is not None
+ and not relative.parts
+ and name in self._PROVIDER_RUNTIME_ENTRIES
+ )
+ if not provider_runtime_event:
+ observed = True
+
+ def close(self) -> None:
+ if self._descriptor >= 0:
+ os.close(self._descriptor)
+ self._descriptor = -1
+
+
_GENERATOR_DISPATCH_JOURNAL = "generator-dispatch.jsonl"
_GENERATOR_DISPATCH_LOCK = "generator-dispatch.lock"
_GENERATOR_DISPATCH_SCHEMA_VERSION = 1
@@ -1817,7 +1923,11 @@ def _preflight(
selection, allow_unsealed_holdout=allow_unsealed_holdout
)
self._assert_injected_fake_generator_admissible(selection, cases)
- if runtime is None and self.suite.evaluation_mode != "objective_only":
+ if (
+ runtime is None
+ and self.suite.evaluation_mode != "objective_only"
+ and not selection.verifier_only
+ ):
runtime = self._load_comparator_runtime()
generator_artifact_kinds = set(
capabilities_for(
@@ -1836,7 +1946,7 @@ def _preflight(
"generator adapter does not support selected artifact kinds: "
+ ", ".join(unsupported_generator_artifacts)
)
- if runtime is not None:
+ if runtime is not None and not selection.verifier_only:
unsupported_profile_artifacts = sorted(
{
case.artifact_contract.kind
@@ -2090,7 +2200,7 @@ def run(
)
return dry_run_result
runtime: ComparatorRuntime | None = None
- if self.suite.evaluation_mode == "judged":
+ if self.suite.evaluation_mode == "judged" and not selection.verifier_only:
runtime = self._load_comparator_runtime()
if runtime is not None and not selection.verifier_only:
assert self.suite.comparator is not None
@@ -3444,12 +3554,15 @@ def _run_arm(
provider_journal_state: str | None = None
provider_entered = False
synchronization_complete = False
+ workspace_mutation_observed = False
+ workspace_mutation_monitor: _WorkspaceMutationMonitor | None = None
normalized_artifact: NormalizedArtifact | None = None
stage = "fixture_copy"
try:
self._assert_case_integrity(case)
_copy_tree(case.fixture_dir, workspace, ignore_generated_caches=True)
before = _read_tree(workspace, ignore_generated_caches=True)
+ before_permissions = _read_tree_permissions(workspace)
hashes["fixture_before_sha256"] = _states_hash(before)
stage = "source_materialization"
source = self._materialize_source(variant, case, temp_root / "source")
@@ -3457,7 +3570,7 @@ def _run_arm(
hashes["context_sha256"] = source.context_hash
system_context = _system_context(case, source)
- def account_dispatch() -> None:
+ def account_dispatch(*, provider_callback: bool = True) -> None:
if provider_attempt_id is None:
raise _GeneratorDispatchJournalError(
"generator dispatch callback preceded durable planning"
@@ -3467,6 +3580,8 @@ def account_dispatch() -> None:
raise _GeneratorDispatchJournalError(
"generator dispatch ledger was not initialized"
)
+ if provider_callback and workspace_mutation_monitor is not None:
+ workspace_mutation_monitor.discard_pending()
ledger.mark_dispatched(provider_attempt_id)
provider_dispatched.set()
@@ -3518,11 +3633,13 @@ def account_dispatch() -> None:
provider_window["started"] = time.monotonic()
synchronization_complete = True
stage = "agent"
+ if case.artifact_contract.kind != "workspace_diff":
+ workspace_mutation_monitor = _WorkspaceMutationMonitor(workspace)
try:
provider_entered = True
provider_result = self.agent_provider.run_agent(request)
if not provider_dispatched.is_set():
- account_dispatch()
+ account_dispatch(provider_callback=False)
provider_journal_state = "dispatched"
if case.artifact_contract.kind != "workspace_diff":
stage = "artifact_normalization"
@@ -3547,6 +3664,13 @@ def account_dispatch() -> None:
provider_journal_state = "completed"
finally:
provider_window["finished"] = time.monotonic()
+ if workspace_mutation_monitor is not None:
+ try:
+ workspace_mutation_observed = (
+ workspace_mutation_monitor.observed()
+ )
+ finally:
+ workspace_mutation_monitor.close()
if source is not None and source.snapshot is not None:
_make_tree_writable(source.snapshot)
hashes["agent_output_sha256"] = _sha256(
@@ -3557,6 +3681,13 @@ def account_dispatch() -> None:
)
stage = "agent_workspace_scan"
after_agent = _read_tree(workspace, ignore_generated_caches=True)
+ after_agent_mutation = _read_tree(workspace)
+ after_agent_permissions = _read_tree_permissions(workspace)
+ agent_workspace_mutated = (
+ before != after_agent_mutation
+ or before_permissions != after_agent_permissions
+ or workspace_mutation_observed
+ )
hashes["workspace_after_agent_sha256"] = _states_hash(after_agent)
diff_text = _diff_states(before, after_agent)
hashes["diff_sha256"] = _sha256(diff_text.encode("utf-8"))
@@ -3588,6 +3719,7 @@ def account_dispatch() -> None:
normalized_artifact,
result_root,
workspace_read_only=final_output_artifact,
+ agent_workspace_mutated=agent_workspace_mutated,
)
verifier_after_hash = _tree_hash(verifier_workspace)
verifier_json["workspace_before_sha256"] = verifier_before_hash
@@ -3798,6 +3930,7 @@ def _run_verifier(
result_root: Path,
*,
workspace_read_only: bool,
+ agent_workspace_mutated: bool,
) -> dict[str, Any]:
command = self._verifier_commands.get(case.id)
if command is None:
@@ -3945,6 +4078,7 @@ def _run_verifier(
f"EVAL_ARTIFACT_PATH={mounted_artifact}",
f"EVAL_ARTIFACT_KIND={artifact.kind}",
f"EVAL_ARTIFACT_SHA256={artifact.sha256}",
+ f"EVAL_AGENT_WORKSPACE_MUTATED={int(agent_workspace_mutated)}",
f"EVAL_CASE_ROOT={mounted_case}",
*(
[f"EVAL_SHARED_ROOT={mounted_shared}"]
@@ -4001,6 +4135,7 @@ def _run_verifier(
"read_only": True,
},
"workspace_read_only": workspace_read_only,
+ "agent_workspace_mutated": agent_workspace_mutated,
"go_root": str(go_root) if go_root is not None else None,
"gcc_exec_prefix": (
str(gcc_exec_prefix) if gcc_exec_prefix is not None else None
@@ -6104,6 +6239,7 @@ def _scan_tree(
*,
ignore_generated_caches: bool = False,
ignore_empty_directories: bool = False,
+ include_directories: bool = False,
) -> list[Path]:
if not root.is_dir() or root.is_symlink():
raise RunnerError(f"tree root must be a regular directory: {root}")
@@ -6111,7 +6247,7 @@ def _scan_tree(
return _scan_normalized_worktree(
root, ignore_generated_caches=ignore_generated_caches
)
- files: list[Path] = []
+ paths: list[Path] = []
total = 0
entries = 0
for current, directories, filenames in os.walk(root, followlinks=False):
@@ -6132,6 +6268,8 @@ def _scan_tree(
f"tree exceeds maximum entries {MAX_TREE_ENTRIES}: {root}"
)
retained_directories.append(name)
+ if include_directories:
+ paths.append(path)
directories[:] = retained_directories
for name in filenames:
path = current_path / name
@@ -6151,8 +6289,8 @@ def _scan_tree(
total += size
if total > MAX_TREE_BYTES:
raise RunnerError(f"tree exceeds {MAX_TREE_BYTES} bytes: {root}")
- files.append(path)
- return sorted(files, key=lambda path: path.relative_to(root).as_posix())
+ paths.append(path)
+ return sorted(paths, key=lambda path: path.relative_to(root).as_posix())
def _read_tree(
@@ -6173,6 +6311,23 @@ def _read_tree(
return states
+def _read_tree_permissions(
+ root: Path,
+ *,
+ ignore_generated_caches: bool = False,
+) -> dict[str, int]:
+ permissions = {".": stat.S_IMODE(root.stat().st_mode)}
+ for path in _scan_tree(
+ root,
+ ignore_generated_caches=ignore_generated_caches,
+ include_directories=True,
+ ):
+ permissions[path.relative_to(root).as_posix()] = stat.S_IMODE(
+ path.stat().st_mode
+ )
+ return permissions
+
+
def _copy_tree(
source: Path,
destination: Path,
diff --git a/suite.json b/suite.json
index 2744bc3..94553c2 100644
--- a/suite.json
+++ b/suite.json
@@ -665,6 +665,226 @@
"qualitative_bases": {}
}
},
+ {
+ "id": "software-compatibility-decision",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/compatibility-decision/prompt.md",
+ "fixture_dir": "cases/software/compatibility-decision/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/compatibility-decision/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "pointed-public-choice",
+ "safe-default-recommendation",
+ "consumer-conflict-grounding",
+ "ask-and-stop-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "pointed-public-choice",
+ "kind": "required_behavior",
+ "text": "Ask one concrete either/or question that contrasts preserving the public v3 field names with authorizing a legacy-name migration."
+ },
+ {
+ "id": "safe-default-recommendation",
+ "kind": "required_behavior",
+ "text": "Recommend preserving the current v3 names until an owner decides the migration."
+ },
+ {
+ "id": "consumer-conflict-grounding",
+ "kind": "required_behavior",
+ "text": "Ground the block in North's deployed v3 dependency, South's legacy-name dependency, the public wire contract, and the missing migration owner or date."
+ },
+ {
+ "id": "ask-and-stop-restraint",
+ "kind": "hard_constraint",
+ "text": "Return exactly status, question, recommendation, reason, and actions_taken; record that no change was made and leave the workspace unchanged."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
+ {
+ "id": "software-root-cause-diagnosis",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/root-cause-diagnosis/prompt.md",
+ "fixture_dir": "cases/software/root-cause-diagnosis/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/root-cause-diagnosis/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "concrete-root-cause",
+ "minimal-line-evidence",
+ "reproducible-verification-check",
+ "diagnostic-report-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "concrete-root-cause",
+ "kind": "required_behavior",
+ "text": "Identify that the inclusive nonnegative condition classifies attempts=0 as ok instead of retry."
+ },
+ {
+ "id": "minimal-line-evidence",
+ "kind": "required_behavior",
+ "text": "Cite the failing observation at failure.log:2 and the causal condition at worker.py:2 using two or three bounded evidence entries."
+ },
+ {
+ "id": "reproducible-verification-check",
+ "kind": "required_behavior",
+ "text": "State one reproducible local check with its expected observation, grounded by the supplied fixture, and name a real remaining gap such as the absent broader specification or test suite."
+ },
+ {
+ "id": "diagnostic-report-restraint",
+ "kind": "hard_constraint",
+ "text": "Return exactly verdict, root_cause, evidence, verification, and unverified; do not patch, document, redesign, or otherwise mutate the workspace."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
+ {
+ "id": "software-surgical-plan",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/surgical-plan/prompt.md",
+ "fixture_dir": "cases/software/surgical-plan/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/surgical-plan/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "precise-production-step",
+ "focused-test-step",
+ "native-verification-command",
+ "proportional-plan-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "precise-production-step",
+ "kind": "required_behavior",
+ "text": "Plan the exact policy.py MAX_RETRIES change from 3 to 4 while preserving may_retry and its signature."
+ },
+ {
+ "id": "focused-test-step",
+ "kind": "required_behavior",
+ "text": "Plan the existing test_policy.py boundary update so attempt 3 is accepted and attempt 4 is rejected."
+ },
+ {
+ "id": "native-verification-command",
+ "kind": "required_behavior",
+ "text": "Name an executable Python unittest command targeting test_policy.py."
+ },
+ {
+ "id": "proportional-plan-restraint",
+ "kind": "hard_constraint",
+ "text": "Return exactly level, steps, verification, and non_goals at surgical scope; add no files, dependencies, unrelated work, or workspace edits."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
+ {
+ "id": "software-evidence-gap",
+ "skill": "engineering",
+ "bundle_source": "skills/engineering",
+ "artifact_contract": {
+ "kind": "final_output_json"
+ },
+ "split": "validation",
+ "prompt_file": "cases/software/evidence-gap/prompt.md",
+ "fixture_dir": "cases/software/evidence-gap/fixture",
+ "verifier": {
+ "argv": [
+ "python3",
+ "cases/software/evidence-gap/oracle/verify.py"
+ ],
+ "required_tools": [],
+ "timeout_seconds": 10
+ },
+ "context_files": [
+ "skills/engineering/SKILL.md"
+ ],
+ "timeout_seconds": 180,
+ "critical_expectations": [
+ "balanced-source-facts",
+ "material-uncertainty",
+ "decision-resolving-check",
+ "research-output-restraint"
+ ],
+ "comparator_contract": {
+ "requirements": [
+ {
+ "id": "balanced-source-facts",
+ "kind": "required_behavior",
+ "text": "Represent source-and-claim facts covering Q-17's 18% median result, its missing tail measurement, and the earlier build's 9% p99 regression."
+ },
+ {
+ "id": "material-uncertainty",
+ "kind": "required_behavior",
+ "text": "Keep the exact release identity and production-shaped p99 evidence as explicit unknowns."
+ },
+ {
+ "id": "decision-resolving-check",
+ "kind": "required_behavior",
+ "text": "Propose measuring p99 for the exact release candidate on the production-shaped workload, optionally against the current engine."
+ },
+ {
+ "id": "research-output-restraint",
+ "kind": "hard_constraint",
+ "text": "Do not recommend replacement yet; return exactly decision, supported_facts, unknowns, and next_check using only supplied sources, no confidence score, and no workspace edits."
+ }
+ ],
+ "performance_basis": null,
+ "qualitative_bases": {}
+ }
+ },
{
"id": "testing-oracle-sensitivity",
"skill": "testing",
diff --git a/tests/run_known_good_smoke.py b/tests/run_known_good_smoke.py
index d78bb24..6593854 100755
--- a/tests/run_known_good_smoke.py
+++ b/tests/run_known_good_smoke.py
@@ -28,11 +28,14 @@
}
-def _apply_known_good(case_id: str, workspace: Path) -> None:
+def _apply_known_good(case_id: str, workspace: Path) -> str:
if case_id.startswith("software-"):
case_directory = (
SUITE_ROOT / "cases" / "software" / case_id.removeprefix("software-")
)
+ artifact = case_directory / "calibration" / "good" / "artifact.json"
+ if artifact.is_file():
+ return artifact.read_text(encoding="utf-8")
apply_script = case_directory / "calibration" / "good" / "apply.py"
completed = subprocess.run(
[sys.executable, str(apply_script), str(workspace)],
@@ -47,13 +50,14 @@ def _apply_known_good(case_id: str, workspace: Path) -> None:
raise RuntimeError(
f"{case_id} known-good patch failed: {completed.stderr.strip()}"
)
- return
+ return "Applied the checked-in known-good calibration."
case_directory = SUITE_ROOT / "cases" / "testing" / case_id.removeprefix("testing-")
target = TEST_TARGETS[case_id]
shutil.copyfile(
case_directory / "calibration" / "good" / target,
workspace / target,
)
+ return "Applied the checked-in known-good calibration."
def main() -> int:
@@ -68,9 +72,9 @@ def main() -> int:
suite = load_suite(SUITE_ROOT / "suite.json")
def agent(request):
- _apply_known_good(request.case_id, request.workspace)
+ final_output = _apply_known_good(request.case_id, request.workspace)
return {
- "final_output": "Applied the checked-in known-good calibration.",
+ "final_output": final_output,
"actual_models": [request.model],
"cost_usd": 0.0,
"tokens": {"input_tokens": 0, "output_tokens": 0},
diff --git a/tests/test_calibrators.py b/tests/test_calibrators.py
index 69d9432..3605ab5 100644
--- a/tests/test_calibrators.py
+++ b/tests/test_calibrators.py
@@ -200,6 +200,61 @@ def test_good_variant_discovery_requires_executable_fixtures(self) -> None:
with self.assertRaisesRegex(AssertionError, "lack apply.py"):
SOFTWARE.discover_good_variants(root)
+ def test_final_output_variant_discovery_requires_artifacts(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ for name in ("adversarial/verbose", "bad", "good", "good-shaped"):
+ root.joinpath(name).mkdir(parents=True)
+ root.joinpath(name, "artifact.json").write_text("{}", encoding="utf-8")
+ self.assertEqual(
+ SOFTWARE.discover_good_variants(root, "final_output_json"),
+ ("good", "good-shaped"),
+ )
+ self.assertEqual(
+ SOFTWARE.discover_variants(root, "final_output_json"),
+ ("adversarial/verbose", "bad", "good", "good-shaped"),
+ )
+ root.joinpath("good-shaped", "artifact.json").unlink()
+ with self.assertRaisesRegex(AssertionError, "lack artifact.json"):
+ SOFTWARE.discover_good_variants(root, "final_output_json")
+
+ def test_workspace_fingerprint_detects_mode_and_content_changes(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary:
+ workspace = Path(temporary)
+ target = workspace / "value.txt"
+ target.write_text("first\n", encoding="utf-8")
+ initial = SOFTWARE.workspace_fingerprint(workspace)
+ workspace.chmod(workspace.stat().st_mode | stat.S_IWGRP)
+ after_root_mode = SOFTWARE.workspace_fingerprint(workspace)
+ target.chmod(target.stat().st_mode | stat.S_IXUSR)
+ after_mode = SOFTWARE.workspace_fingerprint(workspace)
+ target.write_text("second\n", encoding="utf-8")
+ after_content = SOFTWARE.workspace_fingerprint(workspace)
+ self.assertNotEqual(initial, after_root_mode)
+ self.assertNotEqual(after_root_mode, after_mode)
+ self.assertNotEqual(after_mode, after_content)
+
+ def test_workspace_fingerprint_frames_tree_entries(self) -> None:
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ left = root / "left"
+ right = root / "right"
+ left.mkdir()
+ right.mkdir()
+ left_file = left / "a"
+ right_file = right / "a"
+ appended_file = right / "b"
+ right_file.write_bytes(b"")
+ appended_file.write_bytes(b"")
+ left_file.write_bytes(
+ b"b" + appended_file.lstat().st_mode.to_bytes(4, "big") + b"file\0"
+ )
+
+ self.assertNotEqual(
+ SOFTWARE.workspace_fingerprint(left),
+ SOFTWARE.workspace_fingerprint(right),
+ )
+
def test_expectation_opt_in_requires_every_variant(self) -> None:
variants = ("good", "bad", "adversarial/reflection")
with tempfile.TemporaryDirectory() as temporary:
@@ -222,11 +277,8 @@ def test_checked_in_expectation_corpora_are_complete(self) -> None:
if case["skill"] != "engineering":
continue
calibration_root = (SUITE_ROOT / case["prompt_file"]).parent / "calibration"
- variants = tuple(
- sorted(
- path.parent.relative_to(calibration_root).as_posix()
- for path in calibration_root.rglob("apply.py")
- )
+ variants = SOFTWARE.discover_variants(
+ calibration_root, case["artifact_contract"]["kind"]
)
if any(
calibration_root.joinpath(variant, "expect.json").is_file()
@@ -239,10 +291,59 @@ def test_checked_in_expectation_corpora_are_complete(self) -> None:
{
"software-behavioral-subtyping",
"software-domain-simplicity",
+ "software-evidence-gap",
"software-knowledge-boundary",
+ "software-compatibility-decision",
+ "software-root-cause-diagnosis",
"software-secure-archive-restore",
+ "software-surgical-plan",
+ },
+ )
+
+ def test_final_output_corpora_have_shape_and_mutation_sensitivity(self) -> None:
+ manifest = json.loads((SUITE_ROOT / "suite.json").read_text(encoding="utf-8"))
+ cases = {
+ case["id"]: case
+ for case in manifest["cases"]
+ if case["skill"] == "engineering"
+ and case["artifact_contract"]["kind"] == "final_output_json"
+ }
+ self.assertEqual(
+ set(cases),
+ {
+ "software-compatibility-decision",
+ "software-evidence-gap",
+ "software-root-cause-diagnosis",
+ "software-surgical-plan",
},
)
+ for case_id, case in cases.items():
+ with self.subTest(case_id=case_id):
+ calibration_root = (
+ SUITE_ROOT / case["prompt_file"]
+ ).parent / "calibration"
+ self.assertEqual(
+ len(
+ SOFTWARE.discover_good_variants(
+ calibration_root, "final_output_json"
+ )
+ ),
+ 7,
+ )
+ isolated_failures = set()
+ for variant in SOFTWARE.discover_variants(
+ calibration_root, "final_output_json"
+ ):
+ expectation = SOFTWARE.load_expectation(
+ calibration_root / variant / "expect.json"
+ )
+ self.assertIsNotNone(expectation, variant)
+ assert expectation is not None
+ if len(expectation["must_fail"]) == 1:
+ isolated_failures.update(expectation["must_fail"])
+ self.assertEqual(isolated_failures, set(case["critical_expectations"]))
+ mutation = calibration_root / "adversarial/workspace-edit"
+ self.assertTrue(mutation.joinpath("apply.py").is_file())
def test_secure_archive_expectations_cover_every_critical_assertion(self) -> None:
manifest = json.loads((SUITE_ROOT / "suite.json").read_text(encoding="utf-8"))
diff --git a/tests/test_codex_app_server.py b/tests/test_codex_app_server.py
index 1b8cf7a..ac5b4ab 100644
--- a/tests/test_codex_app_server.py
+++ b/tests/test_codex_app_server.py
@@ -5010,6 +5010,10 @@ def test_static_config_denies_credentials_and_command_network(self) -> None:
self.assertIn(f'"{self.runtime / "codex-home" / "auth.json"}" = "deny"', text)
self.assertIn(f'"{self.runtime / "codex-home" / "config.toml"}" = "deny"', text)
self.assertIn(f'HOME = "{self.runtime / "work" / ".skill-eval-home"}"', text)
+ self.assertEqual(
+ config["shell_environment_policy"]["set"]["PYTHONDONTWRITEBYTECODE"],
+ "1",
+ )
self.assertIn(f'"{self.runtime / "work"}" = "write"', text)
self.assertIn(f'"{self.runtime / "work" / ".skill-eval-tmp"}" = "write"', text)
self.assertIn(
diff --git a/tests/test_providers.py b/tests/test_providers.py
index a231263..f7d417e 100644
--- a/tests/test_providers.py
+++ b/tests/test_providers.py
@@ -714,6 +714,7 @@ def test_agent_uses_safe_stateless_budgeted_command_and_captures_exact_usage(
self.assertEqual(run.call_args_list[0].kwargs["timeout_seconds"], 12)
inner = command[command.index("--") + 1 :]
self.assertIn("CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1", inner)
+ self.assertIn("PYTHONDONTWRITEBYTECODE=1", inner)
def test_comparator_delegates_canonical_bytes_to_shared_runtime(self) -> None:
provider = ClaudeCliProvider(self.config())
diff --git a/tests/test_runner.py b/tests/test_runner.py
index 9bcc9c9..88c8aa0 100644
--- a/tests/test_runner.py
+++ b/tests/test_runner.py
@@ -2903,6 +2903,28 @@ def test_verifier_runs_in_disposable_copy_and_cannot_contaminate_agent_diff(
self.assertTrue(arm["verifier"]["workspace_mutated"])
self.assertNotIn("verifier-created.txt", arm["diff"])
+ def test_workspace_diff_excludes_generated_agent_caches(self) -> None:
+ provider = self.fixture.provider()
+ original_handler = provider._agent_handler
+
+ def create_cache(request):
+ result = original_handler(request)
+ cache = request.workspace / "__pycache__"
+ cache.mkdir()
+ (cache / "note.pyc").write_bytes(b"generated cache")
+ return result
+
+ provider._agent_handler = create_cache
+ result = self.runner(provider).run(
+ RunSelection(comparison_ids=("without-current",)),
+ output_dir=self.output("agent-cache-diff"),
+ )
+
+ self.assertTrue(result["passed"], result)
+ for arm in result["pairs"][0]["arms"].values():
+ self.assertTrue(arm["verifier"]["sandbox"]["agent_workspace_mutated"])
+ self.assertNotIn("__pycache__", arm["diff"])
+
def test_run_when_artifact_is_final_text_keeps_workspace_read_only(self) -> None:
# Arrange
self.fixture.use_objective()
@@ -3038,6 +3060,7 @@ def test_run_when_artifact_is_final_json_completes_without_comparator(self) -> N
passed = (
content == b'{"a":1,"b":2}'
and os.environ["EVAL_ARTIFACT_KIND"] == "final_output_json"
+ and os.environ["EVAL_AGENT_WORKSPACE_MUTATED"] == "1"
and os.environ["EVAL_ARTIFACT_SHA256"] == hashlib.sha256(content).hexdigest()
and (workspace / "input.txt").read_text(encoding="utf-8") == "original\\n"
and not (workspace / "poison.txt").exists()
@@ -3085,8 +3108,117 @@ def json_output(request):
)
self.assertTrue(arm["verifier"]["artifact"]["read_only"])
self.assertTrue(arm["verifier"]["sandbox"]["workspace_read_only"])
+ self.assertTrue(arm["verifier"]["sandbox"]["agent_workspace_mutated"])
self.assertFalse(arm["verifier"]["workspace_mutated"])
+ def test_final_output_mutation_signal_covers_all_workspace_entries(
+ self,
+ ) -> None:
+ self.fixture.use_objective(artifact_kind="final_output_json")
+ self.fixture.set_verifier(
+ """import json
+import os
+
+passed = os.environ["EVAL_AGENT_WORKSPACE_MUTATED"] == "1"
+print(json.dumps({
+ "passed": passed,
+ "assertions": [{
+ "id": "answer-present",
+ "passed": passed,
+ "evidence": "agent workspace mutation signal",
+ }],
+ "metrics": {},
+}))
+"""
+ )
+ self.fixture.save_manifest()
+
+ def create_empty_directory(workspace: Path) -> None:
+ (workspace / "empty").mkdir()
+
+ def remove_owner_write_permission(workspace: Path) -> None:
+ (workspace / "input.txt").chmod(0o444)
+
+ def create_cache_file(workspace: Path) -> None:
+ cache = workspace / "__pycache__"
+ cache.mkdir()
+ (cache / "note.txt").write_text("agent-created\n", encoding="utf-8")
+
+ def mutate_then_restore_file(workspace: Path) -> None:
+ target = workspace / "input.txt"
+ original = target.read_bytes()
+ target.write_bytes(b"transient mutation\n")
+ target.write_bytes(original)
+
+ mutations = (
+ ("empty-directory", create_empty_directory),
+ ("file-permission", remove_owner_write_permission),
+ ("cache-file", create_cache_file),
+ ("transient-write", mutate_then_restore_file),
+ )
+ for name, mutate in mutations:
+ with self.subTest(name=name):
+
+ def json_output(request):
+ mutate(request.workspace)
+ return '{"a": 1}'
+
+ result = EvalRunner(
+ self.load(), FakeProvider(agent_handler=json_output)
+ ).run(
+ RunSelection(comparison_ids=("without-current",)),
+ output_dir=self.output(f"final-json-{name}"),
+ )
+
+ self.assertTrue(result["passed"], result)
+ for arm in result["pairs"][0]["arms"].values():
+ self.assertTrue(
+ arm["verifier"]["sandbox"]["agent_workspace_mutated"]
+ )
+
+ def test_final_output_without_workspace_mutation_reports_false(self) -> None:
+ self.fixture.use_objective(artifact_kind="final_output_json")
+ self.fixture.set_verifier(
+ """import json
+import os
+
+passed = os.environ["EVAL_AGENT_WORKSPACE_MUTATED"] == "0"
+print(json.dumps({
+ "passed": passed,
+ "assertions": [{
+ "id": "answer-present",
+ "passed": passed,
+ "evidence": "agent workspace remained untouched",
+ }],
+ "metrics": {},
+}))
+"""
+ )
+ self.fixture.save_manifest()
+
+ def use_provider_runtime_scratch(request):
+ for name in (
+ ".skill-eval-cache",
+ ".skill-eval-home",
+ ".skill-eval-tmp",
+ ):
+ scratch = request.workspace / name
+ scratch.mkdir()
+ (scratch / "runtime").write_text("transient\n", encoding="utf-8")
+ shutil.rmtree(scratch)
+ return '{"a": 1}'
+
+ result = EvalRunner(
+ self.load(), FakeProvider(agent_handler=use_provider_runtime_scratch)
+ ).run(
+ RunSelection(comparison_ids=("without-current",)),
+ output_dir=self.output("final-json-no-mutation"),
+ )
+
+ self.assertTrue(result["passed"], result)
+ for arm in result["pairs"][0]["arms"].values():
+ self.assertFalse(arm["verifier"]["sandbox"]["agent_workspace_mutated"])
+
def test_preflight_when_final_output_is_judged_requires_calibrated_profile(
self,
) -> None:
@@ -3104,6 +3236,15 @@ def test_preflight_when_final_output_is_judged_requires_calibrated_profile(
self.assertEqual(provider.agent_requests, [])
self.assertEqual(provider.comparator_requests, [])
+ verifier_only = runner.preflight(
+ RunSelection(
+ comparison_ids=("without-current",),
+ verifier_only=True,
+ )
+ )
+ self.assertEqual(verifier_only["execution_mode"], "verifier_only")
+ self.assertIsNone(verifier_only["comparator"])
+
# Arrange
for case in self.fixture.manifest["cases"]:
case["artifact_contract"] = {"kind": "workspace_diff"}
@@ -3381,6 +3522,7 @@ def hidden(path):
"EVAL_ARTIFACT_PATH",
"EVAL_ARTIFACT_KIND",
"EVAL_ARTIFACT_SHA256",
+ "EVAL_AGENT_WORKSPACE_MUTATED",
"EVAL_CASE_ROOT",
"EVAL_TOOL_BIN",
"EVAL_RESULT_ROOT",
@@ -8051,9 +8193,9 @@ def test_gcc_attestation_includes_derived_driver_closure(self) -> None:
def test_manifest_is_loadable_and_public_cases_are_not_holdouts(self) -> None:
suite = load_suite(HARNESS_ROOT / "suite.json")
splits = [case.split for case in suite.cases]
- self.assertEqual(len(suite.cases), 17)
+ self.assertEqual(len(suite.cases), 21)
self.assertEqual(splits.count("train"), 10)
- self.assertEqual(splits.count("validation"), 7)
+ self.assertEqual(splits.count("validation"), 11)
self.assertNotIn("holdout", splits)
def test_models_and_frozen_original_are_pinned(self) -> None:
17calibrated cases
+ 21calibrated cases
3artifact contracts
3 × 3comparison arms and repetitions
MITopen-source license
@@ -213,15 +213,15 @@ Evaluate the output you actually care about.
Reference corpus
-Seventeen cases built to break weak evidence.
-Ten engineering and seven testing cases span train and validation splits. The corpus is a reference implementation of the case contract, not the evaluator’s domain limit.
+Twenty-one cases built to break weak evidence.
+Fourteen engineering and seven testing cases span train and validation splits. The corpus is a reference implementation of the case contract, not the evaluator’s domain limit.
| Track | Train | Validation | Focus |
|---|---|---|---|
| Engineering | 5 | 5 | Correctness, compatibility, security, concurrency, performance, simplicity |
| Engineering | 5 | 9 | Correctness, compatibility, diagnosis, planning, research, security, concurrency, performance, simplicity |
| Testing | 5 | 2 | Oracle sensitivity, boundary fidelity, state models, flake control, idempotency |