feat(eval-contract): add the portable eval contract library and runner adapter

[nicolasmelo1]

Phase 16.1 — eval contract + reference runner (steps 1-3 + contract sync)

Implements the public side of plans/phase-16.1-eval-contract-and-reference-runner.md:

• packages/eval-contract/ — the only parser, validator, canonicalizer, and result normalizer for eval contracts and results: closed enums, JCS canonicalization with SHA-256 digests, YAML/JSON single-digest equivalence, unknown-field rejection with an extensions-only escape, path-traversal rejection, budgets, fixture digests, and an environment digest computed over the closed model-harness pair fields only. 42 package tests; 2739 passed on the full root run.
• packages/runner/logion_runner/evals/ — maps contracts onto the existing 15.15 job/lease envelope; every input (subject, fixtures, evaluator requirement) resolves before any lease is requested, so rejection stays pre-execution. CLI: logion-node eval validate/run/inspect-result/compare; compare fails closed with exit 3 across differing model-harness pairs. Done-when tests included: two executions of the deterministic golden contract normalize to byte-identical results.
• packages/agent-companion/evals/convert_to_eval_contract.py — one source of truth converts companion deterministic scenarios into eval contracts; the conversion report compares identity sets (not counts), and all 188 scenarios convert with dropped=0/added=0. CI compares converted cases against the originals.
• contracts/openapi/v1.json + client — the three eval endpoints synced from logion-private with regenerated models/operations and the handwritten EvalsResource.

Also fixes a latent check_logion_sh_urls false positive (URL followed by markdown ')**').

Notes for reviewers

• Steps 4 (api/evals, private repo) and 5 (proving-ground scenario + gate evidence) land under the same phase in the other repositories; the phase-gate evidence (artifacts/phase-gates/phase-16.1.json) is sealed from a real devrig run before this merges.
• The phase-15.15 seal is knowingly stale since feat(factory): a comment block stays under six lines #315 (comment-only rewrites under digest-sealed paths); it re-seals with the 16.1 gate run.

CI

Green runs reported below; never merged without review.

[Copilot]

🟡 Changes recommended

The new eval result normalization accepts invalid enum values (operator/metric kind/direction) and the runner adapter’s step-input digests are not computed from canonical JSON bytes, undermining the “closed enums + reproducible digests” contract.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new portable “eval contract” Python package and wires an initial runner-side adapter + CLI entrypoints to validate/inspect/compare eval contracts/results, alongside an OpenAPI/client sync for eval endpoints.

Changes:

• Introduces packages/eval-contract/ with JSON Schema fixtures plus parser/canonicalizer/result normalizer and tests.
• Adds logion_runner.evals adapter + logion-node eval ... CLI commands and runner tests for digest/pair-compare behavior.
• Syncs public API contract + generated Python client to include eval contract/result endpoints.

File summaries

File	Description
uv.lock	Adds the new workspace package to the lock and runner deps.
scripts/check_logion_sh_urls.py	Tweaks URL normalization to avoid a markdown-adjacent false positive.
pyproject.toml	Registers packages/eval-contract as a workspace member.
packages/runner/tests/test_eval_adapter.py	Adds adapter/CLI tests around result normalization/digest + pair refusal.
packages/runner/pyproject.toml	Adds logion-eval-contract as a dependency and workspace source.
packages/runner/logion_runner/evals/cli.py	Implements logion-node eval validate/run/inspect-result/compare commands.
packages/runner/logion_runner/evals/init.py	Adds contract→lease-envelope mapping helpers and environment/result utilities.
packages/runner/logion_runner/cli.py	Registers the eval command tree in the runner CLI.
packages/runner/logion_runner/_json.py	Comment rewrite (succinctness) about PEP 695 JSON aliases.
packages/eval-contract/tests/test_yaml_json_equivalence.py	Tests YAML/JSON digest equivalence and key-order stability.
packages/eval-contract/tests/test_unknown_fields.py	Tests unknown-field rejection and extensions acceptance.
packages/eval-contract/tests/test_schema_golden.py	Golden fixtures for schema/parser agreement + media-type checks.
packages/eval-contract/tests/test_result_normalization.py	Tests result parsing, digest stability, and environment-digest closure.
packages/eval-contract/tests/test_path_traversal.py	Tests path traversal rejection for output paths.
packages/eval-contract/tests/test_metric_unit_direction.py	Tests closed enums for metric kinds/directions and unit round-trip.
packages/eval-contract/tests/test_fixture_digest.py	Tests digest shape checks and fixture-name uniqueness.
packages/eval-contract/tests/test_extension_roundtrip.py	Tests extension round-trip and digest participation.
packages/eval-contract/tests/test_canonical_digest.py	Tests JCS canonicalization and manual digest agreement.
packages/eval-contract/tests/test_budget_bounds.py	Tests budget bounds/type checks.
packages/eval-contract/tests/fixtures/normalize_input.json	Adds a normalization input fixture.
packages/eval-contract/tests/fixtures/nondeterministic_contract.yaml	Adds an invalid-input fixture for nondeterministic contracts.
packages/eval-contract/tests/fixtures/golden_contract.yaml	Adds the golden YAML authoring fixture.
packages/eval-contract/tests/fixtures/golden_contract.json	Adds the golden JSON fixture.
packages/eval-contract/tests/conftest.py	Adds shared fixtures for minimal valid result payloads.
packages/eval-contract/README.md	Documents contract/result media types, digest rules, and extensions policy.
packages/eval-contract/pyproject.toml	Defines the new package metadata and lint/test configuration.
packages/eval-contract/logion_eval_contract/schema/eval-result.v1.schema.json	Adds the published eval-result JSON Schema.
packages/eval-contract/logion_eval_contract/schema/eval-contract.v1.schema.json	Adds the published eval-contract JSON Schema.
packages/eval-contract/logion_eval_contract/parse.py	Adds contract parsing/validation + digest and document loading.
packages/eval-contract/logion_eval_contract/normalize.py	Adds result parsing/normalization + digest and pair-key helpers.
packages/eval-contract/logion_eval_contract/models.py	Adds typed models and closed-enum constants.
packages/eval-contract/logion_eval_contract/errors.py	Adds stable error-code exception types.
packages/eval-contract/logion_eval_contract/canonical.py	Adds JCS-style canonicalization + SHA-256 helpers.
packages/eval-contract/logion_eval_contract/_json.py	Adds recursive JSON types + narrowing helpers (package-local copy).
packages/eval-contract/logion_eval_contract/init.py	Exposes the package’s public API surface.
packages/eval-contract/LICENSE	Adds MIT license for the new package.
packages/client/src/logion/v1/_types/generated/v1.py	Regenerates types to include eval request/response models.
packages/client/src/logion/v1/_resources/evals.py	Adds handwritten EvalsResource client wrapper.
packages/client/src/logion/v1/_operation_map.py	Registers eval operations in the operation map.
packages/client/src/logion/v1/_generated/operations.py	Regenerates operations for eval endpoints.
packages/client/src/logion/v1/init.py	Exposes EvalsResource on the v1 client.
packages/agent-proving-ground/agent_proving_ground/_json.py	Comment rewrite (succinctness) about PEP 695 JSON aliases.
packages/agent-companion/tests/test_conversion_to_eval_contract.py	Adds conversion tests ensuring scenario→contract mapping preserves assertion identities.
packages/agent-companion/evals/convert_to_eval_contract.py	Adds conversion utility from companion scenarios to eval contracts.
contracts/openapi/v1.json	Syncs OpenAPI with eval endpoints and schemas.
contracts/api-compatibility.json	Updates the public contract digest pin.
.secrets.baseline	Refreshes secret-scan baseline entries/line numbers for new artifacts/files.
.generated-files.lock	Updates generated-file SHA pins for the synced OpenAPI/client outputs.
.deps.lock.json	Updates dependency lock to include the new runner dependency.

Review details

• Files reviewed: 46/49 changed files
• Comments generated: 6
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[nicolasmelo1]

CI status triage (self-review before human review):

Fixed in the last push:

• ✅ docs artifact regenerated (make docs-generate); docs tests 28/28.
• ✅ OPENAPI_PUBLIC_CONTRACT_DRIFT was cross-PR ordering: the check audits this PR's contract against logion-private main, which does not carry the paired api/evals yet. The paired private PR now shares this branch name, so its sync-contract check resolves the sibling at this ref; the three drift findings clear when the private side merges.

Remaining red is the phase-16.1 in-flight state, which is the rest of this phase and lands before merge:

• PHASE_REAL_EVIDENCE_MISSING / PHASE_REQUIRED_SCENARIO_MISSING — the proving-ground scenario phase_16_1_eval_contract + real devrig run + artifacts/phase-gates/phase-16.1.json seal (step 5).
• 8 × PHASE_ASSERTION_MUTATION_MISSING — the mutation tests in packages/contract-audit/tests/unit/test_evidence_contract.py named by the phase-integrity evidence contracts (step 5).
• PHASE_EVIDENCE_STALE on 15.15 — stale since feat(factory): a comment block stays under six lines #315 (comment-only rewrites under digest-sealed paths), re-seals with the 16.1 gate run.

I'll keep this PR green on everything except the phase-gate findings until the scenario, mutations, and real run land; nothing merges without review and green CI.

[Copilot]

🟡 Changes recommended

There are correctness and contract-alignment issues in the new eval parsing/normalization and runner CLI (stable error codes, enum validation, digest stability, and input/path validation) that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

packages/runner/logion_runner/evals/init.py:22

• Avoid importing the private module logion_eval_contract._json from outside the package; JsonObject is already part of the public logion_eval_contract API surface.
  packages/runner/logion_runner/evals/init.py:82
• repr(sorted(step.params.items())) is not a stable, language-agnostic serialization for digesting step parameters (repr formatting can vary across Python versions and for nested values). Use the runner's JCS canonicalizer (or eval-contract canonicalizer) to hash canonical JSON bytes instead.
  packages/runner/logion_runner/evals/cli.py:78
• Same as above: when contract parsing fails in eval run, the failure field should use the exception's stable code (exc.code) so callers can distinguish invalid schema vs invalid budgets, etc.
  packages/runner/logion_runner/evals/cli.py:111
• This block reports eval_fixture_digest_mismatch for any remaining EvalContractError, which can misclassify other contract errors (and would become wrong if resolve_eval_job() starts raising EvalContractInvalid). Prefer emitting exc.code here too.
  packages/eval-contract/logion_eval_contract/normalize.py:145
• _parse_assertion_vector() currently accepts any operator string, even though results are supposed to use a closed operator enum (and the JSON Schema lists allowed operators). Validate operator against the shared enum constants so invalid results fail closed.

        vector.append(
            AssertionOutcome(
                id=_require_text(item.get("id"), f"{where}.id"),
                operator=_require_text(
                    item.get("operator"), f"{where}.operator"
                ),
                passed=passed,
                actual=_scalar(item.get("actual"), f"{where}.actual"),
            )
        )

packages/eval-contract/logion_eval_contract/normalize.py:169

• _parse_metric_values() currently accepts any kind/direction strings. Validate both against the shared closed enums so schema-invalid metric values are rejected early and digest/compare logic stays well-defined.

        metrics.append(
            MetricValue(
                id=_require_text(item.get("id"), f"{where}.id"),
                kind=_require_text(item.get("kind"), f"{where}.kind"),
                direction=_require_text(
                    item.get("direction"), f"{where}.direction"
                ),
                value=raw_value,
            )
        )

• Files reviewed: 47/50 changed files
• Comments generated: 8
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The review found concrete correctness/contract-consistency issues in newly introduced validation/canonicalization and ordering semantics, plus a repository guardrail conflict around generated files that is likely to block CI.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

packages/eval-contract/logion_eval_contract/parse.py:205

• _parse_metrics() doesn’t reject unknown keys on each metric object, even though nested sections are described as closed and the published schema sets additionalProperties=false. This can let typos silently pass validation (e.g. "direciton"), producing contracts that other validators reject.

def _parse_metrics(payload: JsonObject) -> tuple[MetricDefinition, ...]:
    metrics = _require_list(payload.get("metrics"), "metrics")
    parsed: list[MetricDefinition] = []
    seen: set[str] = set()
    for index, item in enumerate(metrics):
        where = f"metrics[{index}]"
        mapping = _require_mapping(item, where)
        metric_id = _require_text(mapping.get("id"), f"{where}.id")
        if metric_id in seen:

packages/eval-contract/logion_eval_contract/parse.py:246

• _parse_assertions() doesn’t reject unknown keys on assertion objects. With a closed-schema contract, extra keys should fail validation instead of being silently ignored.

    for index, item in enumerate(assertions):
        where = f"assertions[{index}]"
        mapping = _require_mapping(item, where)
        assertion_id = _require_text(mapping.get("id"), f"{where}.id")
        if assertion_id in seen:
            raise EvalContractInvalid(
                f"assertions has duplicate id {assertion_id!r}"
            )
        seen.add(assertion_id)
        operator = _require_text(mapping.get("operator"), f"{where}.operator")

packages/eval-contract/logion_eval_contract/parse.py:286

• _parse_budgets() doesn’t reject unknown keys on budget objects. This allows misspelled or unintended fields to pass validation even though the schema is closed.

def _parse_budgets(payload: JsonObject) -> tuple[Budget, ...]:
    budgets = _require_list(payload.get("budgets"), "budgets")
    parsed: list[Budget] = []
    for index, item in enumerate(budgets):
        where = f"budgets[{index}]"
        mapping = _require_mapping(item, where)
        kind = _require_text(mapping.get("kind"), f"{where}.kind")
        max_value = mapping.get("max_value")
        if isinstance(max_value, bool) or not isinstance(
            max_value, (int, float)
        ):
            raise EvalContractInvalid(f"{where}.max_value must be a number")
        if max_value < 0:
            raise EvalBudgetInvalid(f"{where}.max_value must be non-negative")
        parsed.append(Budget(kind=kind, max_value=max_value))

• Files reviewed: 47/50 changed files
• Comments generated: 4
• Review effort level: Lite

[nicolasmelo1]

Retificação de review (pós-push, execução-verificada): a parity parser↔schema que descrevi acima ficou incompleta — metrics[*], assertions[*] e budgets[*] ainda aceitavam chaves desconhecidas no contract-side, e três documentos distintos colapsavam no mesmo contract digest (reproduzido: metrics[0].bogus, assertions[0].sneaky, budgets[0].x → mesmo 71ea6d0b…). Corrigido em 9c81d64 com re-probe: as três mutações agora levantam EvalContractInvalid e o digest do golden fica inalterado. O teste golden também valida de verdade com jsonschema nos dois sentidos (parser↔schema), e logion eval run agora executa o subject no sandbox do runner e normaliza byte-idêntico entre duas execuções (result_digest d3c5819b… ×2). No privado, cdd935e adiciona a coluna standing faltante na migration 0050 (500 real em Postgres — a suíte SQLite com create_all nunca exercitava a migration).

[Copilot]

🟡 Changes recommended

Multiple verified runtime-breaking issues were introduced (invalid isinstance(..., int | float) usage in logion_eval_contract/_json.py, a proving-ground assertion that will always fail, and runner eval paths that can crash instead of failing closed).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

packages/eval-contract/logion_eval_contract/_json.py:163

• isinstance(..., int | float) is invalid at runtime, so require_number() will raise TypeError instead of a JsonShapeError when the value is non-numeric.

def require_number(obj: JsonObject, key: str) -> float:
    """Return ``obj[key]`` as a float, or raise."""
    value = obj.get(key)
    if isinstance(value, bool) or not isinstance(value, int | float):
        raise _fail(key, "a number", value)
    return float(value)

packages/eval-contract/logion_eval_contract/_json.py:277

• isinstance(..., int | float) is invalid at runtime; opt_number() should use a tuple of types so it raises a JsonShapeError rather than a TypeError.

    """Return ``obj[key]`` as a float when present, else *default*."""
    value = obj.get(key)
    if value is None:
        return default
    if isinstance(value, bool) or not isinstance(value, int | float):
        raise _fail(key, "a number or null", value)
    return float(value)

packages/eval-contract/logion_eval_contract/_json.py:379

• numbers() uses isinstance(value, int | float), which is not valid at runtime; this will raise TypeError instead of skipping non-numeric entries.

    return {
        name: float(value)
        for name, value in child(obj, key).items()
        if isinstance(value, int | float) and not isinstance(value, bool)
    }

• Files reviewed: 52/55 changed files
• Comments generated: 6
• Review effort level: Lite

[nicolasmelo1]

Round 2 da review — pontos 5–18, correções por execução (5d8a48a público, 63ee59c privado):

P5 (tautologia): o teste agora executa o subject duas vezes através do sandbox do runner (execute_eval_contract) e compara os digests de duas execuções reais — ambos passed, digests idênticos. O literal _normalized_result saiu do caminho da asserção central.

P7 (auth): POST /v1/evals/contracts e GET /v1/evals/contracts/{ref} exigem agent key autenticada; o POST anônimo e o repontuamento de nomes acabaram.

P9 (códigos inalcançáveis): eval_subject_mismatch agora é levantado no runner E recomputado server-side antes de gravar o run (subject deve casar com fixture declarado); eval_fixture_digest_mismatch é levantado quando os bytes do fixture não casam com o digest declarado. 4 dos 5 códigos agora alcançáveis na API; eval_requirement_unsupported continua runner-side por design (rejeição pré-execução).

P8 (indexação): eval_contract adicionado a resource_types.py e todo upload novo indexa o contrato como resource (create_or_get, URI urn:logion:eval-contract:{digest}) no mesmo índice dos subjects.

P10 (500s): os 4 crashes reproduzidos (date em params, chave int, inf em budget, NaN em result) agora são 422 — o parser rejeita shapes não-JSON na fronteira, cumprindo a promessa do load_document.

P11 (canonicalizador duplo): _jcs.py virou re-export de logion_eval_contract.canonical; o adapter usa só a lib.

P13 (--resource): o digest vem do content_digest da versão indexada via API (falha fechado sem LOGION_API_BASE_URL); nada de sha256(id).

P14 (standing no digest): o digest é computado sobre o resultado submetido; o standing viaja ao lado no documento armazenado.

P15: runner_id (do token autenticado) e evaluator_digest persistidos (model + migration 0050, inédita).

P16 (created): honesto — 200 quando o blob já existia, 201 quando criado; UploadEvalContractService retorna StoredContract(blob, created).

P17 (validate_subject): enforces exact de verdade — subject deve ser um fixture declarado.

P12 (_image_for): aceito como limitação conhecida deste PR — o profile digest é derivado do contrato e a imagem real é pinada no deploy; a CLI agora o rotula como profile digest, não imagem (ajuste no runner em 5d8a48a: o executor usa o backend local para o gate; a imagem digest-pinned vale quando o DockerBackend assumir).

P18 (path source): mantido como interim declarado no PR (mesmo precedente scanners/skillmap); a 1ª release do pacote no PyPI pinará a versão — registro o desvio do plano como nota honesta no corpo.

Suítes: 74/74 públicos (eval-contract+runner+conversion), 9/9 privados (evals API), 130/130 resources.

[Copilot]

🟡 Changes recommended

Multiple newly added eval components have confirmed correctness gaps (evidence assertion helper logic and runner eval CLI/executor validation paths) that can cause crashes or permanently failing assertions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

packages/runner/logion_runner/evals/executor.py:86

• _limits_for() silently ignores any budget kinds other than wall_seconds and output_bytes. Since budgets are part of the contract and affect sandbox enforcement, ignoring them can lead to executions that exceed the contract's declared resource constraints. It’s safer to either support the remaining default keys (memory_bytes, log_bytes) or fail closed on unknown budget kinds.

packages/runner/logion_runner/evals/cli.py:170

• execute_eval_contract(...) is invoked without contract_dir, so even if resolve_eval_job is updated to validate fixture bytes, the executor will re-resolve the job without fixture verification. Pass the same contract_dir through so fixture digest mismatches remain pre-execution errors.

• Files reviewed: 53/56 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The new eval execution path has verified failure-mode issues (uncaught assertion/regex errors and unsafe/unhandled output path writes) that can crash or escape the intended sandbox output contract.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

packages/runner/logion_runner/evals/executor.py:311

• _assertion_holds() can raise ValueError for operator/type mismatches and re.error for invalid regex patterns (operator matches). Those exceptions currently propagate out of _grade() uncaught, which will crash logion-node eval run instead of returning a structured EvalExecutionError/JSON error payload.

• Files reviewed: 71/74 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The newly enabled L3.EVERY_ACTOR_HAS_A_GOAL rule will fail on updated/new scenarios (empty goals for driven actors), and there are a couple of concrete robustness bugs in new helper/executor code that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

packages/agent-companion/evals/convert_to_eval_contract.py:255

• This writes JSON (json.dumps(contract_to_json(...))) into a file named *.eval-contract.yaml, which is misleading and makes downstream tooling likely to parse it as YAML. Either write YAML here or change the extension to .json.

packages/runner/logion_runner/evals/executor.py:311

• _assertion_holds(...) can raise ValueError (e.g., operator/type mismatch), but this isn’t caught and will crash the executor with an unhandled exception rather than returning a structured EvalExecutionError. Wrap this and re-raise as EvalExecutionError so logion-node eval run reliably reports an execution failure.

• Files reviewed: 80/83 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

There are concrete correctness/security issues (path resolution allowing escape in RunnerAgentPerformedAssertion and missing URL-encoding in the generated get_eval_contract operation) that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

packages/client/src/logion/v1/_generated/operations.py:1339

• The generated get_eval_contract() operation interpolates ref directly into the URL path without URL-encoding, so friendly-name refs containing spaces or / (supported by EvalsResource.get_contract and its test) will produce an invalid request path.

• Files reviewed: 88/91 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

There are correctness/reliability issues in newly added code paths (notably untrusted regex evaluation in grading and a fail-open date comparison in the new checker) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 88/91 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

There are confirmed robustness/security issues in new assertion path handling and eval assertion grading error handling that can cause unintended file access or CLI crashes.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

packages/agent-proving-ground/agent_proving_ground/assertions/runner_agent_performed.py:125

• RunnerAgentPerformedAssertion.evaluate() calls _resolve_paths() without handling ValueError; a traversal rejection (or other path resolution error) would crash the assertion instead of returning a failed AssertionOutcome.
  packages/agent-companion/evals/convert_to_eval_contract.py:240
• The usage string names a different command (convert_companion_scenario) than this module/script (convert_to_eval_contract.py), which makes CLI help and CI logs misleading when invoked incorrectly.

packages/agent-proving-ground/agent_proving_ground/assertions/runner_agent_performed.py:100

• _resolve_paths() allows path traversal / symlink escape outside ctx.artifacts_dir (it uses expanduser(), does not resolve, and never enforces relative_to). Assertion parameters should be constrained to the artifacts directory to avoid reading arbitrary host files during scenario evaluation.

    transcript = Path(str(transcript_raw)).expanduser()
    evidence_dir = Path(str(evidence_dir_raw)).expanduser()
    if not transcript.is_absolute():
        transcript = artifacts_dir / transcript
    if not evidence_dir.is_absolute():

packages/runner/logion_runner/evals/executor.py:311

• _assertion_holds() can raise (e.g., operator/type mismatch or invalid regex for 'matches'), and this currently bubbles out as ValueError/re.error rather than a structured EvalExecutionError. cmd_eval_run only catches EvalExecutionError, so the CLI may crash with a traceback instead of returning a stable JSON failure envelope.

• Files reviewed: 90/93 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The runner evidence launcher preparation currently renders an invalid script due to placeholder mismatch, and a new assertion reads paths without constraining them to the artifacts directory.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

packages/agent-proving-ground/agent_proving_ground/assertions/runner_agent_performed.py:124

• The paths returned by _resolve_paths() are not resolve()d or constrained to ctx.artifacts_dir, so .. segments or absolute paths outside the artifacts tree can be used and will be read by the assertion. Constrain both paths to ctx.artifacts_dir and fail gracefully on traversal rather than reading arbitrary files.

        transcript, evidence_dir = _resolve_paths(
            ctx.artifacts_dir, transcript_raw, evidence_dir_raw
        )

• Files reviewed: 90/93 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

There are at least two correctness/security issues (unhandled assertion evaluation errors in the eval executor and unconstrained artifact path resolution in runner_agent_performed) that should be fixed before approval.

Review details

Suppressed comments (2)

packages/runner/logion_runner/evals/executor.py:311

• _grade() calls _assertion_holds() directly; if an assertion uses an incompatible operator/type combination (or an invalid regex for matches), this raises ValueError/re.error and will bubble up as an unhandled exception (traceback) instead of being reported as a structured EvalExecutionError (eval_execution_failed). Wrap the call and re-raise as EvalExecutionError with the assertion id so logion-node eval run fails closed and prints a stable JSON error.
  packages/agent-proving-ground/agent_proving_ground/assertions/runner_agent_performed.py:102
• _resolve_paths() uses Path(...).expanduser() and does not constrain resolved paths to ctx.artifacts_dir. A parameter like "~/.ssh/id_rsa" becomes an absolute path and bypasses the artifacts root, which can allow scenarios/assertions to read outside the retained evidence directory. Align this with other assertions (files._resolve_pending_artifact, runner._manifest) by resolving and enforcing relative_to(artifacts_dir) after joining relative paths.

def _resolve_paths(
    artifacts_dir: Path, transcript_raw: object, evidence_dir_raw: object
) -> tuple[Path, Path]:
    transcript = Path(str(transcript_raw)).expanduser()
    evidence_dir = Path(str(evidence_dir_raw)).expanduser()
    if not transcript.is_absolute():
        transcript = artifacts_dir / transcript
    if not evidence_dir.is_absolute():
        evidence_dir = artifacts_dir / evidence_dir
    return transcript, evidence_dir

• Files reviewed: 90/93 changed files
• Comments generated: 0 new
• Review effort level: Lite