Status: frozen for implementation; v1.1 additive efficiency-evidence amendment frozen for implementation
Eval Contract v1 is a Planr-owned, local-first contract for deciding whether a candidate revision is better than a frozen baseline. It defines authored suite inputs, immutable run evidence, recomputable comparisons, CLI/MCP response shapes, and the ownership boundaries that prevent eval verdicts from becoming a second source of truth for map work closure. The v1.1 amendment is additive: it freezes attempt lineage, metering basis, effective treatment identity, versioned estimate provenance, and derived failure-inclusive efficiency metrics without changing the four comparison verdicts.
- Repository files own authored suite manifests, fixtures, expected lifecycle assertions, and checked-in examples.
- The suite loader owns parsing, repository containment, normalization, canonical ordering, fixture hashing, and suite digests.
- SQLite owns immutable suite snapshots, runs, case results, samples, comparisons, invalidations, rescoring provenance, and evidence references.
- The comparison engine owns compatibility, freshness, coverage, correctness, quality, protected-dimension, performance, and cost gates.
- Existing Planr map state owns item status, dependencies, picks, logs, reviews, approvals, and closure. An eval verdict can be attached as evidence, but it cannot close or reopen work by itself.
- CLI, MCP, and HTTP adapters may render the contract; they must call shared application services and must not implement independent scoring or status transitions.
- Eval Contract v1.1 owns derived efficiency semantics over EvalSample evidence. It does not own a second evidence store, provider catalog, pricing service, router, model leaderboard, or closure authority.
V1.1 extends the V1 objects with the smallest shape needed to compare cost and wall time per independently verified success while preserving backward compatibility. V1 readers may ignore the additive fields and still interpret V1 snapshots, runs, samples, comparisons, invalidations, evidence refs, exit codes, and the four existing comparison verdicts. V1.1 readers must retain the V1 verdict vocabulary exactly: improved, regressed, no_material_difference, and insufficient_evidence.
Attempt lineage is countable evidence, not a separate run model:
- One countable attempt is one case, repetition, and seed execution under one effective treatment, including failures, retries, availability fallbacks, quality escalations, quota downgrades, safety stops, and resumed attempts that re-execute the subject.
- Every countable attempt records
task_id,case_id,repetition_index,seed,attempt_index,retry_of,fallback_of,escalation_of,resume_of, terminal status, reason codes, runner/harness version, and the effective treatment identity described below. - Failed and retried attempts remain durable and participate in aggregate cost and wall-time totals. They cannot be silently dropped, overwritten by a later success, or double-counted as both their original attempt and their retry.
- Warmups may be retained for audit but are excluded from efficiency denominators unless the suite explicitly marks them countable.
Metering basis is per dimension:
actual_trusted: value came from an allowlisted process timer, Planr route observation, host usage receipt, or scorer/event meter whose source and confidence are recorded and accepted by policy.estimated: value was computed from durable measured inputs and a versioned estimate reference, such aspricing_reference_id,pricing_reference_version,currency, andeffective_at. Estimated values are visible but cannot masquerade as trusted actuals.mixed: an aggregate combines multiple accepted bases, such as trusted tool calls with estimated credits, or baseline and candidate runs with different bases. Mixed evidence is non-promotable unless the comparison policy explicitly permits the named mixed basis.unavailable: the value is unknown. Unavailable never coerces to0, never satisfies budget or material-improvement evidence, and must remainnullor omitted with anunavailablebasis in JSON projections.
Allowlisted dimensions are wall_time_ms, tool_calls, input_tokens, output_tokens, total_tokens, credits_micros, and cost_micros. Each EvalSample for these dimensions carries metering_basis, basis_source, basis_confidence, and optional versioned estimate provenance. Unknown or untrusted values stay unavailable rather than falling back to authored defaults, requested route declarations, package metadata, or provider catalog guesses.
Effective treatment identity is the comparison key:
- Each attempt records
client,provider,runtime,model,effort,profile_id,profile_digest,route_policy_digest,runner_version,harness_version, task, repetition, and seed. - Requested route values are not effective treatment evidence. A requested model, effort, provider, runtime, or profile may be stored as intent, but comparison identity uses only resolved/effective values with confidence provenance.
- Silent treatment fallback invalidates matched comparison for the affected cell. If the candidate requested one treatment but effectively ran another and every changed treatment field is not accounted for by a field-specific fallback, escalation, effort downgrade, or quota downgrade reason, the comparison records
treatment_fallback_unattributedand returnsinsufficient_evidence. Retry and resume lineage alone do not attribute treatment changes, and fallback or escalation lineage pointers are audit context rather than blanket attribution for unrelated fields.
Derived efficiency metrics are recomputable from EvalSample and case/attempt status rows:
cost_per_attemptandwall_time_per_attemptdivide all countable attempt totals by countable attempts.cost_per_verified_successandwall_time_per_verified_successdivide all countable attempt totals, including failed attempts, retries, fallbacks, escalations, and downgrades, by independently verified successful attempts.- Zero verified successes produce
unavailableorunbounded, never0, for per-success efficiency. The comparison must useinsufficient_evidenceorregressedaccording to policy, but it must not report a free success. - Solve rate, verified-success count, sample count, confidence intervals, variance, and metering-basis mix stay alongside efficiency metrics. They are not collapsed into a single opaque score.
Schema and surface impact is additive:
- SQLite extends run evidence with attempt lineage and additional EvalSample metadata. It must reuse immutable EvalSample rows and route-observation metering rather than adding a second JSONL evidence store, usage ledger, or provider/pricing catalog.
- CLI, MCP, and HTTP eval responses expose the same efficiency, metering-basis, estimate-provenance, effective-treatment, and reason-code fields inside the existing eval envelope.
- Package export/import includes the attempt lineage, sample metadata, estimate references, and derived comparison outputs needed to recompute the same result locally. Import preflights reject unknown-to-zero coercion and missing effective treatment identity.
- Migrations are additive and preserve existing V1 rows. Missing V1.1 fields on older evidence read as unavailable or non-comparable, not as zero cost, zero time, or a successful matched treatment.
V1.1 executable contract vectors:
{
"object": {
"v1_1_contract_vectors": {
"sample_identity_key": [
"run_id",
"case_id",
"scorer_id",
"scorer_version",
"attempt_id",
"measure"
],
"attempts": [
{
"attempt_id": "evatt-baseline-000001",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-baseline",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 0,
"countable": true,
"retry_of": null,
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": [],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
{
"attempt_id": "evatt-candidate-000001",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 0,
"countable": true,
"retry_of": null,
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "failed",
"reasons": ["process_failed"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
{
"attempt_id": "evatt-candidate-000002",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 1,
"countable": true,
"retry_of": "evatt-candidate-000001",
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": [],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
{
"attempt_id": "evatt-candidate-000003",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 2,
"countable": true,
"retry_of": "evatt-candidate-000002",
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "failed",
"reasons": ["metering_unavailable"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
}
],
"samples": [
{
"sample_id": "evsample-candidate-cost",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"scorer_id": "map-lifecycle",
"scorer_version": "1.0.0",
"attempt_id": "evatt-candidate-000001",
"repetition_index": 0,
"warmup": false,
"seed": 12345,
"measure": "cost_micros",
"value": 300,
"unit": "micros",
"source": "metering",
"metering_basis": "actual_trusted",
"basis_source": "host_usage_receipt",
"basis_confidence": "verified",
"estimate_provenance": null,
"captured_at": "2026-07-21T14:21:10Z"
},
{
"sample_id": "evsample-candidate-wall-time",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"scorer_id": "map-lifecycle",
"scorer_version": "1.0.0",
"attempt_id": "evatt-candidate-000001",
"repetition_index": 0,
"warmup": false,
"seed": 12345,
"measure": "wall_time_ms",
"value": 1200,
"unit": "ms",
"source": "process",
"metering_basis": "actual_trusted",
"basis_source": "allowlisted_process_timer",
"basis_confidence": "verified",
"estimate_provenance": null,
"captured_at": "2026-07-21T14:21:11Z"
},
{
"sample_id": "evsample-candidate-cost-estimated",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"scorer_id": "map-lifecycle",
"scorer_version": "1.0.0",
"attempt_id": "evatt-candidate-000002",
"repetition_index": 0,
"warmup": false,
"seed": 12345,
"measure": "cost_micros",
"value": 700,
"unit": "micros",
"source": "metering",
"metering_basis": "estimated",
"basis_source": "pricing_reference",
"basis_confidence": "estimated",
"estimate_provenance": {
"pricing_reference_id": "openai-prices",
"pricing_reference_version": "2026-07-01",
"currency": "USD",
"effective_at": "2026-07-01T00:00:00Z"
},
"captured_at": "2026-07-21T14:22:11Z"
},
{
"sample_id": "evsample-candidate-cost-unavailable",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"scorer_id": "map-lifecycle",
"scorer_version": "1.0.0",
"attempt_id": "evatt-candidate-000003",
"repetition_index": 0,
"warmup": false,
"seed": 12345,
"measure": "cost_micros",
"value": null,
"unit": "micros",
"source": "metering",
"metering_basis": "unavailable",
"basis_source": "host_usage_receipt_absent",
"basis_confidence": "unavailable",
"estimate_provenance": null,
"captured_at": "2026-07-21T14:23:11Z"
}
],
"efficiency_cases": [
{
"name": "zero-success-is-unavailable-not-zero",
"attempt_ids": ["evatt-candidate-000001"],
"measure": "cost_micros",
"expected_countable_attempts": 1,
"expected_verified_successes": 0,
"expected_total": 300,
"expected_per_verified_success": null,
"expected_per_verified_success_state": "unavailable",
"expected_basis": "unavailable",
"expected_reasons": ["zero_verified_successes", "samples_below_minimum"]
},
{
"name": "retry-accounting-includes-failed-attempt",
"attempt_ids": ["evatt-candidate-000001", "evatt-candidate-000002"],
"measure": "cost_micros",
"expected_countable_attempts": 2,
"expected_verified_successes": 1,
"expected_total": 1000,
"expected_per_verified_success": 1000,
"expected_per_verified_success_state": "available",
"expected_basis": "mixed",
"expected_reasons": []
},
{
"name": "unavailable-failed-attempt-cost-blocks-promotable-per-success",
"attempt_ids": ["evatt-candidate-000002", "evatt-candidate-000003"],
"measure": "cost_micros",
"expected_countable_attempts": 2,
"expected_verified_successes": 1,
"expected_total": null,
"expected_per_verified_success": null,
"expected_per_verified_success_state": "unavailable",
"expected_basis": "unavailable",
"expected_reasons": ["samples_unavailable"]
}
],
"invalid_efficiency_cases": [
{
"name": "successful-only-cost-accounting",
"attempt_ids": ["evatt-candidate-000001", "evatt-candidate-000002"],
"measure": "cost_micros",
"expected_countable_attempts": 2,
"expected_verified_successes": 1,
"expected_total": 700,
"expected_per_verified_success": 700,
"expected_per_verified_success_state": "available",
"expected_basis": "estimated",
"expected_error": "failed_attempt_cost_excluded"
}
],
"invalid_samples": [
{
"name": "unknown-to-zero-coercion",
"sample": {
"sample_id": "evsample-invalid-zero",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"scorer_id": "map-lifecycle",
"scorer_version": "1.0.0",
"attempt_id": "evatt-candidate-000001",
"repetition_index": 0,
"warmup": false,
"seed": 12345,
"measure": "cost_micros",
"value": 0,
"unit": "micros",
"source": "metering",
"metering_basis": "unavailable",
"basis_source": "host_usage_receipt_absent",
"basis_confidence": "unavailable",
"estimate_provenance": null,
"captured_at": "2026-07-21T14:21:10Z"
},
"expected_error": "unavailable_value_must_be_null"
}
],
"valid_treatment_transitions": [
{
"name": "field-specific-model-fallback-attributed",
"attempt": {
"attempt_id": "evatt-candidate-model-fallback-attributed",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 3,
"countable": true,
"retry_of": null,
"fallback_of": "evatt-candidate-primary-unavailable",
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": ["model_fallback"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
}
},
{
"name": "field-specific-escalation-and-downgrade-attributed",
"attempt": {
"attempt_id": "evatt-candidate-escalation-attributed",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 4,
"countable": true,
"retry_of": null,
"fallback_of": null,
"escalation_of": "evatt-candidate-quality-inconclusive",
"resume_of": null,
"terminal_status": "verified_success",
"reasons": ["provider_escalation", "model_escalation", "effort_downgrade"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "high",
"profile_id": "terra-high",
"profile_digest": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "azure-openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "medium",
"profile_id": "terra-high",
"profile_digest": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
}
}
],
"invalid_attempts": [
{
"name": "silent-treatment-fallback",
"attempt": {
"attempt_id": "evatt-candidate-fallback-unattributed",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 0,
"countable": true,
"retry_of": null,
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": [],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "high",
"profile_id": "sol-high",
"profile_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"expected_reason": "treatment_fallback_unattributed"
},
{
"name": "silent-route-policy-digest-change",
"attempt": {
"attempt_id": "evatt-candidate-route-policy-unattributed",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 8,
"countable": true,
"retry_of": null,
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": [],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"expected_reason": "treatment_fallback_unattributed"
},
{
"name": "retry-lineage-does-not-attribute-model-fallback",
"attempt": {
"attempt_id": "evatt-candidate-fallback-retry-only",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 1,
"countable": true,
"retry_of": "evatt-candidate-fallback-unattributed",
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": ["process_failed"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"expected_reason": "treatment_fallback_unattributed"
},
{
"name": "resume-lineage-does-not-attribute-effort-downgrade",
"attempt": {
"attempt_id": "evatt-candidate-downgrade-resume-only",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 2,
"countable": true,
"retry_of": null,
"fallback_of": null,
"escalation_of": null,
"resume_of": "evatt-candidate-fallback-unattributed",
"terminal_status": "verified_success",
"reasons": ["host_resumed"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "high",
"profile_id": "sol-high",
"profile_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "medium",
"profile_id": "sol-high",
"profile_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"expected_reason": "treatment_fallback_unattributed"
},
{
"name": "generic-fallback-lineage-does-not-attribute-model-change",
"attempt": {
"attempt_id": "evatt-candidate-generic-fallback-lineage",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 5,
"countable": true,
"retry_of": null,
"fallback_of": "evatt-candidate-primary-unavailable",
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": ["treatment_fallback_attributed"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"expected_reason": "treatment_fallback_unattributed"
},
{
"name": "generic-escalation-lineage-does-not-attribute-provider-change",
"attempt": {
"attempt_id": "evatt-candidate-generic-escalation-lineage",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 6,
"countable": true,
"retry_of": null,
"fallback_of": null,
"escalation_of": "evatt-candidate-quality-inconclusive",
"resume_of": null,
"terminal_status": "verified_success",
"reasons": ["treatment_transition_attributed"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "azure-openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"expected_reason": "treatment_fallback_unattributed"
},
{
"name": "partial-field-attribution-does-not-cover-profile-change",
"attempt": {
"attempt_id": "evatt-candidate-partial-attribution",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-candidate",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 7,
"countable": true,
"retry_of": null,
"fallback_of": "evatt-candidate-primary-unavailable",
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": ["model_fallback"],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-sol",
"effort": "medium",
"profile_id": "sol-medium",
"profile_digest": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"expected_reason": "treatment_fallback_unattributed"
}
]
}
}
}An authored manifest is a repository-relative JSON or TOML file. V1 accepts one normalized semantic shape regardless of source syntax.
Required fields:
schema_version: exactlyeval.suite.v1.suite_id: stable kebab-case id scoped to the repository.suite_version: SemVer string.description: short human description.fixtures: repository-relative fixture list withid,path,digest_algorithm, and optionalrole.scorers: deterministic scorer list withid,version,kind, and scorer-specificconfig; scorer controls are declared on bounded cases that prove polarity.cases: non-empty ordered input before normalization; each case has identity, fixture/scorer references, command contract, assertions, measures, budgets, and sampling policy.comparison_policy: coverage, freshness, compatibility, quality, performance, cost, and verdict thresholds.safety: execution and retention limits.metadata: optional owner/team labels that are excluded from verdict math unless explicitly named incomparison_policy.
Example:
{
"schema_version": "eval.suite.v1",
"suite_id": "planr-lifecycle-smoke",
"suite_version": "1.0.0",
"description": "Planr lifecycle cases for feature-promotion evidence.",
"fixtures": [
{
"id": "basic-project",
"path": "eval/fixtures/basic-project",
"digest_algorithm": "sha256",
"role": "repository"
}
],
"scorers": [
{
"id": "map-lifecycle",
"version": "1.0.0",
"kind": "planr_lifecycle",
"config": {
"required_events": ["project.created", "map.built", "item.closed"]
}
}
],
"cases": [
{
"case_id": "create-plan-map-close",
"title": "Create a project, build a map, close one item",
"tags": ["lifecycle", "correctness"],
"fixture_ids": ["basic-project"],
"scorer_ids": ["map-lifecycle@1.0.0"],
"subject": {
"kind": "planr_cli",
"argv": ["planr", "project", "init", "EvalDemo"]
},
"assertions": [
{
"kind": "exit_code",
"expected": 0
},
{
"kind": "event_sequence",
"expected": ["project.created", "map.built", "item.closed"]
}
],
"measures": ["duration_ms", "tool_calls"],
"budgets": {
"duration_ms_p95_max": 5000,
"tool_calls_max": 30
},
"sampling": {
"repetitions": 5,
"warmups": 1,
"seed": 12345,
"min_successful_samples": 4
},
"timeout_ms": 15000,
"output_limit_bytes": 65536
}
],
"comparison_policy": {
"freshness_max_age_hours": 168,
"required_case_coverage": 1.0,
"compatible_testbed_fields": ["os", "arch", "planr_version", "cpu_class"],
"quality_non_inferiority_margin": 0.0,
"material_improvement": {
"duration_ms_p95_relative": -0.1
},
"material_regression": {
"duration_ms_p95_relative": 0.1
},
"protected_dimensions": {
"correctness_pass_rate_min": 1.0,
"tool_calls_relative_max": 0.05
}
},
"safety": {
"allow_network": false,
"allow_shell": false,
"max_concurrency": 2,
"capture_stdout": "digest_and_bounded_excerpt",
"capture_stderr": "digest_and_bounded_excerpt",
"forbid_prompt_capture": true,
"forbid_source_capture": true,
"forbid_secret_capture": true
},
"metadata": {
"owner": "planr-core"
}
}The normalized snapshot is the immutable object embedded in each run. It is derived by validation followed by canonicalization:
- Reject unknown
schema_version, duplicate ids, dangling fixture/scorer references, empty case lists, invalid SemVer, ambiguous scorer ids, unsupported status/reason values, and mixed source syntaxes that normalize to different values. - Reject scorer controls with incorrect polarity, duplicate controls, missing oracle or negative coverage for an opted-in scorer, dangling scorer identities, or nondeterministic bounds.
- Resolve every path against the canonical repository root. Reject absolute paths,
.., symlink escapes, home paths, user/global config targets, device paths, and non-file fixture targets. - Normalize path separators to
/, ids to exact case-sensitive strings, durations to integer milliseconds, booleans to booleans, and absent optional fields to explicit documented defaults. - Sort maps by key, fixtures by
id, scorers byid@version, and cases bycase_id; preserve explicit per-case assertion order. - Hash fixture bytes with
sha256, then hash the canonical UTF-8 JSON snapshot withsha256. - Exclude comments, file formatting, and
metadatakeys not named by policy from the digest.
Documented defaults applied before digesting:
fixtures[].role:"repository".cases[].tags:[].cases[].measures:[].cases[].budgets:{}.cases[].scorer_control: omitted for ordinary cases. When present,kindisoracle,known_bad, orno_op(nullnormalizes tono_op), andexpectispassfor oracle controls andfailfor negative controls.cases[].sampling.repetitions:1.cases[].sampling.warmups:0.cases[].sampling.seed:0.cases[].sampling.min_successful_samples: equal torepetitions.cases[].timeout_ms:30000.cases[].output_limit_bytes:65536.comparison_policy.freshness_max_age_hours:168.comparison_policy.required_case_coverage:1.0.comparison_policy.compatible_testbed_fields:["os", "arch", "planr_version"].comparison_policy.quality_non_inferiority_margin:0.0.comparison_policy.material_improvement:{}.comparison_policy.material_regression:{}.comparison_policy.protected_dimensions:{}.safety.allow_network:false.safety.allow_shell:false.safety.max_concurrency:1.safety.capture_stdout:"digest_and_bounded_excerpt".safety.capture_stderr:"digest_and_bounded_excerpt".safety.forbid_prompt_capture:true.safety.forbid_source_capture:true.safety.forbid_secret_capture:true.metadata:{}.
Scorer controls are additive case metadata used to keep weak or inverted scorers from producing promotion evidence:
oraclecontrols must pass under the scorer and prove known-good evidence is accepted.known_badcontrols must fail under the scorer and prove bad evidence is rejected.no_opcontrols must fail and prove null/no-output subject behavior cannot pass accidentally.- Any scorer that opts into controls must include at least one oracle control and at least one negative control (
known_badorno_op). - Controls must be bounded and repeatable: at least two repetitions and no retry/fallback attempts (
max_attempts = 1). - Stored comparisons project controls from the frozen suite snapshot. Failed oracle controls, negative controls that pass, no-op controls that pass, or changed control metadata return explicit non-promotable reasons before ordinary quality and performance gates.
Canonical serialization is RFC 8785 JSON Canonicalization Scheme (JCS): UTF-8 JSON, no insignificant whitespace, object properties sorted lexicographically by unsigned UTF-16 code units, ECMAScript-compatible string escaping, ECMAScript JSON.stringify number serialization, and lowercase literals. Inputs must satisfy I-JSON constraints before digesting: no duplicate object names after parsing, strings must be valid Unicode scalar values, numbers must be finite and integer values must remain within the interoperable [-9007199254740991, 9007199254740991] range unless represented as strings. TOML manifests normalize to this JSON value before serialization, so semantically identical JSON and TOML inputs produce the same canonical bytes and digest.
Digest vectors:
{
"object": {
"canonicalization_vectors": [
{
"name": "utf16-key-order-with-supplementary-plane-key",
"input_json": "{\"\":1,\"𐀀\":2,\"a\":3}",
"semantic_json": {
"": 1,
"𐀀": 2,
"a": 3
},
"canonical_json": "{\"a\":3,\"𐀀\":2,\"\":1}",
"sha256": "e7286ce065b2cc74ec502be56a09b4210f1533604cb4929d1ca01559ef1b8b76"
},
{
"name": "number-edge-cases",
"input_json": "{\"minusZero\":-0,\"small\":0.000001,\"int\":9007199254740991}",
"semantic_json": {
"minusZero": -0.0,
"small": 0.000001,
"int": 9007199254740991
},
"canonical_json": "{\"int\":9007199254740991,\"minusZero\":0,\"small\":0.000001}",
"sha256": "7b423780480affcc244cb986f75813eca804812768b2b8f85b6a94b90144f24a"
},
{
"name": "rfc8785-appendix-b-rounded-ieee754-values",
"input_json": "{\"rfc333\":333333333.33333329,\"rfcfrac\":1.2345678901234567}",
"semantic_json": {
"rfc333": 333333333.33333329,
"rfcfrac": 1.2345678901234567
},
"canonical_json": "{\"rfc333\":333333333.3333333,\"rfcfrac\":1.2345678901234567}",
"sha256": "2da2a5a90d21b3687fa8882b65959bd4e24ce0a72997255b464f3529a33493cf"
}
],
"toml_equivalence": {
"fixture_bytes_utf8": "fixture:basic-project:v1\n",
"json_manifest": {
"schema_version": "eval.suite.v1",
"suite_id": "planr-lifecycle-smoke",
"suite_version": "1.0.0",
"fixtures": [
{
"id": "basic-project",
"path": "eval/fixtures/basic-project",
"digest_algorithm": "sha256"
}
],
"scorers": [
{
"id": "map-lifecycle",
"version": "1.0.0",
"kind": "planr_lifecycle",
"config": {}
}
],
"cases": [
{
"case_id": "create-plan-map-close",
"fixture_ids": ["basic-project"],
"scorer_ids": ["map-lifecycle@1.0.0"],
"subject": {
"kind": "planr_cli",
"argv": ["planr"]
},
"assertions": []
}
],
"comparison_policy": {},
"safety": {}
},
"toml_manifest": "schema_version = \"eval.suite.v1\"\nsuite_id = \"planr-lifecycle-smoke\"\nsuite_version = \"1.0.0\"\n\n[[fixtures]]\nid = \"basic-project\"\npath = \"eval/fixtures/basic-project\"\ndigest_algorithm = \"sha256\"\n\n[[scorers]]\nid = \"map-lifecycle\"\nversion = \"1.0.0\"\nkind = \"planr_lifecycle\"\nconfig = {}\n\n[[cases]]\ncase_id = \"create-plan-map-close\"\nfixture_ids = [\"basic-project\"]\nscorer_ids = [\"map-lifecycle@1.0.0\"]\nassertions = []\n[cases.subject]\nkind = \"planr_cli\"\nargv = [\"planr\"]\n\n[comparison_policy]\n\n[safety]\n",
"json_manifest_digest": "sha256:e2f7f013c72b6519499dfb45df0628d9bb823544a88695b0543464bd1745d79a",
"toml_manifest_digest": "sha256:e2f7f013c72b6519499dfb45df0628d9bb823544a88695b0543464bd1745d79a"
}
}
}Example response:
{
"schema_version": "eval.suite.snapshot.v1",
"suite_id": "planr-lifecycle-smoke",
"suite_version": "1.0.0",
"digest": "sha256:7c4b44e9f6a6113fdf1344f3e14c65b07e32e63e8b22c34784d51a6b9a4ec001",
"case_order": ["create-plan-map-close"],
"fixture_digests": {
"basic-project": "sha256:2bb80d537b1da3e38bd30361aa855686bde0ba89dd8c7d463f537c41b30b25a9"
},
"scorer_identities": ["map-lifecycle@1.0.0"],
"defaults": {
"allow_network": false,
"allow_shell": false,
"capture_stdout": "digest_and_bounded_excerpt",
"capture_stderr": "digest_and_bounded_excerpt"
}
}Run status:
started: run record exists and at least one case may still execute.success: every required case reached a terminal usable status and run-level persistence completed.error: infrastructure or runner failure prevented a usable run.inconclusive: run completed but has no comparison-eligible evidence.invalidated: operator or service provenance marks the run unusable for future comparisons.
Case status:
pass: deterministic assertions passed and samples are comparison-eligible.fail: deterministic assertions prove candidate behavior wrong for the case.error: command, fixture, parser, timeout, or scorer infrastructure failed.skipped: policy excluded the case before execution.inconclusive: execution completed but evidence cannot support pass or fail.
Comparison verdict:
improved: correctness/safety/coverage/freshness/compatibility pass, protected dimensions stay inside budget, and at least one target dimension improves materially.regressed: correctness, safety, quality, protected dimension, performance, or cost is worse beyond the declared threshold.no_material_difference: evidence is compatible and adequate but does not prove material improvement or regression.insufficient_evidence: missing, stale, incompatible, under-covered, under-sampled, invalidated, or excessively noisy evidence prevents a stronger verdict.
Reason codes are stable snake-case strings. V1 reason arrays are additive and ordered by gate.
- Input and safety:
manifest_invalid,unsupported_schema_version,duplicate_identity,dangling_reference,unsafe_path,symlink_escape,absolute_path,shell_not_allowed,network_not_allowed,output_limit_exceeded,secret_like_output. - Run and case:
process_timeout,process_failed,missing_output,invalid_output,scorer_error,assertion_failed,case_skipped,case_inconclusive,run_interrupted. - Compatibility and evidence:
suite_digest_mismatch,fixture_digest_mismatch,scorer_version_mismatch,subject_identity_mismatch,testbed_incompatible,evidence_stale,run_invalidated,coverage_below_minimum,samples_below_minimum,samples_unavailable,variance_too_high,missing_baseline,missing_candidate. - Scorer controls:
scorer_control_incorrect_polarity,scorer_control_missing_oracle,scorer_control_missing_negative,scorer_control_nondeterministic,scorer_control_oracle_failed,scorer_control_negative_passed,scorer_control_unexpected_no_op_success. - V1.1 treatment and metering:
effective_treatment_missing,treatment_fallback_unattributed, field-specific fallback reasons such asclient_fallback,provider_fallback,runtime_fallback,model_fallback,effort_fallback,profile_id_fallback,profile_digest_fallback, androute_policy_digest_fallback, field-specific escalation reasons such asclient_escalation,provider_escalation,runtime_escalation,model_escalation,effort_escalation,profile_id_escalation,profile_digest_escalation, androute_policy_digest_escalation, effort transition reasonseffort_downgradeandquota_downgrade,unavailable_value_must_be_null,mixed_basis_not_allowed,estimate_provenance_missing,attempt_identity_conflict. - Verdict gates:
correctness_regressed,safety_regressed,quality_regressed,protected_dimension_regressed,performance_regressed,cost_regressed,no_material_effect,candidate_improved.
Every stored object carries enough identity to prove where it came from without retaining secrets or full transcripts.
Run identity fields:
run_id,suite_digest,subject_kind,subject_revision,runner_version,planr_version,created_at,completed_at.testbed_fingerprint: OS, arch, Planr version, CPU class, memory class, and allowlisted environment names without values unless the value is explicitly non-secret and hashed.source_state: repository root digest inputs, commit when available, dirty-state summary, and fixture digests.lineage: optional parent run id, resume reason, invalidation id, rescore id, or recompute id.
Case/sample identity fields:
run_id,case_id,scorer_id,scorer_version,fixture_digest,repetition_index,warmup,seed.command: argv array, working directory relative to repository root, timeout, output limit, exit status, stdout digest, stderr digest, and optional bounded artifact ids.attempts: attempt id, task/case/repetition/seed identity, attempt index, countable flag, retry/fallback/escalation/resume lineage, terminal status, reason codes, runner/harness versions, requested treatment intent, and effective treatment evidence.samples: measure name, nullable value, unit, source (process,planr_event,scorer,metering), attempt id, capture timestamp, metering basis, basis source/confidence, and optional estimate provenance.
Comparison identity fields:
comparison_id, baseline run id, candidate run id, comparison policy digest, runner version, created timestamp, gate results, verdict, reason codes, matched effective-treatment identity, efficiency summary, and recompute/rescore lineage.
The fields below are normative for V1 storage and wire projections. Required fields must be present in JSON responses. Optional fields may be null only when the value is unknowable; otherwise defaults are applied before persistence.
Required fields:
schema_versionstring: exactlyeval.suite.snapshot.v1.suite_idstring.suite_versionstring.digeststring:sha256:<64 lowercase hex>.normalized_manifestobject: canonical semantic manifest after defaults.fixture_digestsobject: fixture id tosha256:<hex>.scorer_identitiesarray ofid@versionstrings.case_orderarray of case ids.baseline_selectorobject:kind(explicit_run_id | latest_successful | git_ref) plus selector-specific fields.comparison_policy_digeststring.created_atRFC 3339 timestamp.
Optional fields:
metadataobject, default{}.source_pathrepository-relative string, defaultnull.
{
"object": {
"schema_version": "eval.suite.snapshot.v1",
"suite_id": "planr-lifecycle-smoke",
"suite_version": "1.0.0",
"digest": "sha256:7c4b44e9f6a6113fdf1344f3e14c65b07e32e63e8b22c34784d51a6b9a4ec001",
"normalized_manifest": {
"schema_version": "eval.suite.v1",
"suite_id": "planr-lifecycle-smoke",
"suite_version": "1.0.0"
},
"fixture_digests": {
"basic-project": "sha256:2bb80d537b1da3e38bd30361aa855686bde0ba89dd8c7d463f537c41b30b25a9"
},
"scorer_identities": ["map-lifecycle@1.0.0"],
"case_order": ["create-plan-map-close"],
"baseline_selector": {
"kind": "explicit_run_id",
"run_id": "evrun-20260721-000001"
},
"comparison_policy_digest": "sha256:5a5d4f6ad2c4e98d4f16d857f0f6e2c3f4e5d6c7b8a90123456789abcdef0001",
"created_at": "2026-07-21T14:20:00Z",
"metadata": {},
"source_path": "eval/suites/planr-lifecycle-smoke.json"
}
}Required fields:
run_idstring.suite_digeststring.suite_snapshotEvalSuiteSnapshot object or immutable snapshot id.subjectobject:kind,revision,path,argv, and optionallabel.runner_versionstring.planr_versionstring.statusrun status.created_at,started_at,completed_atRFC 3339 timestamps;completed_atisnulluntil terminal.testbed_fingerprintobject.source_stateobject: commit, dirty summary, root path digest, and fixture digests.case_countsobject with explicit zeros for every case status.lineageobject withparent_run_id,resume_of,rescore_of,recompute_of, all nullable.
Optional fields:
invalidated_byinvalidation id, defaultnull.aggregate_summaryobject, default{}.
{
"object": {
"run_id": "evrun-20260721-000002",
"suite_digest": "sha256:7c4b44e9f6a6113fdf1344f3e14c65b07e32e63e8b22c34784d51a6b9a4ec001",
"suite_snapshot": "evsuite-7c4b44e9",
"subject": {
"kind": "planr_cli",
"revision": "candidate-abc123",
"path": ".",
"argv": ["planr"],
"label": "candidate"
},
"runner_version": "eval-runner-v1",
"planr_version": "1.5.2",
"status": "success",
"created_at": "2026-07-21T14:21:00Z",
"started_at": "2026-07-21T14:21:01Z",
"completed_at": "2026-07-21T14:22:00Z",
"testbed_fingerprint": {
"os": "macos",
"arch": "aarch64",
"planr_version": "1.5.2",
"cpu_class": "apple-silicon",
"memory_class": "16gb",
"env": {
"PLANR_PROFILE": "sha256:7d9c"
}
},
"source_state": {
"commit": "abc123",
"dirty": false,
"root_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
"fixture_digests": {
"basic-project": "sha256:2bb80d537b1da3e38bd30361aa855686bde0ba89dd8c7d463f537c41b30b25a9"
}
},
"case_counts": {
"pass": 1,
"fail": 0,
"error": 0,
"skipped": 0,
"inconclusive": 0
},
"lineage": {
"parent_run_id": null,
"resume_of": null,
"rescore_of": null,
"recompute_of": null
},
"invalidated_by": null,
"aggregate_summary": {}
}
}Required fields:
object_kindstring: exactlyeval_case_result.run_idstring.case_idstring.scorer_idstring.scorer_versionstring.fixture_digeststring.statuscase status.repetition_countinteger, required and non-negative.warmup_countinteger, required and non-negative.assertionsarray of assertion result objects, default[]. Safety verdict replay reads a durable scorer observation from either{"safety_pass": true|false}or a{"kind": "safety_pass", "status": "pass"|"fail"}assertion. Quality verdict replay reads a durable scorer observation from either{"quality_pass": true|false}or a{"kind": "quality_pass", "status": "pass"|"fail"}assertion.commandobject withargv,cwd,timeout_ms,output_limit_bytes,exit_code,stdout_digest,stderr_digest, andartifact_ids.sample_idsarray of strings, default[].reasonsarray of reason codes, default[].
{
"object": {
"object_kind": "eval_case_result",
"run_id": "evrun-20260721-000002",
"case_id": "create-plan-map-close",
"scorer_id": "map-lifecycle",
"scorer_version": "1.0.0",
"fixture_digest": "sha256:2bb80d537b1da3e38bd30361aa855686bde0ba89dd8c7d463f537c41b30b25a9",
"status": "pass",
"repetition_count": 5,
"warmup_count": 1,
"assertions": [
{
"kind": "exit_code",
"status": "pass",
"expected": 0,
"actual": 0
}
],
"command": {
"argv": ["planr", "project", "init", "EvalDemo"],
"cwd": ".",
"timeout_ms": 15000,
"output_limit_bytes": 65536,
"exit_code": 0,
"stdout_digest": "sha256:3333333333333333333333333333333333333333333333333333333333333333",
"stderr_digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"artifact_ids": []
},
"sample_ids": ["evsample-000001"],
"reasons": []
}
}Required fields:
object_kindstring: exactlyeval_attempt.attempt_idstring.task_idstring.run_idstring.case_idstring.repetition_indexinteger, required and non-negative.seedinteger.attempt_indexinteger, required and non-negative within the same case/repetition/seed.countableboolean: false only for explicit warmups or retained non-denominator audit attempts.retry_of,fallback_of,escalation_of,resume_ofnullable attempt ids. At most one lineage pointer should be non-null for a single transition unless the runner records a combined transition reason.terminal_statusstring enum:verified_success | failed | error | skipped | inconclusive | stopped.reasonsarray of reason codes, default[].requested_treatmentobject: client/provider/runtime/model/effort/profile intent as requested before execution. This is audit context only.effective_treatmentobject: client, provider, runtime, model, effort, profile_id, profile_digest, route_policy_digest, runner_version, harness_version, and confidence provenance actually observed for comparison identity.
Optional fields:
started_at,completed_atRFC 3339 timestamps, defaultnull.lineage_notestring, defaultnull.
{
"object": {
"object_kind": "eval_attempt",
"attempt_id": "evatt-candidate-000002",
"task_id": "task-create-plan-map-close",
"run_id": "evrun-20260721-000002",
"case_id": "create-plan-map-close",
"repetition_index": 0,
"seed": 12345,
"attempt_index": 1,
"countable": true,
"retry_of": "evatt-candidate-000001",
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"terminal_status": "verified_success",
"reasons": [],
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
},
"started_at": "2026-07-21T14:21:30Z",
"completed_at": "2026-07-21T14:22:00Z",
"lineage_note": null
}
}Required fields:
sample_idstring.run_idstring.case_idstring.scorer_idstring.scorer_versionstring.attempt_idstring: references EvalAttempt and is part of sample identity.repetition_indexinteger, required and non-negative.warmupboolean.seedinteger.measurestring.valuenumber, boolean, ornull; finite when numeric.nullis required whenmetering_basisisunavailable.unitstring.sourcestring enum:process | planr_event | scorer | metering.metering_basisstring enum:actual_trusted | estimated | mixed | unavailable.basis_sourcestring.basis_confidencestring enum:verified | estimated | mixed | unavailable.estimate_provenanceobject ornull: required for estimated values and carries pricing/reference id, version, currency, and effective timestamp when cost or credit estimates are used.captured_atRFC 3339 timestamp.
Identity invariant:
- The database uniqueness key is
(run_id, case_id, scorer_id, scorer_version, attempt_id, measure).attempt_idincludes task, repetition, seed, and attempt_index identity, so retries and multiple measures within one repetition can coexist without overwriting each other. metering_basis = unavailablerequiresvalue = null,basis_confidence = unavailable, and no estimate provenance. Missing or unavailable values must not be imported, exported, aggregated, or compared as0.
{
"object": {
"sample_id": "evsample-000001",
"run_id": "evrun-20260721-000002",
"case_id": "create-plan-map-close",
"scorer_id": "map-lifecycle",
"scorer_version": "1.0.0",
"attempt_id": "evatt-candidate-000002",
"repetition_index": 0,
"warmup": false,
"seed": 12345,
"measure": "wall_time_ms",
"value": 912,
"unit": "ms",
"source": "process",
"metering_basis": "actual_trusted",
"basis_source": "allowlisted_process_timer",
"basis_confidence": "verified",
"estimate_provenance": null,
"captured_at": "2026-07-21T14:21:10Z"
}
}Required fields:
comparison_idstring.baseline_run_idstring.candidate_run_idstring.policy_digeststring.runner_versionstring.created_atRFC 3339 timestamp.verdictcomparison verdict.reasonsarray of reason codes.gatesarray of gate result objects.effect_estimatesobject of measure name to numeric effect.uncertaintyobject of measure name to interval or variance metadata.protected_dimensionsobject of measure name to numeric result.matched_effective_treatmentobject with baseline and candidate effective-treatment identities used for compatible matching.efficiency_summaryobject with countable attempts, verified successes, solve rate, total cost/wall-time, per-attempt and per-verified-success metrics, metering-basis mix, sample count, confidence intervals, and reason codes. Per-success values arenullwithbasis = unavailableorstate = unboundedwhen verified successes are zero.lineageobject with nullablerecompute_ofandrescore_of.
{
"object": {
"comparison_id": "evcmp-20260721-000001",
"baseline_run_id": "evrun-20260721-000001",
"candidate_run_id": "evrun-20260721-000002",
"policy_digest": "sha256:5a5d4f6ad2c4e98d4f16d857f0f6e2c3f4e5d6c7b8a90123456789abcdef0001",
"runner_version": "eval-runner-v1",
"created_at": "2026-07-21T14:23:00Z",
"verdict": "insufficient_evidence",
"reasons": ["samples_unavailable"],
"gates": [
{
"gate": "compatibility",
"status": "pass",
"reasons": []
},
{
"gate": "correctness",
"status": "pass",
"reasons": []
},
{
"gate": "performance",
"status": "blocked",
"reasons": ["samples_unavailable"]
}
],
"effect_estimates": {
"duration_ms_p95_relative": -0.12
},
"uncertainty": {
"duration_ms_p95_ci95": [-0.18, -0.07],
"variance": "acceptable"
},
"protected_dimensions": {
"correctness_pass_rate": 1.0,
"tool_calls_relative": 0.0
},
"matched_effective_treatment": {
"baseline": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
},
"candidate": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"efficiency_summary": {
"countable_attempts": 2,
"verified_successes": 1,
"solve_rate": 0.5,
"total_cost_micros": {
"value": null,
"basis": "unavailable",
"reasons": ["samples_unavailable"]
},
"cost_per_attempt_micros": {
"value": null,
"basis": "unavailable",
"state": "unavailable"
},
"cost_per_verified_success_micros": {
"value": null,
"basis": "unavailable",
"state": "unavailable",
"reasons": ["samples_unavailable"]
},
"wall_time_per_verified_success_ms": {
"value": 2112,
"basis": "actual_trusted"
},
"sample_count": 4,
"basis_mix": ["actual_trusted", "estimated", "unavailable"],
"confidence_intervals": {},
"reasons": ["samples_unavailable"]
},
"lineage": {
"recompute_of": null,
"rescore_of": null
}
}
}Required fields:
invalidation_idstring.target_kindstring enum:run | comparison.target_idstring.reasonstring.reason_codesarray of reason codes.created_atRFC 3339 timestamp.created_bystring.replacement_hintstring ornull.
{
"object": {
"invalidation_id": "evinv-20260721-000001",
"target_kind": "run",
"target_id": "evrun-20260721-000001",
"reason": "baseline captured on incompatible testbed",
"reason_codes": ["testbed_incompatible"],
"created_at": "2026-07-21T14:24:00Z",
"created_by": "maker-1",
"replacement_hint": "recapture baseline on the current testbed"
}
}Required fields:
evidence_refstring.target_kindstring enum:run | comparison.target_idstring.planr_attachment_kindstring enum:log | artifact.planr_attachment_idstring.item_idstring.created_atRFC 3339 timestamp.closure_authorityboolean, alwaysfalsein V1.
{
"object": {
"evidence_ref": "eval-comparison:evcmp-20260721-000001",
"target_kind": "comparison",
"target_id": "evcmp-20260721-000001",
"planr_attachment_kind": "log",
"planr_attachment_id": "log-2ae87225",
"item_id": "i-freeze-eval-contract-v1-and-owne-e51f",
"created_at": "2026-07-21T14:25:00Z",
"closure_authority": false
}
}All eval CLI JSON and MCP tool responses use one envelope:
okboolean.commandstring: canonical command name.objectobject ornull: primary result.warningsarray, default[].reasonsarray, default[].errorobject ornull:{ code, message, reasons, field }on failure.
MCP tools mirror the CLI object exactly inside the normal MCP text content wrapper. HTTP, if added, returns the same envelope as the response body.
Suite check:
{
"ok": true,
"command": "eval.suite.check",
"object": {
"suite_id": "planr-lifecycle-smoke",
"suite_version": "1.0.0",
"digest": "sha256:7c4b44e9f6a6113fdf1344f3e14c65b07e32e63e8b22c34784d51a6b9a4ec001",
"case_count": 1,
"scorer_count": 1,
"fixture_count": 1
},
"warnings": [],
"reasons": [],
"error": null
}Run:
{
"ok": true,
"command": "eval.run",
"object": {
"run_id": "evrun-20260721-000001",
"suite_digest": "sha256:7c4b44e9f6a6113fdf1344f3e14c65b07e32e63e8b22c34784d51a6b9a4ec001",
"status": "success",
"attempt_summary": {
"countable_attempts": 2,
"verified_successes": 1,
"failed_attempts": 1,
"retry_attempts": 1,
"basis_mix": ["actual_trusted", "estimated"]
},
"attempt_lineage": [
{
"attempt_id": "evatt-candidate-000001",
"attempt_index": 0,
"terminal_status": "failed",
"retry_of": null,
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
},
"reasons": ["process_failed"]
},
{
"attempt_id": "evatt-candidate-000002",
"attempt_index": 1,
"terminal_status": "verified_success",
"retry_of": "evatt-candidate-000001",
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"requested_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
},
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
},
"reasons": []
}
],
"sample_metering": [
{
"sample_id": "evsample-candidate-cost",
"attempt_id": "evatt-candidate-000001",
"measure": "cost_micros",
"metering_basis": "actual_trusted",
"basis_source": "host_usage_receipt",
"basis_confidence": "verified",
"estimate_provenance": null
},
{
"sample_id": "evsample-candidate-cost-estimated",
"attempt_id": "evatt-candidate-000002",
"measure": "cost_micros",
"metering_basis": "estimated",
"basis_source": "pricing_reference",
"basis_confidence": "estimated",
"estimate_provenance": {
"pricing_reference_id": "openai-prices",
"pricing_reference_version": "2026-07-01",
"currency": "USD",
"effective_at": "2026-07-01T00:00:00Z"
}
}
],
"efficiency_summary": {
"countable_attempts": 2,
"verified_successes": 1,
"total_cost_micros": {
"value": 1000,
"basis": "mixed",
"estimate_provenance": ["openai-prices@2026-07-01"]
},
"cost_per_verified_success_micros": {
"value": 1000,
"basis": "mixed",
"state": "available",
"estimate_provenance": ["openai-prices@2026-07-01"]
},
"reasons": []
},
"case_counts": {
"pass": 1,
"fail": 0,
"error": 0,
"skipped": 0,
"inconclusive": 0
},
"evidence_ref": "eval-run:evrun-20260721-000001"
},
"warnings": [],
"reasons": [],
"error": null
}Show run or comparison:
{
"ok": true,
"command": "eval.show",
"object": {
"target_kind": "comparison",
"target_id": "evcmp-20260721-000001",
"summary": {
"verdict": "insufficient_evidence",
"first_blocking_gate": "performance"
},
"matched_effective_treatment": {
"baseline": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
},
"candidate": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"attempt_lineage": [
{
"attempt_id": "evatt-candidate-000002",
"retry_of": "evatt-candidate-000001",
"fallback_of": null,
"escalation_of": null,
"resume_of": null,
"reasons": ["process_failed"],
"effective_treatment": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
}
],
"sample_metering": [
{
"attempt_id": "evatt-candidate-000002",
"measure": "cost_micros",
"metering_basis": "unavailable",
"basis_source": "provider_unavailable",
"basis_confidence": "unavailable",
"estimate_provenance": null
}
],
"efficiency_summary": {
"countable_attempts": 2,
"verified_successes": 1,
"cost_per_verified_success_micros": {
"value": null,
"basis": "unavailable",
"state": "unavailable",
"reasons": ["samples_unavailable"]
},
"reasons": ["samples_unavailable"]
},
"evidence_ref": "eval-comparison:evcmp-20260721-000001"
},
"warnings": [],
"reasons": [],
"error": null
}Compare:
{
"ok": true,
"command": "eval.compare",
"object": {
"comparison_id": "evcmp-20260721-000001",
"baseline_run_id": "evrun-20260721-000001",
"candidate_run_id": "evrun-20260721-000002",
"verdict": "insufficient_evidence",
"reasons": ["samples_unavailable"],
"matched_effective_treatment": {
"baseline": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
},
"candidate": {
"client": "codex",
"provider": "openai",
"runtime": "codex-cli",
"model": "gpt-5.6-terra",
"effort": "medium",
"profile_id": "terra-medium",
"profile_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"route_policy_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"runner_version": "eval-runner-v1",
"harness_version": "planr-eval-harness-v1",
"confidence": "verified"
}
},
"efficiency_summary": {
"countable_attempts": 2,
"verified_successes": 1,
"solve_rate": 0.5,
"cost_per_verified_success_micros": {
"value": null,
"basis": "unavailable",
"state": "unavailable",
"reasons": ["samples_unavailable"]
},
"wall_time_per_verified_success_ms": {
"value": 2112,
"basis": "actual_trusted"
},
"basis_mix": ["actual_trusted", "estimated", "unavailable"],
"reasons": ["samples_unavailable"]
},
"gates": [
{
"gate": "compatibility",
"status": "pass",
"reasons": []
},
{
"gate": "correctness",
"status": "pass",
"reasons": []
},
{
"gate": "performance",
"status": "blocked",
"reasons": ["samples_unavailable"]
}
],
"evidence_ref": "eval-comparison:evcmp-20260721-000001"
},
"warnings": [],
"reasons": [],
"error": null
}Gate:
{
"ok": true,
"command": "eval.gate",
"object": {
"comparison_id": "evcmp-20260721-000001",
"verdict": "insufficient_evidence",
"first_blocking_gate": "performance",
"promotion_allowed": false,
"reasons": ["samples_unavailable"]
},
"warnings": [],
"reasons": ["samples_unavailable"],
"error": null
}Invalidate:
{
"ok": true,
"command": "eval.invalidate",
"object": {
"invalidation_id": "evinv-20260721-000001",
"target_kind": "run",
"target_id": "evrun-20260721-000001",
"status": "invalidated",
"reason_codes": ["testbed_incompatible"]
},
"warnings": [],
"reasons": ["testbed_incompatible"],
"error": null
}Rescore:
{
"ok": true,
"command": "eval.rescore",
"object": {
"source_run_id": "evrun-20260721-000002",
"rescored_run_id": "evrun-20260721-000003",
"scorer_identity": "map-lifecycle@1.1.0",
"lineage": {
"rescore_of": "evrun-20260721-000002"
}
},
"warnings": [],
"reasons": [],
"error": null
}Invalid input:
{
"ok": false,
"command": "eval.suite.check",
"object": null,
"warnings": [],
"reasons": ["unsafe_path", "symlink_escape"],
"error": {
"code": "invalid_input",
"message": "manifest contains an unsafe fixture path",
"reasons": ["unsafe_path", "symlink_escape"],
"field": "fixtures[0].path"
}
}0: successful command; gate may beimprovedor another non-error verdict depending on subcommand.1: regression or gate failure for commands intended to enforce promotion.2: insufficient evidence.3: invalid input or unsafe manifest.4: infrastructure error.
The first implementation slice must add SQLite tables or typed repository methods for run evidence only. It must not add item status columns, alternate closure state, or eval-owned review gates.
Required persistence rules:
- Insert one immutable suite snapshot per digest.
- Insert one run transition or one completed case plus its samples in a transaction.
- Insert one immutable attempt row per
(run_id, case_id, repetition_index, seed, attempt_index)and preserveretry_of,fallback_of,escalation_of, andresume_ofreferences as nullable foreign-key-style ids. Attempt rows own terminal status, reason codes, requested treatment intent, and effective treatment evidence. - Use uniqueness constraints for
(run_id, case_id, scorer_id, scorer_version, attempt_id, measure)samples. Multiple measures in one attempt and multiple retry attempts in one repetition must coexist; importing another row with the same tuple is an identity conflict unless every field is byte-identical. - Preserve nullable sample values.
metering_basis = unavailablerequiresvalue IS NULL;actual_trusted,estimated, andmixedrequire a finite number or boolean according to the measure. Do not materialize unavailable cost, token, time, or tool-call values as zero in migration defaults, package imports, CLI output, MCP output, HTTP output, aggregate caches, or comparison math. - Append invalidation, rescore, and recompute provenance; never update historical verdicts in place.
- Store aggregate summaries as derived cache fields only when raw samples remain available.
- Store Planr log/review attachments as
EvalEvidenceRefrows or existing artifact/log references that point to eval ids.
Package and migration projection rules:
- Eval packages include
suite_snapshots,runs,case_results,attempts,samples,comparisons,invalidations, andevidence_refsarrays.attemptsandsamplescarry the same field names and nullable invariants as the stored objects above. - Package import preflight validates every sample against its attempt, rejects
unavailablesamples with numeric or boolean values, rejects estimated samples without estimate provenance, and rejects silent treatment fallback unless field-specific transition reasons account for every requested/effective treatment field mismatch. Generic fallback/escalation pointers, retry lineage, resume lineage,treatment_fallback_attributed, andtreatment_transition_attributeddo not satisfy this preflight. - Additive migrations create attempt and sample metadata columns with nullable defaults. Backfilled V1 rows are marked non-comparable for V1.1 efficiency until effective treatment identity and metering basis are supplied by durable evidence.
- CLI, MCP, HTTP, package export, package import preview, and migration diagnostics must all use the stable reason-code vocabulary above; adapters may not invent surface-local strings for v1.1 metering or treatment failures.
- Execute only explicit argv arrays; do not pass authored commands through a shell.
- Bound time, output bytes, concurrency, repetitions, and retained artifact size before execution.
- Reject path traversal, absolute paths, symlink escapes, and user/global configuration targets before any execution or database mutation.
- Do not capture prompts, responses, source files, secrets, full transcripts, or unrestricted stdout/stderr by default.
- Scrub bounded excerpts with the existing secret detection policy before persistence.
- Treat fabricated result files, stale baselines, and incompatible testbeds as
insufficient_evidence, not as passes.