Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions evals/gate/02-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ home**. Results never transfer between providers, between homes, or between
candidates. A provider whose sequence is interrupted restarts at `L1` with a
new clean home.

`L1–L7` is the entry condition for that provider's Part C run. Keep the
installed payload, `GATE_HOME`, and `GATE_CWD` in place while `G1–G6` run.
`L8` is deferred until Part C completes or stops, so teardown cannot destroy
the state that the live behavior cases must exercise.

Part A must be complete, and `H1` must be authorized before any step that
starts a provider session (`L7`, and any optional in-session check).

Expand Down Expand Up @@ -282,25 +287,28 @@ Requires `H1`.
version and model id only. It does not prove the model will select the Skill
for an arbitrary phrasing; routing behavior is out of scope for this Gate.

## L8 — Teardown
## L8 — Deferred teardown after Part C

- **Setup.** `L7` recorded, or the sequence stopped.
- **Setup.** If any case stops before `L7 PASS`, run teardown immediately.
Only an `L7 PASS` defers teardown until the corresponding provider's
`G1–G6` completes or stops.
- **Action.** `run_skills_remove`; remove `GATE_HOME`, `GATE_CWD`, and
any temporary manifest directory created by the session.
- **Oracle.** No `run-with-first-tree` payload remains under either directory;
the fixture directory digests are unchanged from `A3`.
- **Evidence.** Final listings, the removal record, and the after-run fixture
digests.
- **Stop / fail.** A fixture digest changed during Part B — the Skill or the
session modified the source tickets. That is a `FAIL` for the run, not a
cleanup note.
- **Stop / fail.** A fixture digest changed during Part B or Part C — the Skill
or a session modified the source tickets. That is a `FAIL` for the run, not
a cleanup note.
- **Reset.** Not applicable; this is the reset.
- **Claim limits.** Teardown covers local files only. Any First Tree chat
created during Part B is recorded, not deleted.
created during Part C is recorded, not deleted.

## Provider matrix

Run every case above for both providers. The differences that matter:
Run `L1`–`L7`, then the corresponding provider's Part C `G1`–`G6`, then `L8`
for both providers. The differences that matter:

| | Codex | Claude Code |
| --- | --- | --- |
Expand Down
18 changes: 14 additions & 4 deletions evals/gate/03-behavior-cases.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Part C — Live behavior cases

Part C is the only part of this Gate that qualifies live agent behavior. It
requires `H1`, a completed Part A, and a passing Part B for the provider being
exercised.
requires `H1`, a completed Part A, and passing Part B cases `L1–L7` for the
provider being exercised. Part B's `L8` remains deferred until this provider's
`G1–G6` complete or stop.

Run `G1`–`G6` **once per provider**. A case passed on Codex grants nothing to
Claude Code and the reverse. Within a provider, run the cases in order; each
Expand All @@ -22,6 +23,11 @@ GATE_TMP=$(mktemp -d) # operator-owned, separate from the session's own
`GATE_HOME` and `GATE_CWD` keep the values Part B recorded for the provider
under test.

Start each provider session from `GATE_CWD` with `HOME="$GATE_HOME"`. Run all
of `G1`–`G6` using the installed `PAYLOAD` that Part B recorded. Cases may
start new sessions as their setup and reset instructions require; every such
session reuses these Part B values.

Copy [fixtures/](fixtures) from the clean candidate checkout to
`GATE_FIXTURE_ROOT` once, record the per-file and directory digests from `A3`,
and never edit them during the run.
Expand Down Expand Up @@ -485,8 +491,12 @@ session did with them.

## Provider notes

- Run the whole of `G1`–`G6` in a session of the provider under test, using the
payload installed by that provider's Part B run.
- Run the whole of `G1`–`G6` in sessions of the provider under test. Start each
from `GATE_CWD` with `HOME="$GATE_HOME"` and use the payload installed by that
provider's Part B run.
- Run `L8` immediately after that provider's `G1`–`G6` completes or stops.
Teardown is still required when Part C fails, stops, or is inconclusive; it
does not change the Part C verdict.
- The assignee agents in `G5` and `G6` may run on any provider; the case
qualifies the **parent** session's provider. Record each assignee's runtime
anyway, because an assignee's failure to answer makes the case
Expand Down
4 changes: 2 additions & 2 deletions evals/gate/evidence-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Fixture root (absolute, stable for the run):
| `L5` rollback / recovery | | | | |
| `L6` uninstall | | | | |
| `L7` reinstall + fresh session | | | | |
| `L8` teardown | | | | |
| `L8` deferred teardown after Part C | | | | |

### Claude Code

Expand All @@ -132,7 +132,7 @@ Fixture root (absolute, stable for the run):
| `L5` rollback / recovery | | | | |
| `L6` uninstall | | | | |
| `L7` reinstall + fresh session | | | | |
| `L8` teardown | | | | |
| `L8` deferred teardown after Part C | | | | |

Exact removal command form that worked:

Expand Down
40 changes: 40 additions & 0 deletions tests/test_release_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@

ROOT = Path(__file__).resolve().parents[1]
LIFECYCLE_GATE = ROOT / "evals" / "gate" / "02-lifecycle.md"
BEHAVIOR_GATE = ROOT / "evals" / "gate" / "03-behavior-cases.md"
RELEASE_GATE = ROOT / "evals" / "gate" / "04-release-smoke.md"
EVIDENCE_TEMPLATE = ROOT / "evals" / "gate" / "evidence-template.md"
SUPPORTED_AGENTS = ["codex", "claude-code"]


Expand Down Expand Up @@ -45,7 +47,9 @@ class ReleaseGateContractTests(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
cls.lifecycle = LIFECYCLE_GATE.read_text(encoding="utf-8")
cls.behavior = BEHAVIOR_GATE.read_text(encoding="utf-8")
cls.release = RELEASE_GATE.read_text(encoding="utf-8")
cls.evidence = EVIDENCE_TEMPLATE.read_text(encoding="utf-8")
cls.sections = {
"Part B": cls.lifecycle,
"D7": section(cls.release, "## D7 —", "## D8 —"),
Expand Down Expand Up @@ -103,6 +107,42 @@ def test_local_checkout_update_observation_is_not_overclaimed(self) -> None:
self.assertIn("no installed skills found matching", update)
self.assertIn("does not qualify a release update", update)

def test_part_c_uses_l1_through_l7_payload_before_deferred_l8(self) -> None:
lifecycle_intro = normalized_prose(
section(self.lifecycle, "# Part B —", "## Shared setup")
)
lifecycle_teardown = normalized_prose(
section(self.lifecycle, "## L8 —", "## Provider matrix")
)
behavior_intro = normalized_prose(
section(self.behavior, "# Part C —", "## Shared harness")
)
behavior_harness = section(
self.behavior, "## Shared harness", "## The confirmation contract"
)
provider_notes = normalized_prose(
self.behavior.split("## Provider notes", 1)[1]
)
d0 = section(self.release, "## D0 —", "## D1 —")

self.assertIn("l1–l7 is the entry condition", lifecycle_intro)
self.assertIn("keep the installed payload", lifecycle_intro)
self.assertIn("before l7 pass", lifecycle_teardown)
self.assertIn("only an l7 pass defers teardown", lifecycle_teardown)
self.assertIn("g1–g6 completes or stops", lifecycle_teardown)
self.assertIn("l1–l7", behavior_intro)
self.assertIn("l8 remains deferred", behavior_intro)
self.assertIn(
'Start each provider session from `GATE_CWD` with `HOME="$GATE_HOME"`.',
behavior_harness,
)
self.assertIn("run l8", provider_notes)
self.assertEqual(self.evidence.count("`L8` deferred teardown after Part C"), 2)
self.assertIn("| Part B `L1`–`L8` on Codex | `PASS` |", d0)
self.assertIn("| Part B `L1`–`L8` on Claude Code | `PASS` |", d0)
self.assertIn("| Part C `G1`–`G6` on Codex | `PASS` |", d0)
self.assertIn("| Part C `G1`–`G6` on Claude Code | `PASS` |", d0)


if __name__ == "__main__":
unittest.main()
Loading