spec: cut OA 1.6.0 — runtime redefined around contracts, pipeline, cost observability, sandboxing#90
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…st observability, sandboxing Promote the 1.5 draft to a release-status 1.6.0 formal spec, restructured so the runtime is defined as a deterministic pipeline with explicit design goals (predictability, cost transparency, right-sized inference, bounded execution). Newly formalised — previously implemented but spec'd nowhere: - sandbox:/IIS declaration, per-key override resolution, SANDBOX_* codes - reserved history input convention and message-list injection semantics - chain-wide input immutability at every task boundary - diamond-DAG legality and the linear-time cycle-detection requirement - per-engine default-endpoint isolation - certified conformance via the runtime-agnostic adapter protocol Usage/cost reporting is promoted from an envelope footnote (old §8.3) to its own normative section (§10) covering normalisation, multi-call summation, no-guessed-rates, and fail-closed pricing overrides. Cut oas-schema-1.6.json (1.5 + root/task sandbox definitions; additive — every valid 1.5.x document remains valid; all bundled examples verified). Reposition README to lead with cost efficiency & effectiveness, point the spec table at the 1.6 artifacts, and mark the 1.5 doc superseded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166kWcJuF5ip5pbVmmGL4QU
9b685ce to
1b5346f
Compare
aswhitehouse
left a comment
There was a problem hiding this comment.
Good cut overall — this is the right moment to promote the formal spec out of the lag it's been in since 1.5, and the four-pillar framing (contracts / deterministic pipeline / cost observability / sandboxing) is a clean public story. Schema is Draft-07-valid, additive over 1.5, examples/sandboxed-agent validates against it, and CI is green.
One blocking normative contradiction, plus a few things that should land before calling this Status: Release.
Blocking: §7.3 contradicts itself on transitive execution
Step 2a says:
Execute the dependency task recursively (applying the same semantics).
Two paragraphs later:
Only the direct dependencies of the invoked task are executed;
depends_onis not transitive execution.
Those cannot both be true. The second paragraph matches the reference runtimes (_resolve_chain → _run_single_task, which does not re-enter _resolve_chain) and is what the depends-on/diamond-graph and depends-on/deep-lattice conformance cases encode. An independent implementor following step 2a literally would run transitive deps and fail both cases.
Please rewrite step 2 so it's unambiguous, e.g.:
For each direct dependency in listed order: execute that dependency as a single-task invocation (its own
depends_on, if any, is not resolved as part of this call), then merge itsoutputinto the running input map.
Cycle detection remains transitive (walk the full graph); execution stays direct-only. That distinction is the whole point of the lattice work and should be crystal-clear in a Release-status doc.
Should fix before merge
1. Envelope §9.2 vs usage §10.1 — MUST vs SHOULD tension.
§9.2 lists usage inside the envelope a runtime "MUST return". §10.1 says a runtime "SHOULD include a usage key" and that it MUST be null/omitted when the engine reports no counts. Make §9.2 mark usage as optional (present when known, null/omitted otherwise) so the two sections agree — otherwise a local/Codex runtime that correctly omits usage is non-conformant by §9.2 and conformant by §10.
2. Bundled runtime schema has drifted from the canonical one.
oas_cli/schemas/oas-schema.json (what oa validate actually loads) still has no sandbox. So after this PR, the "canonical" schema accepts sandboxed specs and the CLI schema doesn't know they exist. Either:
- sync the bundled schema to 1.6 (preferred — one source of truth), or
- call out explicitly in the PR/docs that the formal schema and the CLI schema are deliberately different until a follow-up.
Leaving them silently divergent undercuts "an independent implementor can build from the formal spec alone" when our own reference CLI doesn't use that schema.
3. Changelog placement.
The 1.6.0 package already shipped (changelog has ## [1.6.0] - 2026-07-05). This formal-spec cut is currently under a buried ## [Unreleased] below that heading. Either fold it into the 1.6.0 block as a documentation amendment of the already-released version, or keep it under a top-of-file [Unreleased] for 1.6.1 — but don't leave Unreleased below a released section. Also, Status: Release on the doc while the changelog calls it Unreleased is a mild inconsistency.
Nits (non-blocking)
- §8.3 history honesty. "MUST NOT silently drop… SHOULD reject or warn" is softer than the sandbox honesty rule in §11.3. Prefer the same shape: a runtime that does not claim
historyMUST refuse (or ignore-with-structured-warning) rather than silently drop. Silent drop is how chat agents "forget". agent.roleenum still rejectsassistant/writer— the 7 example failures are correctly acknowledged as pre-existing, but if this is the Release cut, loosening that enum (or dropping it) would be a cheap win so the multi-agent examples validate against the formal schema.- Conformance cases still embed
open_agent_spec: "1.5.0". Fine under the additive guarantee, but a one-line note inspec/conformance/README.mdthat cases pin the minimum version they require (not the suite version) would prevent future "why doesn't the 1.6 suite use 1.6?" confusion.
What's solid
- Design goals (§1.1) tying cost transparency and cheap validation to the runtime model — this is the right public framing for post-1.6 OA.
- Sandbox §11 is faithful to the runtime (per-key override, resolved paths, three error codes, honesty rule).
- Linear-time cycle-detection requirement + diamond legality (§7.3) correctly captures #88.
- Usage/cost promoted to its own section with fail-closed pricing — much better than the old envelope footnote.
- Adapter protocol / capability registry called out in §14.2.
- Schema additive; sandboxed-agent validates; CI green.
Happy to re-review once the §7.3 contradiction is resolved — that's the one that would actually mislead an implementor.
…; sync bundled schema to 1.6 Address aswhitehouse's review on #90: - §7.3 (blocking): dependency execution is now explicitly direct-only — each direct dependency runs as a single-task invocation whose own depends_on is never resolved — while cycle detection remains transitive. Also documents that each dependency receives a deep copy of the running merged input, matching both reference runtimes. - §9.2: usage is CONDITIONAL in the envelope, reconciling with §10.1 so a runtime whose engine reports no counts is unambiguously conformant. - Bundled runtime schema (oas_cli/schemas/oas-schema.json) synced to the canonical 1.6 schema — one source of truth; oa validate now knows sandbox:. - CHANGELOG: [Unreleased] moved above [1.6.0]; stale pre-1.4 notes carried along, labelled as older. - §8.3: history honesty rule strengthened to match the sandbox rule (MUST refuse or surface a structured warning; silent drop is a conformance violation). - agent.role loosened to a free-form string in the canonical + bundled schemas and §4.1 (informational field, open set, well-known values RECOMMENDED); role test updated for the new semantics. - Multi-agent persona examples gain the required agent.description — every bundled example now validates against the canonical 1.6 schema. - Conformance README: cases pin the minimum spec version they require, not the suite version. 503 tests pass; all 16 example specs validate against oas-schema-1.6.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0166kWcJuF5ip5pbVmmGL4QU
|
All review items addressed in 6456af4 — ready for re-review. Blocking — §7.3: rewritten per your suggested shape. Step 2 now reads "execute that dependency as a single-task invocation … its own Should-fix:
Nits (took all three): §8.3 history honesty now mirrors §11.3 (MUST refuse or surface a structured warning; silent drop is a conformance violation). 503 tests pass locally; ruff clean. Generated by Claude Code |
aswhitehouse
left a comment
There was a problem hiding this comment.
Re-reviewed 6456af4 — all requested changes addressed cleanly.
| Item | Status |
|---|---|
| §7.3 direct-only vs transitive contradiction | ✅ Fixed — execution is explicit direct-only; cycle detection stays transitive; deep-copy of running input documented |
| §9.2 / §10.1 usage MUST vs SHOULD | ✅ Fixed — usage is CONDITIONAL in the envelope |
| Bundled schema drift | ✅ Fixed — oas_cli/schemas/oas-schema.json ≡ oas-schema-1.6.json, includes sandbox: |
Changelog [Unreleased] placement |
✅ Fixed — moved above [1.6.0] |
| History honesty rule | ✅ Strengthened to match sandbox |
agent.role enum |
✅ Open string; persona examples get description; role test updated |
| Conformance cases pin note | ✅ Added |
CI green (test + conformance). Ship it.
One optional follow-up, not blocking: §8.3's "surface a structured warning" doesn't define a warning shape or code. Fine for now — if we ever grow a warnings channel on the envelope, wire it there.
What changed
Cuts the OA 1.6.0 formal specification (
spec/open-agent-spec-1.6.md, status: Release) and canonical schema (spec/schema/oas-schema-1.6.json), superseding the 1.5 draft. The runtime is redefined around four pillars — typed contracts, a deterministic execution pipeline, first-class usage/cost observability, and declarative sandboxing — and the README is repositioned to lead with cost efficiency & effectiveness as headline advantages.Why
After the 1.6.0 release, the formal spec lagged the actual runtime badly:
sandbox:/IIS, history threading, and input immutability were implemented (since 1.5.0) and marketed in the README, but defined nowhere in the formal spec or canonical schema — the 1.5.0 changelog claimed they were folded in, but they never were.Newly formalised in 1.6.0:
sandbox:declaration, per-key task-override resolution, the threeSANDBOX_*error codes, and the honesty rule (§11, §13.2)historyinput convention and its[system, history..., user]injection semantics (§6.4, §8.3)The schema adds root-level and task-level
sandboxdefinitions. Everything is additive: every valid 1.5.x document remains a valid 1.6.0 document.How tested
pytest tests/— 498 passed, 10 skipped)examples/sandboxed-agentnow validates formally; the 7 pre-existing failures are byte-identical under the 1.5 schema)open_agent_spec: "1.6.0"(Python validator + npm loader pattern)Type of change
Checklist
ruff check . && ruff format --check .— no Python touched)pyproject.tomluntouched (no runtime change)