From 2822aa78e10947ad740e72d8e85466702ff68f36 Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Wed, 3 Jun 2026 11:16:29 +1200 Subject: [PATCH 1/4] docs(specs): add draft Saddle multi-charm unit-testing spec (OPxxx) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the working draft of "Saddle: Multi-Charm Unit Testing in ops.testing" as a self-contained spec for team review prior to submission to specs.canonical.com. The OP number is not yet claimed (placeholder OPxxx); the filename and in-document Index field will be updated once a number is assigned. The spec defines: - A Juju-shaped multi-charm test surface in `ops.testing`: `deploy` / `integrate` / `add-unit` / `remove-unit` / `update-config` / `run_action`, with events triggered automatically and a convergence loop. - Subprocess + per-charm venv as the isolation mechanism (subinterpreters ruled out); persistent worker model behind the same surface. - Self-contained resolutions for six previously-open questions: StoredState fidelity (always-on typed escape hatch); JSON `State` schema stability (explicit `state_schema_version` field + dispatch table); settle-policy determinism requirement (byte-for-byte reproducible) with implementation choice deferred to step 5; inline-vs-on-disk docstring convention for new operations; persistent- worker 2× acceptance bar on a 4-charm/20-event workload; six-shape metadata-resolution test acceptance set for step 6. - Six-step incremental delivery plan; each step independently useful. Drafting against `main` rather than `docs-add-specs` (#1) so the spec can be reviewed independently of that larger PR's status. --- ...multi-charm-unit-testing-in-ops-testing.md | 253 ++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md diff --git a/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md b/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md new file mode 100644 index 0000000..ad9c5ee --- /dev/null +++ b/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md @@ -0,0 +1,253 @@ +| Index | OPxxx (number not yet claimed) | | | +| :---- | :---- | :---- | :---- | +| Title | Saddle: Multi-Charm Unit Testing in ops.testing | | | +| **[Type](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Author(s)** | **[Status](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?tab=t.0)** | **Created** | +| Implementation | [Tony Meyer](mailto:tony.meyer@canonical.com) | Draft | 26 May 2026 | +| | **Reviewer(s)** | **Status** | **Date** | +| | Person | Pending Review | Date | + +# Abstract + +This spec defines **Saddle**: a multi-charm unit (state-transition) test surface folded into `ops.testing`. It provides `deploy`, `integrate`, `add-unit`, `remove-unit`, and `update-config` as test operations, with the corresponding Juju events triggered automatically as the model converges. Each charm runs in its own isolated subprocess (its own venv), so charms with mutually incompatible dependency sets can be tested together in a single test. Saddle targets `ops.testing` in `canonical/operator`; it is not a companion package. It draws the abstraction from [Catan](https://github.com/PietroPasotti/catan) (Apps + convergence) but is not a direct port. + +# Rationale + +## The gap in current tooling + +`ops.testing` (Scenario) runs one charm at a time: the developer writes an event and a `State`, calls `Context.run`, and asserts on the output. This is efficient for single-charm logic but leaves four pain points unaddressed: + +- **Integration tests are too slow for the inner loop.** A real Juju controller bootstrap and deploy round-trip takes minutes; unit tests must be measured in seconds. +- **Scenario is unlike the `juju` commands charmers think in.** Charmers reason about `juju deploy`, `juju integrate`, `juju add-unit`. Translating that mental model into hand-crafted event sequences and `State`s is friction the tool should absorb. +- **Charm libraries don't yet ship mocks.** Cross-charm relation data is hand-crafted per test. Running the other charm's own code against the relation removes that hand-crafting: the other charm's actual reaction becomes the fixture. This may change with the acceptance of OP077 — Charm library testing. +- **Bundle and topology assertions are unreachable from unit tests.** Asserting "if I deploy A and B and integrate them, A's leader ends up with config X" requires real Juju or fragile mocks today. + +## Validated abstraction + +The shape charmers ask Charm Tech for is "a multi-charm state-transition test tool that feels like Juju". This is Catan's abstraction (Apps, `integrate`, convergence/settle), validated by demand rather than speculative. + +## Why ops.testing rather than a companion package + +Catan today pins `ops-scenario~=6.0`, a superseded API that has been merged into `ops.testing` as Scenario 7. Porting off that API is **net-new work regardless of the destination**; there is no path that productises the companion-package form without a full re-targeting. Since we have to re-target anyway, we target `ops.testing` directly. + +# Specification + +## Location and scope + +The new surface lives in `ops.testing` (`canonical/operator`), the same module that provides `Context`, `State`, and `Manager`. The single-charm primitives are **unchanged and remain the substrate**. The new surface adds a model-level layer above `Context` that owns a set of `App`s, an integration graph, and a convergence loop. Single-charm Scenario tests continue to work exactly as today and do not require the new surface. + +## Operation surface + +The test API uses Juju-shaped operations — `deploy`, `integrate`, `add-unit`, `remove-unit`, `update-config`, `run_action` — that describe **intents, not events**. The model generates the correct Juju-faithful event sequence and delivers it to the affected units. + +```python +from ops import testing + +model = testing.Model() + +db = model.deploy( + charm="./charms/postgresql", + app="postgresql", + config={"profile": "testing"}, + num_units=1, +) +web = model.deploy( + charm="./charms/myapp", + app="myapp", + num_units=2, +) + +model.integrate(db["db"], web["pg"]) +model.add_unit(web) +model.update_config(web, {"log_level": "debug"}) +model.remove_unit(web, unit=1) +model.run_action(db.leader, "create-backup", params={}) +``` + +`Model.deploy(...)` returns an `App` handle. An `App` owns its charm reference, the metadata resolved from the charm on disk, the environment it runs in (see Isolation model below), and its per-unit `State`s. Exact type names are illustrative and will be settled in code review. + +## Automatic event triggering and convergence + +Each operation enqueues the appropriate events (`relation-created`, `relation-joined`, `relation-changed`, `config-changed`, `leader-elected`, etc.) on the affected units. A **convergence loop** drains the queue: it picks an event, dispatches it through that unit's `Context.run(...)`, observes any new relation-data writes, status changes, or secret operations, and enqueues the downstream events those produce on peers and related units. Convergence terminates when the queue is empty and no unit's last run produced new observable change. + +```python +model.settle() # explicit convergence point +assert web.leader.state.unit_status == ActiveStatus("ready") +assert db.leader.state.get_relation("db").local_app_data["database"] == "myapp" +``` + +`settle()` is also called implicitly before reading any unit's `State` if there are pending events, so most tests are a sequence of operations followed by assertions on `app.leader.state` or `app.units[i].state`. The `State` exposed on each unit is the same `ops.testing.State` dataclass that single-charm Scenario tests use today; all existing assertions are available unchanged. + +**Determinism requirement (resolved 2026-06-03).** Given the same starting `Model` and the same input event sequence, `settle()` MUST produce the same final `State` per unit and the same sequence of intermediate events, byte-for-byte, across runs and across machines. **No wall-clock dependence, no PID-derived ordering, no Python-dict-iteration-order leakage** is permitted in the policy. The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is picked in step 5 against this requirement, when there is working code to validate it. See Further Information. + +## Isolation model: subprocess + per-charm venv + +Each `App` runs in its own process with its own interpreter (an existing venv's `bin/python`, or one the harness provisioned — see Environment provisioning below). For each event dispatch the harness serialises `(meta, config, actions, event, State)` as JSON, sends it to the charm's worker process, the worker runs `testing.Context(...).run(...)`, and serialises the resulting `State` back. Two `App`s in the same `Model` may use mutually incompatible dependency sets — `cryptography==41` vs `==3`, different `pydantic` majors, conflicting charm-lib versions — demonstrated end-to-end by the isolation spike: two charms with intentionally conflicting `confdep` versions, each in its own venv, both producing correct output in the same test session via both a lightweight `sys.path`-injected dep dir and a full real venv per charm. + +**Required invariant:** each per-charm venv contains an `ops` / `ops-scenario` build whose JSON `State` wire format is compatible with the parent harness's. Only the charm's own runtime deps differ between venvs — realistic, since all charms build on `ops`. + +The isolation worker is persistent: one long-lived process per `App`, spawned lazily on first dispatch and torn down at `Model` teardown or after an idle timeout. A framed JSON protocol over stdin/stdout carries requests and responses. Spawn-per-event remains available as an explicit debug mode (easier to attach a debugger; no shared interpreter state between events). Worker crashes propagate as a typed `IsolationError` and fail the test cleanly; the harness never silently re-spawns mid-test. + +**Acceptance criterion (fixed 2026-06-03):** + +- **Workload:** the canonical perf yardstick is a **4-charm bundle with 20 events per `settle()`**. Lives in the test suite from step 3 onward and is re-run by every later step to catch regressions. +- **Bar:** **persistent-worker mode within 2× the in-process Scenario baseline** on that workload. 2× is the threshold where "slow but acceptable for a CI unit-test suite" tips into "too slow, charmers will avoid it"; subprocess + JSON serialisation has a real floor below that. +- **Benchmark shape:** the step 3 PR ships a benchmark suite measuring (a) in-process Scenario baseline, (b) spawn-per-event isolated, and (c) persistent-worker isolated, all on the same 4-charm/20-event workload. +- **If (c) misses the bar:** the PR either records a clear "this is slower than the target; here's why; here's what could close the gap" trade-off note, or the persistent-worker design is reconsidered before step 3 is signed off. + +### Subinterpreters are explicitly ruled out + +PEP 734 subinterpreters are **not** the mechanism, even granting a 3.14+ floor: + +- **Binary-dependency conflicts remain unsolved.** Many C extensions (`cryptography`, `pydantic-core`, `cffi`-based packages) are not multi-interpreter-safe; two different versions of the same extension cannot coexist in one process at any Python version. These are the exact libraries motivating the work. +- **State boundary still costs a serialisation roundtrip** — the same JSON encode/decode as a subprocess, taken without the isolation benefit. +- **Maintenance cost is high** on a bleeding-edge API for no payoff over the subprocess design. + +Revisit only if the C-extension multi-interpreter story matures enough to load conflicting binary versions in one process — not merely because the Python version floor moves. + +## Environment provisioning + +How a charmer declares an `App`'s environment: + +| Form | API | When to use | +| --- | --- | --- | +| Auto-provision from declared deps | `Model.deploy(..., build_env=True)` | Canonical recommended UX | +| Point at an existing venv | `Model.deploy(..., python="/path/to/venv/bin/python")` | CI where the venv is already managed | +| Inherit the test process | explicit opt-in flag | Only when deps are known not to conflict | + +For auto-provisioning the harness detects the charm's build plugin (from `charmcraft.yaml` first; file-based heuristic second — `uv.lock` → `uv`; `poetry.lock` → `poetry`; `pyproject.toml` → `python`; else → `charm`) and extracts the dependency list: + +| Plugin | Extraction method | +| --- | --- | +| `charm` | reads `charm-requirements` files + `charm-python-packages` + AST-harvested `PYDEPS` from `lib/**/*.py` | +| `python` | reads `python-requirements` / `python-packages` + PYDEPS (included even though charmcraft's python plugin does not auto-install them — the test harness needs the lib importable) | +| `uv` | shells to `uv export --frozen --no-emit-project --no-hashes --format requirements-txt` with forwarded `--extra` / `--group` flags | +| `poetry` | shells to `poetry export --without-hashes --format requirements.txt` with forwarded `--with` flags | + +All forms normalise to a flat pip specifier list, then install via `uv pip install` into the per-charm venv. The venv is cached on disk, keyed by SHA-256 of (plugin, extracted requirements, lockfile/pyproject content, interpreter version, harness `ops` version): cold build ~3s for a test charm; warm cache hit <0.05s (measured). The detection-and-extraction path was validated against two real Canonical charms (loki-k8s-operator via the uv plugin; postgresql-k8s-operator via the poetry plugin); the harness emits a clear, typed error when an extracted requirement cannot be resolved against an available index, when a lockfile is stale relative to the pyproject, or when a required external tool (`uv` / `poetry`) is missing. + +**Tool dependency:** `uv` on PATH for all install steps; `poetry` + `poetry-plugin-export` for the poetry-plugin path. The harness emits a clear error when a required tool is absent. + +**Future:** a charmcraft-managed flow for charms already packed locally. + +## State (de)serialisation + +A typed schema-aware encoder/decoder pair — not `json.dumps(dataclasses.asdict(state))` — covers all `ops.testing.State` leaf types verified against the spike's findings: + +| Type | Encoding | +| --- | --- | +| `frozenset` (on `relations`, `containers`, `secrets`, etc.) | list on encode; re-frozen on decode (dataclass `__post_init__`s re-freeze) | +| `datetime` / `timedelta` | ISO-8601 string / fractional seconds | +| `pathlib.Path` / `PurePosixPath` | string with type tag | +| Pebble enums (`ServiceStatus`, `NoticeType`, `SecretRotate`, …) | enum name | +| `pebble.Layer` (`Container.layers`) | `to_dict()` / `Layer(dict)` round-trip (not a dataclass; `asdict` won't descend) | +| Int-keyed dicts (`Secret.remote_grants: Mapping[int, set[str]]`) | keys stringified on encode; restored to `int`, inner `set` restored on decode | + +**StoredState fidelity (resolved 2026-06-03).** `StoredState.content` and `Container._base_plan` are `Mapping[str, Any]` of marshallable types — a superset of JSON: `bytes`, `tuple`-vs-`list` distinction, and arbitrary `set`s. The spec commits to: + +1. **A typed escape hatch, applied always** — tag-and-encode `bytes` as base64 and `tuple` as a `["__tuple__", ...]` envelope so they round-trip losslessly. **No flag.** An opt-in would create a "passed locally because today's stored-state had no `bytes`, failed in CI when it did" failure mode; the cost (a few extra bytes per typed value) doesn't justify it. +2. **Loud error on unrecognised types** — the encoder raises `TypeError` with the type name and the path through `State`. The decoder mirrors: unknown `__type__` tags raise `TypeError`. Never silent lossy coercion. + +A small follow-up survey of in-tree charm `StoredState` usage is left as a separate work item, to confirm no charm relies on round-tripping more exotic marshallable types than `bytes`/`tuple`/`set`. The survey does not gate the implementation; it informs whether the encoder's type set needs to be extended. + +**JSON `State` schema stability (resolved 2026-06-03).** The JSON `State` representation is **part of `ops.testing`'s public API** — a per-charm venv may have a different `ops` build from the parent, and both sides must agree on the encoded form. + +1. **Explicit `state_schema_version` integer field** is embedded in every encoded `State` payload. Cheaper and more legible than overloading `ops`'s SemVer: we don't have to ship an `ops` major just because we added one field to `State` serialisation. +2. **Per-version interpretation is locked at the source-code level** — a dispatch table keyed by version. Adding a new version is ordinary work; changing the *meaning* of an existing version is not. +3. **Decoder rejects unsupported versions with a clear typed error.** A v1 producer and a v1+v2 consumer work fine. A version the decoder doesn't recognise surfaces as an explicit error, not silent corruption. +4. **`ops.testing` API-stability documentation commits to (2).** "We may add new versions; we don't change the meaning of an existing version." + +Implementation honours this from step 2 onward (the encoder writes `state_schema_version`; the decoder dispatches on it). + +## Metadata resolution + +The harness reads `metadata.yaml` / `charmcraft.yaml` from the charm-source path without importing the charm in the test process. Work required for productisation: + +- Resolve unified-vs-legacy metadata variants and the merged `charmcraft.yaml` + `metadata.yaml` form charms commonly produce +- Resolve charm libraries the charm declares but that live elsewhere on disk +- An isolated equivalent of Catan's `App.from_git` (clone, detect build system, provision venv, run) +- Inline charm classes (`from_type` charms defined in the test): kept as in-process, same-environment only; isolation requires charms on disk (`from_isolated_path`). See Further Information. + +**Step 6 acceptance set for metadata-resolution coverage (locked 2026-06-03).** At least one passing test for each of these charm shapes: + +1. Unified `charmcraft.yaml` only (no `metadata.yaml`) — the current default-modern shape. +2. Legacy split: `metadata.yaml` + `actions.yaml` + `config.yaml` — older charms still in production. +3. Mixed / transitional: `charmcraft.yaml` + `metadata.yaml` (the form charmcraft merges) — charms mid-migration. +4. Multi-bases (`bases:` / `platforms:` variants per supported platform). +5. Charm with libs outside `src/` (e.g. installed via `pip` from PyPI for testing) — exercises the "where did this lib come from" resolution. +6. Inline `from_type` (no metadata file; class declares everything) — documented as `inline: ok` per the docstring convention in Further Information. + +## Migration from Catan / ops-scenario~=6.0 + +The existing `PietroPasotti/catan` package pins `ops-scenario~=6.0` (superseded by `ops.testing`). There is **no compatibility shim**: tests written against Catan must be rewritten against the Saddle surface. The `ops-scenario~=6.0` form is throwaway regardless of whether Saddle is adopted — porting off it is net-new work either way. + +What Saddle carries forward from Catan: + +- The model: Apps + integration graph + convergence/settle — the abstraction demand validates +- The Juju-like operation vocabulary: `deploy` / `integrate` / `add-unit` / … +- The isolation and provisioning spikes as seed implementations for steps 1 and 6 + +What is discarded: + +- The `ops-scenario~=6.0` companion-package API +- The Catan package as a separate distribution artifact (PyPI release, separate repo, separate release cadence) + +## Incremental delivery plan + +Sequenced so each merge is **independently useful** and the area can pause cleanly after any step. This is the handle on the priority reservation: agentic coding lowers per-step build cost, and the sequencing means a pause never strands a half-built feature. + +| Step | Deliverable | Value without later steps | +| --- | --- | --- | +| 1 | Lift isolation into `ops.testing`: `IsolatedEnv`, persistent worker, `from_isolated_path` | Immediately useful: run one charm in an isolated venv from a Scenario test; no `Model`, no convergence | +| 2 | Typed `State` (de)serialiser with full leaf-type coverage and the StoredState escape hatch | Snapshot-style tests and debugging; required by every later step | +| 3 | Persistent-worker mode replacing spawn-per-event (benchmarked in the same PR). **Acceptance:** within 2× of in-process Scenario on the 4-charm/20-event workload — see *Isolation model* | Performance baseline; unblocks realistic convergence-run testing | +| 4 | `Model` + `App` + non-relating operations: `deploy`, `add-unit`, `remove-unit`, `update-config`, `run_action`, single-app convergence | Drive a single charm through Juju-shaped operations without writing event sequences by hand | +| 5 | `integrate` + cross-app event propagation + settle policy locked. **Acceptance:** see *Settle policy considerations* (Further Information) — at least one deterministic policy passes a 100×-soak; user-override hook exposed | Unlocks the multi-charm pain points; steps 1–4 stand alone if priorities pull us off here | +| 6 | `build_env=True` environment provisioning (all four plugins, caching, lockfile handling) + full metadata-resolution coverage. **Acceptance:** the six-shape metadata test set — see *Metadata resolution* | Productisation tail; not on the critical path for any earlier step | + +# Further Information + +## Settle policy considerations (resolved 2026-06-03) + +The determinism requirement is locked in *Automatic event triggering and convergence*: byte-for-byte reproducible across runs and machines; no wall-clock, PID, or dict-iteration-order leakage. + +The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is picked in step 5 against that requirement, with the following acceptance: + +- At least one deterministic policy ships and passes a **"run 100×, all final `State`s identical"** soak test. +- The API exposes a **user-override hook** for cases the default policy doesn't fit. The hook may ship empty initially. + +Picked here rather than locked up front because the choice benefits from being able to play with the convergence loop in working code. + +## Inline charm classes vs on-disk charms (resolved 2026-06-03) + +`from_type` charms defined inline in a test cannot be trivially transferred to a fresh subprocess (the class object is not serialisable). Isolation realistically requires charms on disk. The API keeps both paths: inline classes run in-process with no isolation (like Scenario today); on-disk charms run isolated by default. + +The matrix of "works inline but not isolated" surprises is closed by a **documentation convention** rather than a runtime check. Every operation introduced from step 4 onward — anything beyond `IsolatedContext.run()` (`deploy`, `add-unit`, `integrate`, `update-config`, `run_action`, …) — declares in its docstring two explicit fields: + +- `inline: ok | error | n/a` +- `isolated: ok | error | n/a` + +Where `ok` means works as documented, `error` means raises a clear typed error explaining why it's not supported in this mode, and `n/a` means doesn't make sense in this mode. + +Enforced by API-doc review at PR time. Trivial to grep for. The matrix becomes readable in the docs rather than discoverable only by trial. No separate validation layer. + +## Real-world charm coverage + +The isolation and provisioning spikes were demonstrated end-to-end on synthetic test charms with intentionally conflicting deps, and the provisioning detection/extraction path was validated against two real Canonical charms (loki-k8s-operator, postgresql-k8s-operator) via a read-only `describe` CLI. Full end-to-end "provision venv, run the charm's code" against a real production charm remains unproven at spike scope. Likely first regression class: Python-version mismatch (charms targeting `requires-python = "~=3.12"` in a 3.11 harness). + +## Out of scope + +- Changes to Scenario's existing single-charm API or behaviour +- charmcraft `reactive` plugin support (not in current use; defer) + +## Prior art and references + +- [Catan](https://github.com/PietroPasotti/catan) — the abstraction this spec draws from; spike branch `explore/per-charm-dependency-isolation` at commit `1f4c357` +- [operator#1929](https://github.com/canonical/operator/issues/1929) — origin of the Saddle name and the `ops.testing`-integrated direction + +# Spec History and Changelog + +| Author(s) | Status | Date | Comment | +| :---- | :---- | :---- | :---- | +| [Tony Meyer](mailto:tony.meyer@canonical.com) | Draft | 26 May 2026 | Initial draft; seeded from the isolation spike, the environment-provisioning spike, and the decision to target `ops.testing` rather than a companion package | +| [Tony Meyer](mailto:tony.meyer@canonical.com) | Draft | 3 June 2026 | Resolved six open questions: StoredState fidelity (always-on typed escape hatch), JSON `State` schema stability (explicit `state_schema_version` field + dispatch table), settle-policy determinism requirement (byte-for-byte reproducible), inline-vs-on-disk docstring convention, perf 2× acceptance bar (4-charm/20-event workload), metadata-resolution six-shape step-6 acceptance set. Small follow-up survey of in-tree charm `StoredState` usage is queued separately. | +| Person | Pending Review | Date | | From fa88e79186ee43847ff5c485ba2b1963ca3f140a Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 24 Jul 2026 16:51:06 +1200 Subject: [PATCH 2/4] docs: refresh spec after reviewer comment responses - v1 charm= is local path only; Charmhub deferred; git-deploy out of scope - integrate() takes App objects (Jubilant-shaped); get_endpoint() for disambiguation - Elaborate on determinism vs Juju concurrency; anticipate settle() strategy args - Add stepping() context + settle() event/state trace for mid-flight / post-hoc assertions - Cross-ops-version compat policy: handshake negotiation, directionality (parent-authoritative, orchestrator ops >= any charm ops), documented compat window, pin_ops escape hatch - Thread step-3 measured timings into acceptance criteria; 2x bar up for revision - Restructure Environment provisioning as three layers (inherit / explicit python+requirements / opt-in auto-provision helper) --- ...multi-charm-unit-testing-in-ops-testing.md | 668 +++++++++++++++--- 1 file changed, 556 insertions(+), 112 deletions(-) diff --git a/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md b/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md index ad9c5ee..daa1cd9 100644 --- a/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md +++ b/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md @@ -1,48 +1,46 @@ -| Index | OPxxx (number not yet claimed) | | | +| Index | OP089 | | | | :---- | :---- | :---- | :---- | -| Title | Saddle: Multi-Charm Unit Testing in ops.testing | | | -| **[Type](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Author(s)** | **[Status](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?tab=t.0)** | **Created** | -| Implementation | [Tony Meyer](mailto:tony.meyer@canonical.com) | Draft | 26 May 2026 | +| Title | Multiple-charm state transition tests | | | +| **[Type](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Author(s)** | **[Status](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Created** | +| Implementation | [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 8, 2026 | | | **Reviewer(s)** | **Status** | **Date** | | | Person | Pending Review | Date | # Abstract -This spec defines **Saddle**: a multi-charm unit (state-transition) test surface folded into `ops.testing`. It provides `deploy`, `integrate`, `add-unit`, `remove-unit`, and `update-config` as test operations, with the corresponding Juju events triggered automatically as the model converges. Each charm runs in its own isolated subprocess (its own venv), so charms with mutually incompatible dependency sets can be tested together in a single test. Saddle targets `ops.testing` in `canonical/operator`; it is not a companion package. It draws the abstraction from [Catan](https://github.com/PietroPasotti/catan) (Apps + convergence) but is not a direct port. +This spec defines a multi-charm unit (state-transition) test surface folded into `ops.testing`. It provides `deploy`, `integrate`, `add-unit`, `remove-unit`, and `update-config` as test operations, with the corresponding Juju events triggered automatically as the model converges. Each charm runs in its own isolated subprocess (its own venv), so charms with mutually incompatible dependency sets can be tested together in a single test. This spec targets `ops.testing` in `canonical/operator`; it is not a companion package. It draws the abstraction from [Catan](https://github.com/PietroPasotti/catan) (Apps \+ convergence) but is not a direct port. # Rationale -## The gap in current tooling +### **The gap in current tooling** `ops.testing` (Scenario) runs one charm at a time: the developer writes an event and a `State`, calls `Context.run`, and asserts on the output. This is efficient for single-charm logic but leaves four pain points unaddressed: -- **Integration tests are too slow for the inner loop.** A real Juju controller bootstrap and deploy round-trip takes minutes; unit tests must be measured in seconds. -- **Scenario is unlike the `juju` commands charmers think in.** Charmers reason about `juju deploy`, `juju integrate`, `juju add-unit`. Translating that mental model into hand-crafted event sequences and `State`s is friction the tool should absorb. -- **Charm libraries don't yet ship mocks.** Cross-charm relation data is hand-crafted per test. Running the other charm's own code against the relation removes that hand-crafting: the other charm's actual reaction becomes the fixture. This may change with the acceptance of OP077 — Charm library testing. -- **Bundle and topology assertions are unreachable from unit tests.** Asserting "if I deploy A and B and integrate them, A's leader ends up with config X" requires real Juju or fragile mocks today. +- **Integration tests are slow.** A real Juju controller bootstrap and deploy round-trip takes minutes; unit tests should be measured in seconds. +- **Scenario is unlike the `juju` commands charmers think in.** Charmers reason about `juju deploy`, `juju integrate`, `juju add-unit`. Translating that mental model into hand-crafted event sequences and `State`s is friction the tool should absorb. This has been evident in the success of Jubilant (which itself further strengthens the issue, because charmers are now also used to thinking that way in integration tests too). +- **Charm libraries don't yet ship mocks.** Cross-charm relation data is hand-crafted per test. Running the other charm's own code against the relation removes that hand-crafting: the other charm's actual reaction becomes the fixture. This may change with the acceptance of [OP077](https://docs.google.com/document/d/1A50MlUpNz6YPVvIFxFZYTDBgxNboVYlpfZB8uy8o58c/edit?usp=sharing) (Charm library testing). +- **Bundle (or solution) and topology assertions are unreachable from unit tests.** Asserting "if I deploy A and B and integrate them, A's leader ends up with config X" requires real Juju or fragile mocks today. -## Validated abstraction +The shape charmers ask Charm Tech for is ‘a multi-charm state-transition test tool that feels like Juju’. This is Catan’s abstraction (Apps, `integrate`, convergence / settle). -The shape charmers ask Charm Tech for is "a multi-charm state-transition test tool that feels like Juju". This is Catan's abstraction (Apps, `integrate`, convergence/settle), validated by demand rather than speculative. - -## Why ops.testing rather than a companion package - -Catan today pins `ops-scenario~=6.0`, a superseded API that has been merged into `ops.testing` as Scenario 7. Porting off that API is **net-new work regardless of the destination**; there is no path that productises the companion-package form without a full re-targeting. Since we have to re-target anyway, we target `ops.testing` directly. +Why ops.testing rather than a companion package? Catan today pins `ops-scenario~=6.0`, a superseded API that has been merged into `ops.testing` as Scenario 7 (Scenario 8 is identical to 7, other than requiring Python 3.10+). Porting off that API is **net-new work regardless of the destination**; there is no path that productises the companion-package form without a full re-targeting. Since we have to re-target anyway, we target `ops.testing` directly. This is a non-trivial addition to ops-scenario, but not so large that it merits a new ops-{something} package, with the packaging overhead that comes from that. # Specification -## Location and scope +### **Location and scope** + +The new surface lives in `ops.testing` (`canonical/operator`), the same module that provides `Context` and `State`. The single-charm primitives are **unchanged and remain the substrate**. The new surface adds a model-level layer above `Context` that owns a set of `App`s, an integration graph, and a convergence loop. Single-charm Scenario (state transition) tests continue to work exactly as today and do not require the new surface. -The new surface lives in `ops.testing` (`canonical/operator`), the same module that provides `Context`, `State`, and `Manager`. The single-charm primitives are **unchanged and remain the substrate**. The new surface adds a model-level layer above `Context` that owns a set of `App`s, an integration graph, and a convergence loop. Single-charm Scenario tests continue to work exactly as today and do not require the new surface. +### **Operation surface** -## Operation surface +The test API uses Juju-shaped operations (`deploy`, `integrate`, `add-unit`, `remove-unit`, `update-config`, `run_action`) that describe **intents, not events**. The model generates the correct Juju-faithful event sequence and delivers it to the affected units. -The test API uses Juju-shaped operations — `deploy`, `integrate`, `add-unit`, `remove-unit`, `update-config`, `run_action` — that describe **intents, not events**. The model generates the correct Juju-faithful event sequence and delivers it to the affected units. +# -```python +```py from ops import testing -model = testing.Model() +model = testing.Model() # Already exists, but *not* the same as ops.Model! db = model.deploy( charm="./charms/postgresql", @@ -56,85 +54,154 @@ web = model.deploy( num_units=2, ) -model.integrate(db["db"], web["pg"]) +model.integrate(db, web) +# Or, when the endpoints need to be disambiguated: +# model.integrate(db.get_endpoint("db"), web.get_endpoint("pg")) model.add_unit(web) model.update_config(web, {"log_level": "debug"}) model.remove_unit(web, unit=1) model.run_action(db.leader, "create-backup", params={}) ``` +# + `Model.deploy(...)` returns an `App` handle. An `App` owns its charm reference, the metadata resolved from the charm on disk, the environment it runs in (see Isolation model below), and its per-unit `State`s. Exact type names are illustrative and will be settled in code review. -## Automatic event triggering and convergence +`Model.integrate(...)` follows Jubilant's shape but takes `App` objects rather than strings — we have the objects here, and there is no live Juju model to look them up in. The default form `integrate(db, web)` works whenever the endpoint pair is unambiguous from the two charms' metadata. When it isn't, endpoints are named explicitly via `App.get_endpoint(name)`: `integrate(db.get_endpoint("db"), web.get_endpoint("pg"))`. This mirrors Jubilant's `integrate("postgresql:db", "myapp:pg")` while keeping the API object-typed. + +### **Charm source: local paths only in v1** + +`charm=` in v1 is **a path to charm source on the local filesystem**. Type is `pathlib.Path | str`; a `str` must be a relative path, matching `juju deploy` behaviour. This covers test charms and monorepos out of the box. -Each operation enqueues the appropriate events (`relation-created`, `relation-joined`, `relation-changed`, `config-changed`, `leader-elected`, etc.) on the affected units. A **convergence loop** drains the queue: it picks an event, dispatches it through that unit's `Context.run(...)`, observes any new relation-data writes, status changes, or secret operations, and enqueues the downstream events those produce on peers and related units. Convergence terminates when the queue is empty and no unit's last run produced new observable change. +**Not in v1:** -```python +- **Deploy from git** (with or without patches, as Catan supports) is out of scope for v1 and possibly ever — the complexity is high and the value versus a conftest-managed clone is low. +- **Deploy from Charmhub** — feasible (download → unzip → treat the unpacked source as a local path), but not v1. If added later, the same `charm=` argument would grow a Charmhub-URL form; no prototype yet. + +**Recommendation for charms defined outside your repository:** the test infra (typically a `conftest.py` fixture) should source the charm from somewhere — `git clone` is the obvious choice — and pass the resulting local path to `deploy(charm=...)`. This keeps the harness surface small and puts the "where does this charm come from" decision in the test author's hands, where it can be tailored to CI, caching, and gitignore rules per project. + +### **Automatic event triggering and convergence** + +Each operation enqueues the appropriate events (`relation-created`, `relation-joined`, `relation-changed`, `config-changed`, and so on) on the affected units. A **convergence loop** drains the queue: it picks an event, dispatches it through that unit's `Context.run(...)`, observes any new relation-data writes, status changes, or secret operations, and enqueues the downstream events those produce on peers and related units. Convergence terminates when the queue is empty and no unit's last run produced new observable changes. + +```py model.settle() # explicit convergence point assert web.leader.state.unit_status == ActiveStatus("ready") assert db.leader.state.get_relation("db").local_app_data["database"] == "myapp" ``` -`settle()` is also called implicitly before reading any unit's `State` if there are pending events, so most tests are a sequence of operations followed by assertions on `app.leader.state` or `app.units[i].state`. The `State` exposed on each unit is the same `ops.testing.State` dataclass that single-charm Scenario tests use today; all existing assertions are available unchanged. +Catan shows that this can be done with suitable fidelity to Juju without *actually reimplementing Juju*. We are able to take advantage of the non-specific ordering of some Juju events, and also of the access to the Juju code to understand its behaviour. + +`settle()` is also called implicitly before reading any unit’s `State` if there are pending events, so most tests are a sequence of operations followed by assertions on `app.leader.state` or `app.units[i].state`. The `State` exposed on each unit is the same `ops.testing.State` dataclass that single-charm Scenario tests use today; all existing assertions are available unchanged. + +For tests that need to assert on **intermediate** state — partway through convergence, or on the exact sequence of events dispatched — two complementary APIs are exposed: + +- **Step-at-a-time context** — mirrors Scenario's `with ctx(...) as mgr: mgr.run(...)` shape. Inside the block, the implicit-settle-on-read is suspended so assertions between steps do not drain the queue: + + ```py + with model.stepping() as stepper: + stepper.step() # dispatch exactly one queued event + assert web.leader.state.unit_status == MaintenanceStatus("configuring") + stepper.step() + assert ... + # outside the block, implicit-settle-on-read is back + ``` + + `stepper.step()` returns the `(event, unit)` it just dispatched so the loop can be predicate-driven (`while stepper.step() != ("relation-changed", web.leader): ...`) without a separate peek API. Exiting the block re-enables implicit settling. + +- **Recorded event trace** — `settle()` returns the deterministic sequence of `(event, unit, state_snapshot)` tuples it dispatched, so post-hoc assertions can walk what happened without needing to steer the loop: + + ```py + trace = model.settle() + assert [e.name for e, _, _ in trace] == ["relation-created", "relation-joined", "relation-changed", "config-changed"] + ``` + + No behaviour change to the default settle path — the return value is additive. + +Mid-flight assertions with `stepping()`, post-hoc assertions with the returned trace; neither leaks into the default settle path a simple test uses. + +Given the same starting `Model` and the same input event sequence, `settle()` **must** produce the same final `State` per unit and the same sequence of intermediate events, byte-for-byte, across runs and across machines. **No wall-clock dependence, no PID-derived ordering, no Python-dict-iteration-order leakage** is permitted in the policy. The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is decided against this requirement, when there is working code to validate it. See [Further Information](#settle-policy-considerations). + +Real Juju runs charms concurrently and per-unit event ordering is non-deterministic. Unit tests must be deterministic, so v1 ships a single ordering policy — but the design keeps the door open for exercising alternative orderings against the *same* starting sequence of events. Two mechanisms are anticipated (concrete API picked in step 5 alongside the default policy): + +- **Alternative strategies via `settle(...)` arguments** — e.g. `model.settle(strategy="data-flow")` vs `strategy="round-robin"` vs a caller-supplied deterministic function. Running the same test under two strategies is one way to smoke out charms that accidentally depend on one particular ordering. +- **Per-app / per-unit scheduling hints** — e.g. mark an `App` as "running late" so its events are held until the queue is otherwise empty, or bias one unit to drain first. This lets charmers reproduce specific ordering bugs seen in production. -**Determinism requirement (resolved 2026-06-03).** Given the same starting `Model` and the same input event sequence, `settle()` MUST produce the same final `State` per unit and the same sequence of intermediate events, byte-for-byte, across runs and across machines. **No wall-clock dependence, no PID-derived ordering, no Python-dict-iteration-order leakage** is permitted in the policy. The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is picked in step 5 against this requirement, when there is working code to validate it. See Further Information. +Both are optional layers on top of the default deterministic policy; the default remains what `settle()` with no arguments does. -## Isolation model: subprocess + per-charm venv +### **Isolation model: subprocess \+ per-charm venv** -Each `App` runs in its own process with its own interpreter (an existing venv's `bin/python`, or one the harness provisioned — see Environment provisioning below). For each event dispatch the harness serialises `(meta, config, actions, event, State)` as JSON, sends it to the charm's worker process, the worker runs `testing.Context(...).run(...)`, and serialises the resulting `State` back. Two `App`s in the same `Model` may use mutually incompatible dependency sets — `cryptography==41` vs `==3`, different `pydantic` majors, conflicting charm-lib versions — demonstrated end-to-end by the isolation spike: two charms with intentionally conflicting `confdep` versions, each in its own venv, both producing correct output in the same test session via both a lightweight `sys.path`-injected dep dir and a full real venv per charm. +Each `App` runs in its own process with its own interpreter (an existing venv's `bin/python`, or one the harness provisioned; see Environment provisioning below). For each event dispatch the harness serialises `(meta, config, actions, event, State)` as JSON, sends it to the charm's worker process, the worker runs `testing.Context(...).run(...)`, and serialises the resulting `State` back. Two `App`s in the same `Model` may use mutually incompatible dependency sets (`cryptography==41` vs `==3`, different `pydantic` majors, conflicting charm-lib versions, …). -**Required invariant:** each per-charm venv contains an `ops` / `ops-scenario` build whose JSON `State` wire format is compatible with the parent harness's. Only the charm's own runtime deps differ between venvs — realistic, since all charms build on `ops`. +**Required invariant:** each per-charm venv contains an `ops` / `ops-scenario` build whose JSON `State` wire format is compatible with the parent process's. Only the charm's own runtime deps differ between venvs. This is realistic, since all charms (that matter here) build on `ops`. + +**Cross-`ops`-version compatibility policy.** Running each charm in its own venv means charms may sit on different `ops` versions than the parent process — a real ongoing cost. The policy has three parts: + +- **Version negotiation at worker handshake.** On startup, the worker declares the wire-format version it supports. The parent process picks the intersection or raises `StateSchemaVersionError` (already introduced by the typed-`State` serialiser, step 2 of the incremental delivery plan). Mismatches fail fast at handshake, not silently mid-test. +- **Directionality: unidirectional, parent-authoritative.** The parent process (running `ops.testing`) is the authority; it supports schema versions `1..N` where `N` is its own current version. A charm worker on any version in `1..N` is fine — the parent decodes at the worker's declared version. A worker on `>N` (a newer `ops` than the parent) raises at handshake, because the parent cannot invent decoder logic for a future schema. **Practical invariant: the orchestrating test suite's `ops` must be ≥ every charm-under-test's `ops`.** This matches the natural test-suite topology (bump `ops.testing` first, then adopt in charms) and keeps the compat matrix strictly triangular rather than a full N×N grid. +- **Documented compatibility window.** Wire-format changes are treated as `ops` release-note items with an explicit "supported charm-side `ops` versions" range published against each `ops.testing` release — this is the *floor* (how far back a charm can be); the *ceiling* is the parent's own version (directionality, above). Charms outside the window raise at handshake with a message pointing at the range. Bounded surface, small compat matrix, testable. +- **Escape hatch: pin the charm's `ops` to the parent process's.** When `build_env=True`, the provisioner can override the charm's declared `ops` pin with the parent process's exact version via `deploy(..., pin_ops=True)`. Cuts through the compat matrix at the cost of not exercising the charm on its own declared `ops`. Off by default; opt-in per `deploy(...)`. The isolation worker is persistent: one long-lived process per `App`, spawned lazily on first dispatch and torn down at `Model` teardown or after an idle timeout. A framed JSON protocol over stdin/stdout carries requests and responses. Spawn-per-event remains available as an explicit debug mode (easier to attach a debugger; no shared interpreter state between events). Worker crashes propagate as a typed `IsolationError` and fail the test cleanly; the harness never silently re-spawns mid-test. -**Acceptance criterion (fixed 2026-06-03):** +Acceptance criteria: + +- **Workload:** the performance yardstick is a **4-charm bundle with 20 events per `settle()`** (80 dispatches total). Lives in the test suite and is re-run by every later step to catch regressions. +- **Bar:** **persistent-worker mode within 2× the in-process Scenario baseline** on that workload. 2× is the threshold where "slow but acceptable for a CI unit-test suite" tips into "too slow, charmers will avoid it and just use integration tests"; subprocess \+ JSON serialisation has a real floor below that. +- **Benchmark shape:** a benchmark suite measuring (a) in-process Scenario baseline, (b) spawn-per-event isolated, and (c) persistent-worker isolated, all on the same 4-charm/20-event workload. +- **Measured absolute numbers (step 3 draft implementation, Python 3.11.15, median of 5 reps):** -- **Workload:** the canonical perf yardstick is a **4-charm bundle with 20 events per `settle()`**. Lives in the test suite from step 3 onward and is re-run by every later step to catch regressions. -- **Bar:** **persistent-worker mode within 2× the in-process Scenario baseline** on that workload. 2× is the threshold where "slow but acceptable for a CI unit-test suite" tips into "too slow, charmers will avoid it"; subprocess + JSON serialisation has a real floor below that. -- **Benchmark shape:** the step 3 PR ships a benchmark suite measuring (a) in-process Scenario baseline, (b) spawn-per-event isolated, and (c) persistent-worker isolated, all on the same 4-charm/20-event workload. -- **If (c) misses the bar:** the PR either records a clear "this is slower than the target; here's why; here's what could close the gap" trade-off note, or the persistent-worker design is reconsidered before step 3 is signed off. + | mode | total (80 dispatches) | per event | vs baseline | + | :---- | :---- | :---- | :---- | + | (a) in-process Scenario baseline | 0.332 s | 4.15 ms | 1.00× | + | (c) persistent-worker isolated | 1.671 s | 20.89 ms | **5.04×** | + | (b) spawn-per-event isolated | 22.94 s | 286.77 ms | 69× | -### Subinterpreters are explicitly ruled out + Persistent-worker mode does **not** currently meet the 2× bar (see step 3 log in [`step3-persistent-worker-log.md`](./step3-persistent-worker-log.md)). Dominant cost is `import ops` per worker (~243 ms), paid once per isolated environment and structurally unshareable across environments; JSON/IPC is sub-millisecond per event. On this 4-worker workload the ~1.1 s of worker spawns alone already exceeds the 2× budget (~0.66 s). Persistent is nonetheless ~13.7× faster than spawn-per-event and is the correct foundation for step 5's convergence loop, where more events per worker amortise the spawn cost further. **The 2× bar itself is up for revision** in light of the numbers — the working assumption is that the threshold is comfortably higher than 2× in practice, and step 5 will re-measure with real bundle-shaped workloads before locking it in. -PEP 734 subinterpreters are **not** the mechanism, even granting a 3.14+ floor: +### **Subinterpreters are explicitly ruled out** -- **Binary-dependency conflicts remain unsolved.** Many C extensions (`cryptography`, `pydantic-core`, `cffi`-based packages) are not multi-interpreter-safe; two different versions of the same extension cannot coexist in one process at any Python version. These are the exact libraries motivating the work. -- **State boundary still costs a serialisation roundtrip** — the same JSON encode/decode as a subprocess, taken without the isolation benefit. +PEP 734 subinterpreters are **not** the mechanism, even assuming a Python 3.14+ requirement would be acceptable: + +- **Binary-dependency conflicts remain unsolved.** Many C extensions (`cryptography`, `pydantic-core`, `cffi`\-based packages) are not multi-interpreter-safe; two different versions of the same extension cannot coexist in one process at any Python version. These are the exact libraries motivating the work. +- **State boundary still costs a serialisation roundtrip**. The same JSON encode/decode as a subprocess, taken without the isolation benefit. - **Maintenance cost is high** on a bleeding-edge API for no payoff over the subprocess design. -Revisit only if the C-extension multi-interpreter story matures enough to load conflicting binary versions in one process — not merely because the Python version floor moves. +This will be revisited only if the C-extension multi-interpreter story matures enough to load conflicting binary versions in one process, not merely because the Python version floor moves. ## Environment provisioning -How a charmer declares an `App`'s environment: +The environment API is deliberately layered so the core primitive has a small surface and no external-tool dependencies, and the ergonomic auto-detection form is a convenience helper on top rather than the load-bearing default. + +**Layer 1 — inherit the test process** (default; no `python=` argument). No isolation, no venv work, no external tools. Runs the charm in the same interpreter as the test. Unblocks Scenario-style tests with locally-defined test charms and monorepo cases where dependencies are known compatible. + +**Layer 2 — explicit interpreter + resolved requirements** (`Model.deploy(..., python="/path/to/venv/bin/python", requirements=[...])`). The core primitive: a Python interpreter path plus a fully-resolved list of pip specifiers. Zero magic; no charmcraft-plugin knowledge; no dependency on `uv` or `poetry`. The caller has already done whatever resolution they want to do (their own build script, their own lockfile export, whatever). This form unlocks the conflicting-dependency case without pulling any external-tool complexity into the core. -| Form | API | When to use | -| --- | --- | --- | -| Auto-provision from declared deps | `Model.deploy(..., build_env=True)` | Canonical recommended UX | -| Point at an existing venv | `Model.deploy(..., python="/path/to/venv/bin/python")` | CI where the venv is already managed | -| Inherit the test process | explicit opt-in flag | Only when deps are known not to conflict | + Passing `python=` without `requirements=` means "use this interpreter as-is" — the caller has already provisioned the venv. Passing `requirements=` without `python=` is an error. -For auto-provisioning the harness detects the charm's build plugin (from `charmcraft.yaml` first; file-based heuristic second — `uv.lock` → `uv`; `poetry.lock` → `poetry`; `pyproject.toml` → `python`; else → `charm`) and extracts the dependency list: +**Layer 3 — auto-provision from declared deps** (opt-in `Model.deploy(..., build_env=True)`). A convenience helper that detects the charm's build plugin, extracts the dependency list, and calls into layer 2 with the extracted values. Charmers who want ergonomics use this; charmers whose plugin detection fails, or who want control, drop to layer 2 without losing anything. -| Plugin | Extraction method | -| --- | --- | -| `charm` | reads `charm-requirements` files + `charm-python-packages` + AST-harvested `PYDEPS` from `lib/**/*.py` | -| `python` | reads `python-requirements` / `python-packages` + PYDEPS (included even though charmcraft's python plugin does not auto-install them — the test harness needs the lib importable) | -| `uv` | shells to `uv export --frozen --no-emit-project --no-hashes --format requirements-txt` with forwarded `--extra` / `--group` flags | -| `poetry` | shells to `poetry export --without-hashes --format requirements.txt` with forwarded `--with` flags | + Plugin detection is `charmcraft.yaml` first; file-based heuristic second (`uv.lock` → `uv`; `poetry.lock` → `poetry`; `pyproject.toml` → `python`; else → `charm`). Extraction per plugin: -All forms normalise to a flat pip specifier list, then install via `uv pip install` into the per-charm venv. The venv is cached on disk, keyed by SHA-256 of (plugin, extracted requirements, lockfile/pyproject content, interpreter version, harness `ops` version): cold build ~3s for a test charm; warm cache hit <0.05s (measured). The detection-and-extraction path was validated against two real Canonical charms (loki-k8s-operator via the uv plugin; postgresql-k8s-operator via the poetry plugin); the harness emits a clear, typed error when an extracted requirement cannot be resolved against an available index, when a lockfile is stale relative to the pyproject, or when a required external tool (`uv` / `poetry`) is missing. + | Plugin | Extraction method | + | :---- | :---- | + | charm | reads `charm-requirements` files \+ `charm-python-packages` \+ AST-harvested `PYDEPS` from `lib/**/*.py` | + | python | reads `python-requirements` / `python-packages` \+ PYDEPS (included even though charmcraft's python plugin does not auto-install them: the isolation worker needs the lib importable) | + | uv | shells to `uv export --frozen --no-emit-project --no-hashes --format requirements-txt` with forwarded `--extra` / `--group` flags | + | poetry | shells to `poetry export --without-hashes --format requirements.txt` with forwarded `--with` flags | -**Tool dependency:** `uv` on PATH for all install steps; `poetry` + `poetry-plugin-export` for the poetry-plugin path. The harness emits a clear error when a required tool is absent. + All forms normalise to a flat pip specifier list, then install via `uv pip install` into the per-charm venv. The venv is cached on disk, keyed by SHA-256 of (plugin, extracted requirements, lockfile/pyproject content, interpreter version, parent-process `ops` version): cold build \~3s for a test charm; warm cache hit \<0.05s (measured). Detection-and-extraction was validated against two real charms (loki-k8s-operator via the uv plugin; postgresql-k8s-operator via the poetry plugin). Layer 3 emits a clear, typed error when an extracted requirement cannot be resolved, when a lockfile is stale relative to the pyproject, or when a required external tool (`uv` / `poetry`) is missing — the failure mode is "drop to layer 2", not "fall through silently". -**Future:** a charmcraft-managed flow for charms already packed locally. +**Tool dependency (layer 3 only):** `uv` on PATH for all install steps; `poetry` \+ `poetry-plugin-export` for the poetry-plugin path. Layers 1 and 2 depend on nothing beyond the Python standard library. This scopes the external-tool concern to the convenience layer where charmers opted in to the ergonomics. + +**Future:** a charmcraft-managed flow for charms already packed locally, sitting at layer 3 alongside the plugin-based extraction. ## State (de)serialisation -A typed schema-aware encoder/decoder pair — not `json.dumps(dataclasses.asdict(state))` — covers all `ops.testing.State` leaf types verified against the spike's findings: +A typed schema-aware encoder/decoder pair (not `json.dumps(dataclasses.asdict(state))`) covers all `ops.testing.State` leaf types: | Type | Encoding | -| --- | --- | +| :---- | :---- | | `frozenset` (on `relations`, `containers`, `secrets`, etc.) | list on encode; re-frozen on decode (dataclass `__post_init__`s re-freeze) | | `datetime` / `timedelta` | ISO-8601 string / fractional seconds | | `pathlib.Path` / `PurePosixPath` | string with type tag | @@ -142,112 +209,489 @@ A typed schema-aware encoder/decoder pair — not `json.dumps(dataclasses.asdict | `pebble.Layer` (`Container.layers`) | `to_dict()` / `Layer(dict)` round-trip (not a dataclass; `asdict` won't descend) | | Int-keyed dicts (`Secret.remote_grants: Mapping[int, set[str]]`) | keys stringified on encode; restored to `int`, inner `set` restored on decode | -**StoredState fidelity (resolved 2026-06-03).** `StoredState.content` and `Container._base_plan` are `Mapping[str, Any]` of marshallable types — a superset of JSON: `bytes`, `tuple`-vs-`list` distinction, and arbitrary `set`s. The spec commits to: +`StoredState.content` and `Container._base_plan` are `Mapping[str, Any]` of marshallable types, a superset of JSON: `bytes`, `tuple`\-vs-`list` distinction, and arbitrary `set`s. The spec commits to: -1. **A typed escape hatch, applied always** — tag-and-encode `bytes` as base64 and `tuple` as a `["__tuple__", ...]` envelope so they round-trip losslessly. **No flag.** An opt-in would create a "passed locally because today's stored-state had no `bytes`, failed in CI when it did" failure mode; the cost (a few extra bytes per typed value) doesn't justify it. -2. **Loud error on unrecognised types** — the encoder raises `TypeError` with the type name and the path through `State`. The decoder mirrors: unknown `__type__` tags raise `TypeError`. Never silent lossy coercion. +1. **A typed escape hatch, applied always**: tag-and-encode `bytes` as base64 and `tuple` as a `["__tuple__", ...]` envelope so they round-trip losslessly. **No flag.** An opt-in would create a "passed locally because today's stored-state had no `bytes`, failed in CI when it did" failure mode; the cost (a few extra bytes per typed value) doesn't justify it. +2. **Loud error on unrecognised types**: the encoder raises `TypeError` with the type name and the path through `State`. The decoder mirrors: unknown `__type__` tags raise `TypeError`. Never silent lossy coercion. A small follow-up survey of in-tree charm `StoredState` usage is left as a separate work item, to confirm no charm relies on round-tripping more exotic marshallable types than `bytes`/`tuple`/`set`. The survey does not gate the implementation; it informs whether the encoder's type set needs to be extended. -**JSON `State` schema stability (resolved 2026-06-03).** The JSON `State` representation is **part of `ops.testing`'s public API** — a per-charm venv may have a different `ops` build from the parent, and both sides must agree on the encoded form. +The JSON `State` representation is **part of `ops.testing`'s public API**: a per-charm venv may have a different `ops` build from the parent, and both sides must agree on the encoded form. -1. **Explicit `state_schema_version` integer field** is embedded in every encoded `State` payload. Cheaper and more legible than overloading `ops`'s SemVer: we don't have to ship an `ops` major just because we added one field to `State` serialisation. -2. **Per-version interpretation is locked at the source-code level** — a dispatch table keyed by version. Adding a new version is ordinary work; changing the *meaning* of an existing version is not. -3. **Decoder rejects unsupported versions with a clear typed error.** A v1 producer and a v1+v2 consumer work fine. A version the decoder doesn't recognise surfaces as an explicit error, not silent corruption. +1. **Explicit `state_schema_version` integer field** is embedded in every encoded `State` payload. Cheaper and more legible than overloading `ops`'s SemVer: we don't have to ship an `ops` major just because we added one field to `State` serialisation. +2. **Per-version interpretation is locked at the source-code level**: a dispatch table keyed by version. Adding a new version is ordinary work; changing the *meaning* of an existing version is not. +3. **Decoder rejects unsupported versions with a clear typed error.** A v1 producer and a v1+v2 consumer work fine. A version the decoder doesn't recognise surfaces as an explicit error, not silent corruption. 4. **`ops.testing` API-stability documentation commits to (2).** "We may add new versions; we don't change the meaning of an existing version." -Implementation honours this from step 2 onward (the encoder writes `state_schema_version`; the decoder dispatches on it). +### **Serialisation examples** + +#### Container with a Pebble layer, notices, check infos, service statuses, and mounts + +This example shows the most type-envelope diversity in one place: `pebble.Layer` → **t**: "layer", `pebble.ServiceStatus` enum → **t**: "enum", `pebble.NoticeType` enum, `datetime` fields, `timedelta` fields, `PurePosixPath` and `Path` in a `Mount`, and `CheckInfo` containing multiple enum types. + +```py + state = State( + containers=frozenset([ + Container( + name='myapp', + can_connect=True, + layers={ + 'base': pebble.Layer({ + 'services': { + 'app': { + 'command': '/bin/app --port 8080', + 'startup': 'enabled', + 'override': 'replace', + }, + }, + }), + }, + service_statuses={'app': pebble.ServiceStatus.ACTIVE}, + notices=[ + Notice( + key='example.com/event', + type=pebble.NoticeType.CUSTOM, + repeat_after=datetime.timedelta(minutes=30), + first_occurred=datetime.datetime( + 2025, 6, 15, 10, 0, 0, tzinfo=datetime.timezone.utc + ), + last_occurred=datetime.datetime( + 2025, 6, 15, 10, 30, 0, tzinfo=datetime.timezone.utc + ), + last_repeated=datetime.datetime( + 2025, 6, 15, 10, 30, 0, tzinfo=datetime.timezone.utc + ), + ), + ], + check_infos=frozenset([ + CheckInfo( + name='http', + level=pebble.CheckLevel.ALIVE, + status=pebble.CheckStatus.UP, + ), + ]), + mounts={ + 'cfg': Mount( + location=pathlib.PurePosixPath('/etc/config'), + source=pathlib.Path('/tmp/cfg'), + ), + }, + ), + ]), + ) +``` + +```json + { + "state_schema_version": 1, + "state": { + "__t__": "dc", + "cls": "State", + "f": { + "config": {}, + "relations": {"__t__": "frozenset", "v": []}, + "networks": {"__t__": "frozenset", "v": []}, + "containers": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Container", + "f": { + "name": "myapp", + "can_connect": true, + "_base_plan": {}, + "layers": { + "base": { + "__t__": "layer", + "v": { + "services": { + "app": { + "startup": "enabled", + "override": "replace", + "command": "/bin/app --port 8080" + } + } + } + } + }, + "service_statuses": { + "app": {"__t__": "enum", "cls": "ServiceStatus", "name": +"ACTIVE"} + }, + "mounts": { + "cfg": { + "__t__": "dc", + "cls": "Mount", + "f": { + "location": {"__t__": "PurePosixPath", "v": +"/etc/config"}, + "source": {"__t__": "Path", "v": "/tmp/cfg"} + } + } + }, + "execs": {"__t__": "frozenset", "v": []}, + "notices": [ + { + "__t__": "dc", + "cls": "Notice", + "f": { + "key": "example.com/event", + "id": "1", + "user_id": null, + "type": {"__t__": "enum", "cls": "NoticeType", "name": +"CUSTOM"}, + "first_occurred": {"__t__": "datetime", "v": +"2025-06-15T10:00:00+00:00"}, + "last_occurred": {"__t__": "datetime", "v": +"2025-06-15T10:30:00+00:00"}, + "last_repeated": {"__t__": "datetime", "v": +"2025-06-15T10:30:00+00:00"}, + "occurrences": 1, + "last_data": {}, + "repeat_after": {"__t__": "timedelta", "v": 1800.0}, + "expire_after": null + } + } + ], + "check_infos": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "CheckInfo", + "f": { + "name": "http", + "level": {"__t__": "enum", "cls": "CheckLevel", "name": +"ALIVE"}, + "startup": {"__t__": "enum", "cls": "CheckStartup", +"name": "ENABLED"}, + "status": {"__t__": "enum", "cls": "CheckStatus", "name": +"UP"}, + "successes": 0, + "failures": 0, + "threshold": 3, + "change_id": "1" + } + } + ]} + } + } + ]}, + "storages": {"__t__": "frozenset", "v": []}, + "opened_ports": {"__t__": "frozenset", "v": []}, + "leader": false, + "model": {"__t__": "dc", "cls": "Model", "f": {"name": "...", "uuid": +"...", "type": "kubernetes", "cloud_spec": null}}, + "secrets": {"__t__": "frozenset", "v": []}, + "resources": {"__t__": "frozenset", "v": []}, + "planned_units": 1, + "deferred": [], + "stored_states": {"__t__": "frozenset", "v": []}, + "app_status": {"__t__": "status", "name": "unknown", "msg": ""}, + "unit_status": {"__t__": "status", "name": "unknown", "msg": ""}, + "workload_version": "" + } + } + } +``` + +#### Secrets with `int`\-keyed `remote_grants`, rotation policy, and Relations with `int`\-keyed `remote_units_data` + +This example shows the "idict" envelope (for `dict[int, ...]` which JSON can’t represent natively), the "set" envelope inside an `idict`, and `SecretRotate` enum. + +```py + state = State( + secrets=frozenset([ + Secret( + tracked_content={'password': 'x2', 'cert': '---BEGIN CERT---'}, + latest_content={'password': 'x3', 'cert': '---BEGIN CERT---'}, + owner='app', + label='db-creds', + description='Database credentials', + expire=datetime.datetime( + 2030, 12, 31, 23, 59, 59, tzinfo=datetime.timezone.utc + ), + rotate=SecretRotate.DAILY, + remote_grants={0: {'related-app'}, 2: {'other-app/0'}}, + ), + ]), + relations=frozenset([ + Relation( + endpoint='db', + interface='postgresql_client', + remote_app_name='postgresql', + remote_units_data={ + 0: {'host': '10.0.0.1', 'port': '5432'}, + 1: {'host': '10.0.0.2', 'port': '5432'}, + }, + remote_app_data={'version': '14', 'db': 'mydb'}, + ), + PeerRelation( + endpoint='peers', + peers_data={1: {'leader': 'true', 'ready': 'yes'}}, + ), + ]), + ) +``` + +```json + { + "relations": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Relation", + "f": { + "endpoint": "db", + "interface": "postgresql_client", + "id": 1, + "local_app_data": {}, + "local_unit_data": { + "egress-subnets": "192.0.2.0", + "ingress-address": "192.0.2.0", + "private-address": "192.0.2.0" + }, + "remote_app_name": "postgresql", + "limit": 1, + "remote_app_data": {"version": "14", "db": "mydb"}, + "remote_units_data": { + "__t__": "idict", + "v": { + "0": {"host": "10.0.0.1", "port": "5432"}, + "1": {"host": "10.0.0.2", "port": "5432"} + } + }, + "remote_model_uuid": null + } + }, + { + "__t__": "dc", + "cls": "PeerRelation", + "f": { + "endpoint": "peers", + "interface": null, + "id": 2, + "local_app_data": {}, + "local_unit_data": {"egress-subnets": "192.0.2.0", +"ingress-address": "192.0.2.0", "private-address": "192.0.2.0"}, + "peers_data": { + "__t__": "idict", + "v": {"1": {"leader": "true", "ready": "yes"}} + } + } + } + ]}, + "secrets": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Secret", + "f": { + "tracked_content": {"password": "hunter2", "cert": "---BEGIN +CERT---"}, + "latest_content": {"password": "hunter3", "cert": "---BEGIN +CERT---"}, + "id": "secret:oup6t3bz6wzkkjylya5u", + "owner": "app", + "remote_grants": { + "__t__": "idict", + "v": { + "0": {"__t__": "set", "v": ["related-app"]}, + "2": {"__t__": "set", "v": ["other-app/0"]} + } + }, + "label": "db-creds", + "description": "Database credentials", + "expire": {"__t__": "datetime", "v": "2030-12-31T23:59:59+00:00"}, + "rotate": {"__t__": "enum", "cls": "SecretRotate", "name": "DAILY"}, + "_tracked_revision": 1, + "_latest_revision": 1 + } + } + ]} + } +``` + +#### `StoredState` escape hatch (`bytes`, `tuple`, `set`) \+ statuses, resources, ports, and deferred events + +This example shows the escape-hatch types that `StoredState.content` can contain: `bytes` (base64-encoded), `tuple` (`list`\-based envelope), and `set`. It also shows the "status" envelope for non-trivial status values, "Path" in a Resource, and `DeferredEvent`. + +```py + state = State( + config={'log_level': 'debug', 'port': 8080, 'enabled': True}, + leader=True, + unit_status=ActiveStatus('ready'), + app_status=BlockedStatus('needs database'), + workload_version='1.2.3', + planned_units=3, + stored_states=frozenset([ + StoredState( + name='_stored', + owner_path='MyCharm', + content={ + 'count': 42, + 'raw': b'\xde\xad\xbe\xef', + 'coords': (1, 2, 3), + 'tags': {'alpha', 'beta'}, + 'mapping': {'nested': True}, + }, + ), + ]), + resources=frozenset([ + Resource(name='oci-image', path=pathlib.Path('/tmp/image.tar')), + ]), + opened_ports=frozenset([TCPPort(8080), UDPPort(53)]), + deferred=[ + DeferredEvent( + handle_path='MyCharm/on/config_changed[1]', + owner='MyCharm', + observer='_on_config_changed', + ), + ], + ) +``` + +```json + { + "config": {"log_level": "debug", "port": 8080, "enabled": true}, + "opened_ports": {"__t__": "frozenset", "v": [ + {"__t__": "dc", "cls": "UDPPort", "f": {"port": 53, "protocol": "udp"}}, + {"__t__": "dc", "cls": "TCPPort", "f": {"port": 8080, "protocol": +"tcp"}} + ]}, + "leader": true, + "resources": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Resource", + "f": { + "name": "oci-image", + "path": {"__t__": "Path", "v": "/tmp/image.tar"} + } + } + ]}, + "planned_units": 3, + "deferred": [ + { + "__t__": "dc", + "cls": "DeferredEvent", + "f": { + "handle_path": "MyCharm/on/config_changed[1]", + "owner": "MyCharm", + "observer": "_on_config_changed", + "snapshot_data": {} + } + } + ], + "stored_states": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "StoredState", + "f": { + "name": "_stored", + "owner_path": "MyCharm", + "content": { + "count": 42, + "raw": {"__t__": "bytes", "v": "3q2+7w=="}, + "coords": ["__tuple__", 1, 2, 3], + "tags": {"__t__": "set", "v": ["alpha", "beta"]}, + "mapping": {"nested": true} + }, + "_data_type_name": "StoredStateData" + } + } + ]}, + "app_status": {"__t__": "status", "name": "blocked", "msg": "needs +database"}, + "unit_status": {"__t__": "status", "name": "active", "msg": "ready"}, + "workload_version": "1.2.3" + } +``` ## Metadata resolution -The harness reads `metadata.yaml` / `charmcraft.yaml` from the charm-source path without importing the charm in the test process. Work required for productisation: +The harness reads `metadata.yaml` / `charmcraft.yaml` from the charm source path without importing the charm in the test process. Less common cases are left as future work: -- Resolve unified-vs-legacy metadata variants and the merged `charmcraft.yaml` + `metadata.yaml` form charms commonly produce -- Resolve charm libraries the charm declares but that live elsewhere on disk -- An isolated equivalent of Catan's `App.from_git` (clone, detect build system, provision venv, run) +- Resolve unified-vs-legacy metadata variants and the merged `charmcraft.yaml` \+ `metadata.yaml` form charms commonly produce +- Resolve charm libraries the charm declares but that live elsewhere on disk - Inline charm classes (`from_type` charms defined in the test): kept as in-process, same-environment only; isolation requires charms on disk (`from_isolated_path`). See Further Information. -**Step 6 acceptance set for metadata-resolution coverage (locked 2026-06-03).** At least one passing test for each of these charm shapes: +At least one passing test for each of these charm shapes must be included in the implementation: -1. Unified `charmcraft.yaml` only (no `metadata.yaml`) — the current default-modern shape. -2. Legacy split: `metadata.yaml` + `actions.yaml` + `config.yaml` — older charms still in production. -3. Mixed / transitional: `charmcraft.yaml` + `metadata.yaml` (the form charmcraft merges) — charms mid-migration. -4. Multi-bases (`bases:` / `platforms:` variants per supported platform). -5. Charm with libs outside `src/` (e.g. installed via `pip` from PyPI for testing) — exercises the "where did this lib come from" resolution. +1. Unified `charmcraft.yaml` only (no `metadata.yaml`) — the current default-modern shape. +2. Legacy split: `metadata.yaml` \+ `actions.yaml` \+ `config.yaml` — older charms still in production. +3. Mixed / transitional: `charmcraft.yaml` \+ `metadata.yaml` (the form charmcraft merges) — charms mid-migration. +4. Multi-bases (`bases:` / `platforms:` variants per supported platform). +5. Charm with libs outside `src/` (e.g. installed via `pip` from PyPI for testing) — exercises the "where did this lib come from" resolution. 6. Inline `from_type` (no metadata file; class declares everything) — documented as `inline: ok` per the docstring convention in Further Information. -## Migration from Catan / ops-scenario~=6.0 +## Migration from Catan / ops-scenario\~=6.0 -The existing `PietroPasotti/catan` package pins `ops-scenario~=6.0` (superseded by `ops.testing`). There is **no compatibility shim**: tests written against Catan must be rewritten against the Saddle surface. The `ops-scenario~=6.0` form is throwaway regardless of whether Saddle is adopted — porting off it is net-new work either way. +The existing `PietroPasotti/catan` package pins `ops-scenario~=6.0` (superseded by `ops.testing`). There is **no compatibility shim**: tests written against Catan must be rewritten against the new API surface. The `ops-scenario~=6.0` form is throwaway regardless of whether this spec is adopted. Porting off it is net-new work either way. -What Saddle carries forward from Catan: +What the spec carries forward from Catan: -- The model: Apps + integration graph + convergence/settle — the abstraction demand validates +- The model: Apps \+ integration graph \+ convergence/settle - The Juju-like operation vocabulary: `deploy` / `integrate` / `add-unit` / … -- The isolation and provisioning spikes as seed implementations for steps 1 and 6 - -What is discarded: - -- The `ops-scenario~=6.0` companion-package API -- The Catan package as a separate distribution artifact (PyPI release, separate repo, separate release cadence) ## Incremental delivery plan Sequenced so each merge is **independently useful** and the area can pause cleanly after any step. This is the handle on the priority reservation: agentic coding lowers per-step build cost, and the sequencing means a pause never strands a half-built feature. | Step | Deliverable | Value without later steps | -| --- | --- | --- | -| 1 | Lift isolation into `ops.testing`: `IsolatedEnv`, persistent worker, `from_isolated_path` | Immediately useful: run one charm in an isolated venv from a Scenario test; no `Model`, no convergence | -| 2 | Typed `State` (de)serialiser with full leaf-type coverage and the StoredState escape hatch | Snapshot-style tests and debugging; required by every later step | -| 3 | Persistent-worker mode replacing spawn-per-event (benchmarked in the same PR). **Acceptance:** within 2× of in-process Scenario on the 4-charm/20-event workload — see *Isolation model* | Performance baseline; unblocks realistic convergence-run testing | -| 4 | `Model` + `App` + non-relating operations: `deploy`, `add-unit`, `remove-unit`, `update-config`, `run_action`, single-app convergence | Drive a single charm through Juju-shaped operations without writing event sequences by hand | -| 5 | `integrate` + cross-app event propagation + settle policy locked. **Acceptance:** see *Settle policy considerations* (Further Information) — at least one deterministic policy passes a 100×-soak; user-override hook exposed | Unlocks the multi-charm pain points; steps 1–4 stand alone if priorities pull us off here | -| 6 | `build_env=True` environment provisioning (all four plugins, caching, lockfile handling) + full metadata-resolution coverage. **Acceptance:** the six-shape metadata test set — see *Metadata resolution* | Productisation tail; not on the critical path for any earlier step | +| :---- | :---- | :---- | +| 1 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/18)) | Lift isolation into `ops.testing`: `IsolatedEnv`, persistent worker, `from_isolated_path` | Immediately useful: run one charm in an isolated venv from a Scenario test; no `Model`, no convergence | +| 2 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/20)) | Typed `State` (de)serialiser with full leaf-type coverage and the StoredState escape hatch | Snapshot-style tests and debugging; required by every later step | +| 3 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/22)) | Persistent-worker mode replacing spawn-per-event (benchmarked in the same PR). **Acceptance:** within 2× of in-process Scenario on the 4-charm/20-event workload. See *Isolation model* | Performance baseline; unblocks realistic convergence-run testing | +| 4 | `Model` \+ `App` \+ non-relating operations: `deploy`, `add-unit`, `remove-unit`, `update-config`, `run_action`, single-app convergence | Drive a single charm through Juju-shaped operations without writing event sequences by hand | +| 5 | `integrate` \+ cross-app event propagation \+ settle policy locked. **Acceptance:** see *Settle policy considerations* (Further Information), at least one deterministic policy passes a 100×-soak; user-override hook exposed | Unlocks the multi-charm pain points; steps 1–4 stand alone if priorities pull us off here | +| 6 | `build_env=True` environment provisioning (all four plugins, caching, lockfile handling) \+ full metadata-resolution coverage. **Acceptance:** the six-shape metadata test set. See *Metadata resolution* | Productisation tail; not on the critical path for any earlier step | # Further Information -## Settle policy considerations (resolved 2026-06-03) +### **Settle policy considerations** {#settle-policy-considerations} The determinism requirement is locked in *Automatic event triggering and convergence*: byte-for-byte reproducible across runs and machines; no wall-clock, PID, or dict-iteration-order leakage. The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is picked in step 5 against that requirement, with the following acceptance: -- At least one deterministic policy ships and passes a **"run 100×, all final `State`s identical"** soak test. +- At least one deterministic policy ships and passes a **"run 100×, all final `State`s identical"** soak test. - The API exposes a **user-override hook** for cases the default policy doesn't fit. The hook may ship empty initially. Picked here rather than locked up front because the choice benefits from being able to play with the convergence loop in working code. -## Inline charm classes vs on-disk charms (resolved 2026-06-03) +## **Inline charm classes vs. on-disk charms** `from_type` charms defined inline in a test cannot be trivially transferred to a fresh subprocess (the class object is not serialisable). Isolation realistically requires charms on disk. The API keeps both paths: inline classes run in-process with no isolation (like Scenario today); on-disk charms run isolated by default. -The matrix of "works inline but not isolated" surprises is closed by a **documentation convention** rather than a runtime check. Every operation introduced from step 4 onward — anything beyond `IsolatedContext.run()` (`deploy`, `add-unit`, `integrate`, `update-config`, `run_action`, …) — declares in its docstring two explicit fields: +The matrix of ‘works inline but not isolated’ surprises is closed by a **documentation convention** rather than a runtime check. Every operation beyond `IsolatedContext.run()` (`deploy`, `add-unit`, `integrate`, `update-config`, `run_action`, …) declares in its docstring two explicit fields: -- `inline: ok | error | n/a` -- `isolated: ok | error | n/a` +- inline: ok | error | n/a +- isolated: ok | error | n/a Where `ok` means works as documented, `error` means raises a clear typed error explaining why it's not supported in this mode, and `n/a` means doesn't make sense in this mode. -Enforced by API-doc review at PR time. Trivial to grep for. The matrix becomes readable in the docs rather than discoverable only by trial. No separate validation layer. +This is enforced by API-doc review at PR time. It’s trivial to grep for. The matrix becomes readable in the docs rather than discoverable only by trial. There is no separate validation layer. -## Real-world charm coverage +### **Real-world charm coverage** -The isolation and provisioning spikes were demonstrated end-to-end on synthetic test charms with intentionally conflicting deps, and the provisioning detection/extraction path was validated against two real Canonical charms (loki-k8s-operator, postgresql-k8s-operator) via a read-only `describe` CLI. Full end-to-end "provision venv, run the charm's code" against a real production charm remains unproven at spike scope. Likely first regression class: Python-version mismatch (charms targeting `requires-python = "~=3.12"` in a 3.11 harness). +The isolation and provisioning proof-of-concept implementations were validated end-to-end on synthetic test charms with intentionally conflicting deps, and the provisioning detection/extraction path was validated against two real charms (loki-k8s-operator, postgresql-k8s-operator) via a read-only `describe` CLI. Full end-to-end "provision venv, run the charm's code" against a real production charm remains unproven at spec writing time. Likely first regression class: Python-version mismatch (charms targeting `requires-python = "~=3.12"` in a 3.11 harness). -## Out of scope +### **Out of scope** -- Changes to Scenario's existing single-charm API or behaviour -- charmcraft `reactive` plugin support (not in current use; defer) +- Changes to Scenario's existing single-charm API or behaviour +- charmcraft `reactive` plugin support -## Prior art and references +### **Prior art and references** -- [Catan](https://github.com/PietroPasotti/catan) — the abstraction this spec draws from; spike branch `explore/per-charm-dependency-isolation` at commit `1f4c357` -- [operator#1929](https://github.com/canonical/operator/issues/1929) — origin of the Saddle name and the `ops.testing`-integrated direction +- [Catan](https://github.com/PietroPasotti/catan) — the abstraction this spec draws from; spike branch `explore/per-charm-dependency-isolation` at commit `1f4c357` +- [operator\#1929](https://github.com/canonical/operator/issues/1929) — origin of the `ops.testing`\-integrated direction # Spec History and Changelog +Please be thorough when recording changes and progress with the spec itself and the work resulting from it. Record every meeting, attendees and conclusions from the meeting. + | Author(s) | Status | Date | Comment | | :---- | :---- | :---- | :---- | -| [Tony Meyer](mailto:tony.meyer@canonical.com) | Draft | 26 May 2026 | Initial draft; seeded from the isolation spike, the environment-provisioning spike, and the decision to target `ops.testing` rather than a companion package | -| [Tony Meyer](mailto:tony.meyer@canonical.com) | Draft | 3 June 2026 | Resolved six open questions: StoredState fidelity (always-on typed escape hatch), JSON `State` schema stability (explicit `state_schema_version` field + dispatch table), settle-policy determinism requirement (byte-for-byte reproducible), inline-vs-on-disk docstring convention, perf 2× acceptance bar (4-charm/20-event workload), metadata-resolution six-shape step-6 acceptance set. Small follow-up survey of in-tree charm `StoredState` usage is queued separately. | -| Person | Pending Review | Date | | +| [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 8, 2026 | Initial draft | +| [Ben Hoyt](mailto:ben.hoyt@canonical.com) | Drafting | Jun 15, 2026 | Initial review, comments | +| [James Garner](mailto:james.garner@canonical.com) | Drafting | Jun 15, 2026 | Initial review, comments | +| [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 24, 2026 | Addressed comments. Added serialisation example section. | +| Person | Braindump | Date | | +| | | | | + From 3f0af27ec6f1598c8a9396c5b2b91479a86cf781 Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 24 Jul 2026 16:51:57 +1200 Subject: [PATCH 3/4] docs: rename spec file to reflect OP089 index and current title --- ...9-multiple-charm-state-transition-tests.md | 697 ++++++++++++++++++ 1 file changed, 697 insertions(+) create mode 100644 specs/OP089-multiple-charm-state-transition-tests.md diff --git a/specs/OP089-multiple-charm-state-transition-tests.md b/specs/OP089-multiple-charm-state-transition-tests.md new file mode 100644 index 0000000..daa1cd9 --- /dev/null +++ b/specs/OP089-multiple-charm-state-transition-tests.md @@ -0,0 +1,697 @@ +| Index | OP089 | | | +| :---- | :---- | :---- | :---- | +| Title | Multiple-charm state transition tests | | | +| **[Type](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Author(s)** | **[Status](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Created** | +| Implementation | [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 8, 2026 | +| | **Reviewer(s)** | **Status** | **Date** | +| | Person | Pending Review | Date | + +# Abstract + +This spec defines a multi-charm unit (state-transition) test surface folded into `ops.testing`. It provides `deploy`, `integrate`, `add-unit`, `remove-unit`, and `update-config` as test operations, with the corresponding Juju events triggered automatically as the model converges. Each charm runs in its own isolated subprocess (its own venv), so charms with mutually incompatible dependency sets can be tested together in a single test. This spec targets `ops.testing` in `canonical/operator`; it is not a companion package. It draws the abstraction from [Catan](https://github.com/PietroPasotti/catan) (Apps \+ convergence) but is not a direct port. + +# Rationale + +### **The gap in current tooling** + +`ops.testing` (Scenario) runs one charm at a time: the developer writes an event and a `State`, calls `Context.run`, and asserts on the output. This is efficient for single-charm logic but leaves four pain points unaddressed: + +- **Integration tests are slow.** A real Juju controller bootstrap and deploy round-trip takes minutes; unit tests should be measured in seconds. +- **Scenario is unlike the `juju` commands charmers think in.** Charmers reason about `juju deploy`, `juju integrate`, `juju add-unit`. Translating that mental model into hand-crafted event sequences and `State`s is friction the tool should absorb. This has been evident in the success of Jubilant (which itself further strengthens the issue, because charmers are now also used to thinking that way in integration tests too). +- **Charm libraries don't yet ship mocks.** Cross-charm relation data is hand-crafted per test. Running the other charm's own code against the relation removes that hand-crafting: the other charm's actual reaction becomes the fixture. This may change with the acceptance of [OP077](https://docs.google.com/document/d/1A50MlUpNz6YPVvIFxFZYTDBgxNboVYlpfZB8uy8o58c/edit?usp=sharing) (Charm library testing). +- **Bundle (or solution) and topology assertions are unreachable from unit tests.** Asserting "if I deploy A and B and integrate them, A's leader ends up with config X" requires real Juju or fragile mocks today. + +The shape charmers ask Charm Tech for is ‘a multi-charm state-transition test tool that feels like Juju’. This is Catan’s abstraction (Apps, `integrate`, convergence / settle). + +Why ops.testing rather than a companion package? Catan today pins `ops-scenario~=6.0`, a superseded API that has been merged into `ops.testing` as Scenario 7 (Scenario 8 is identical to 7, other than requiring Python 3.10+). Porting off that API is **net-new work regardless of the destination**; there is no path that productises the companion-package form without a full re-targeting. Since we have to re-target anyway, we target `ops.testing` directly. This is a non-trivial addition to ops-scenario, but not so large that it merits a new ops-{something} package, with the packaging overhead that comes from that. + +# Specification + +### **Location and scope** + +The new surface lives in `ops.testing` (`canonical/operator`), the same module that provides `Context` and `State`. The single-charm primitives are **unchanged and remain the substrate**. The new surface adds a model-level layer above `Context` that owns a set of `App`s, an integration graph, and a convergence loop. Single-charm Scenario (state transition) tests continue to work exactly as today and do not require the new surface. + +### **Operation surface** + +The test API uses Juju-shaped operations (`deploy`, `integrate`, `add-unit`, `remove-unit`, `update-config`, `run_action`) that describe **intents, not events**. The model generates the correct Juju-faithful event sequence and delivers it to the affected units. + +# + +```py +from ops import testing + +model = testing.Model() # Already exists, but *not* the same as ops.Model! + +db = model.deploy( + charm="./charms/postgresql", + app="postgresql", + config={"profile": "testing"}, + num_units=1, +) +web = model.deploy( + charm="./charms/myapp", + app="myapp", + num_units=2, +) + +model.integrate(db, web) +# Or, when the endpoints need to be disambiguated: +# model.integrate(db.get_endpoint("db"), web.get_endpoint("pg")) +model.add_unit(web) +model.update_config(web, {"log_level": "debug"}) +model.remove_unit(web, unit=1) +model.run_action(db.leader, "create-backup", params={}) +``` + +# + +`Model.deploy(...)` returns an `App` handle. An `App` owns its charm reference, the metadata resolved from the charm on disk, the environment it runs in (see Isolation model below), and its per-unit `State`s. Exact type names are illustrative and will be settled in code review. + +`Model.integrate(...)` follows Jubilant's shape but takes `App` objects rather than strings — we have the objects here, and there is no live Juju model to look them up in. The default form `integrate(db, web)` works whenever the endpoint pair is unambiguous from the two charms' metadata. When it isn't, endpoints are named explicitly via `App.get_endpoint(name)`: `integrate(db.get_endpoint("db"), web.get_endpoint("pg"))`. This mirrors Jubilant's `integrate("postgresql:db", "myapp:pg")` while keeping the API object-typed. + +### **Charm source: local paths only in v1** + +`charm=` in v1 is **a path to charm source on the local filesystem**. Type is `pathlib.Path | str`; a `str` must be a relative path, matching `juju deploy` behaviour. This covers test charms and monorepos out of the box. + +**Not in v1:** + +- **Deploy from git** (with or without patches, as Catan supports) is out of scope for v1 and possibly ever — the complexity is high and the value versus a conftest-managed clone is low. +- **Deploy from Charmhub** — feasible (download → unzip → treat the unpacked source as a local path), but not v1. If added later, the same `charm=` argument would grow a Charmhub-URL form; no prototype yet. + +**Recommendation for charms defined outside your repository:** the test infra (typically a `conftest.py` fixture) should source the charm from somewhere — `git clone` is the obvious choice — and pass the resulting local path to `deploy(charm=...)`. This keeps the harness surface small and puts the "where does this charm come from" decision in the test author's hands, where it can be tailored to CI, caching, and gitignore rules per project. + +### **Automatic event triggering and convergence** + +Each operation enqueues the appropriate events (`relation-created`, `relation-joined`, `relation-changed`, `config-changed`, and so on) on the affected units. A **convergence loop** drains the queue: it picks an event, dispatches it through that unit's `Context.run(...)`, observes any new relation-data writes, status changes, or secret operations, and enqueues the downstream events those produce on peers and related units. Convergence terminates when the queue is empty and no unit's last run produced new observable changes. + +```py +model.settle() # explicit convergence point +assert web.leader.state.unit_status == ActiveStatus("ready") +assert db.leader.state.get_relation("db").local_app_data["database"] == "myapp" +``` + +Catan shows that this can be done with suitable fidelity to Juju without *actually reimplementing Juju*. We are able to take advantage of the non-specific ordering of some Juju events, and also of the access to the Juju code to understand its behaviour. + +`settle()` is also called implicitly before reading any unit’s `State` if there are pending events, so most tests are a sequence of operations followed by assertions on `app.leader.state` or `app.units[i].state`. The `State` exposed on each unit is the same `ops.testing.State` dataclass that single-charm Scenario tests use today; all existing assertions are available unchanged. + +For tests that need to assert on **intermediate** state — partway through convergence, or on the exact sequence of events dispatched — two complementary APIs are exposed: + +- **Step-at-a-time context** — mirrors Scenario's `with ctx(...) as mgr: mgr.run(...)` shape. Inside the block, the implicit-settle-on-read is suspended so assertions between steps do not drain the queue: + + ```py + with model.stepping() as stepper: + stepper.step() # dispatch exactly one queued event + assert web.leader.state.unit_status == MaintenanceStatus("configuring") + stepper.step() + assert ... + # outside the block, implicit-settle-on-read is back + ``` + + `stepper.step()` returns the `(event, unit)` it just dispatched so the loop can be predicate-driven (`while stepper.step() != ("relation-changed", web.leader): ...`) without a separate peek API. Exiting the block re-enables implicit settling. + +- **Recorded event trace** — `settle()` returns the deterministic sequence of `(event, unit, state_snapshot)` tuples it dispatched, so post-hoc assertions can walk what happened without needing to steer the loop: + + ```py + trace = model.settle() + assert [e.name for e, _, _ in trace] == ["relation-created", "relation-joined", "relation-changed", "config-changed"] + ``` + + No behaviour change to the default settle path — the return value is additive. + +Mid-flight assertions with `stepping()`, post-hoc assertions with the returned trace; neither leaks into the default settle path a simple test uses. + +Given the same starting `Model` and the same input event sequence, `settle()` **must** produce the same final `State` per unit and the same sequence of intermediate events, byte-for-byte, across runs and across machines. **No wall-clock dependence, no PID-derived ordering, no Python-dict-iteration-order leakage** is permitted in the policy. The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is decided against this requirement, when there is working code to validate it. See [Further Information](#settle-policy-considerations). + +Real Juju runs charms concurrently and per-unit event ordering is non-deterministic. Unit tests must be deterministic, so v1 ships a single ordering policy — but the design keeps the door open for exercising alternative orderings against the *same* starting sequence of events. Two mechanisms are anticipated (concrete API picked in step 5 alongside the default policy): + +- **Alternative strategies via `settle(...)` arguments** — e.g. `model.settle(strategy="data-flow")` vs `strategy="round-robin"` vs a caller-supplied deterministic function. Running the same test under two strategies is one way to smoke out charms that accidentally depend on one particular ordering. +- **Per-app / per-unit scheduling hints** — e.g. mark an `App` as "running late" so its events are held until the queue is otherwise empty, or bias one unit to drain first. This lets charmers reproduce specific ordering bugs seen in production. + +Both are optional layers on top of the default deterministic policy; the default remains what `settle()` with no arguments does. + +### **Isolation model: subprocess \+ per-charm venv** + +Each `App` runs in its own process with its own interpreter (an existing venv's `bin/python`, or one the harness provisioned; see Environment provisioning below). For each event dispatch the harness serialises `(meta, config, actions, event, State)` as JSON, sends it to the charm's worker process, the worker runs `testing.Context(...).run(...)`, and serialises the resulting `State` back. Two `App`s in the same `Model` may use mutually incompatible dependency sets (`cryptography==41` vs `==3`, different `pydantic` majors, conflicting charm-lib versions, …). + +**Required invariant:** each per-charm venv contains an `ops` / `ops-scenario` build whose JSON `State` wire format is compatible with the parent process's. Only the charm's own runtime deps differ between venvs. This is realistic, since all charms (that matter here) build on `ops`. + +**Cross-`ops`-version compatibility policy.** Running each charm in its own venv means charms may sit on different `ops` versions than the parent process — a real ongoing cost. The policy has three parts: + +- **Version negotiation at worker handshake.** On startup, the worker declares the wire-format version it supports. The parent process picks the intersection or raises `StateSchemaVersionError` (already introduced by the typed-`State` serialiser, step 2 of the incremental delivery plan). Mismatches fail fast at handshake, not silently mid-test. +- **Directionality: unidirectional, parent-authoritative.** The parent process (running `ops.testing`) is the authority; it supports schema versions `1..N` where `N` is its own current version. A charm worker on any version in `1..N` is fine — the parent decodes at the worker's declared version. A worker on `>N` (a newer `ops` than the parent) raises at handshake, because the parent cannot invent decoder logic for a future schema. **Practical invariant: the orchestrating test suite's `ops` must be ≥ every charm-under-test's `ops`.** This matches the natural test-suite topology (bump `ops.testing` first, then adopt in charms) and keeps the compat matrix strictly triangular rather than a full N×N grid. +- **Documented compatibility window.** Wire-format changes are treated as `ops` release-note items with an explicit "supported charm-side `ops` versions" range published against each `ops.testing` release — this is the *floor* (how far back a charm can be); the *ceiling* is the parent's own version (directionality, above). Charms outside the window raise at handshake with a message pointing at the range. Bounded surface, small compat matrix, testable. +- **Escape hatch: pin the charm's `ops` to the parent process's.** When `build_env=True`, the provisioner can override the charm's declared `ops` pin with the parent process's exact version via `deploy(..., pin_ops=True)`. Cuts through the compat matrix at the cost of not exercising the charm on its own declared `ops`. Off by default; opt-in per `deploy(...)`. + +The isolation worker is persistent: one long-lived process per `App`, spawned lazily on first dispatch and torn down at `Model` teardown or after an idle timeout. A framed JSON protocol over stdin/stdout carries requests and responses. Spawn-per-event remains available as an explicit debug mode (easier to attach a debugger; no shared interpreter state between events). Worker crashes propagate as a typed `IsolationError` and fail the test cleanly; the harness never silently re-spawns mid-test. + +Acceptance criteria: + +- **Workload:** the performance yardstick is a **4-charm bundle with 20 events per `settle()`** (80 dispatches total). Lives in the test suite and is re-run by every later step to catch regressions. +- **Bar:** **persistent-worker mode within 2× the in-process Scenario baseline** on that workload. 2× is the threshold where "slow but acceptable for a CI unit-test suite" tips into "too slow, charmers will avoid it and just use integration tests"; subprocess \+ JSON serialisation has a real floor below that. +- **Benchmark shape:** a benchmark suite measuring (a) in-process Scenario baseline, (b) spawn-per-event isolated, and (c) persistent-worker isolated, all on the same 4-charm/20-event workload. +- **Measured absolute numbers (step 3 draft implementation, Python 3.11.15, median of 5 reps):** + + | mode | total (80 dispatches) | per event | vs baseline | + | :---- | :---- | :---- | :---- | + | (a) in-process Scenario baseline | 0.332 s | 4.15 ms | 1.00× | + | (c) persistent-worker isolated | 1.671 s | 20.89 ms | **5.04×** | + | (b) spawn-per-event isolated | 22.94 s | 286.77 ms | 69× | + + Persistent-worker mode does **not** currently meet the 2× bar (see step 3 log in [`step3-persistent-worker-log.md`](./step3-persistent-worker-log.md)). Dominant cost is `import ops` per worker (~243 ms), paid once per isolated environment and structurally unshareable across environments; JSON/IPC is sub-millisecond per event. On this 4-worker workload the ~1.1 s of worker spawns alone already exceeds the 2× budget (~0.66 s). Persistent is nonetheless ~13.7× faster than spawn-per-event and is the correct foundation for step 5's convergence loop, where more events per worker amortise the spawn cost further. **The 2× bar itself is up for revision** in light of the numbers — the working assumption is that the threshold is comfortably higher than 2× in practice, and step 5 will re-measure with real bundle-shaped workloads before locking it in. + +### **Subinterpreters are explicitly ruled out** + +PEP 734 subinterpreters are **not** the mechanism, even assuming a Python 3.14+ requirement would be acceptable: + +- **Binary-dependency conflicts remain unsolved.** Many C extensions (`cryptography`, `pydantic-core`, `cffi`\-based packages) are not multi-interpreter-safe; two different versions of the same extension cannot coexist in one process at any Python version. These are the exact libraries motivating the work. +- **State boundary still costs a serialisation roundtrip**. The same JSON encode/decode as a subprocess, taken without the isolation benefit. +- **Maintenance cost is high** on a bleeding-edge API for no payoff over the subprocess design. + +This will be revisited only if the C-extension multi-interpreter story matures enough to load conflicting binary versions in one process, not merely because the Python version floor moves. + +## Environment provisioning + +The environment API is deliberately layered so the core primitive has a small surface and no external-tool dependencies, and the ergonomic auto-detection form is a convenience helper on top rather than the load-bearing default. + +**Layer 1 — inherit the test process** (default; no `python=` argument). No isolation, no venv work, no external tools. Runs the charm in the same interpreter as the test. Unblocks Scenario-style tests with locally-defined test charms and monorepo cases where dependencies are known compatible. + +**Layer 2 — explicit interpreter + resolved requirements** (`Model.deploy(..., python="/path/to/venv/bin/python", requirements=[...])`). The core primitive: a Python interpreter path plus a fully-resolved list of pip specifiers. Zero magic; no charmcraft-plugin knowledge; no dependency on `uv` or `poetry`. The caller has already done whatever resolution they want to do (their own build script, their own lockfile export, whatever). This form unlocks the conflicting-dependency case without pulling any external-tool complexity into the core. + + Passing `python=` without `requirements=` means "use this interpreter as-is" — the caller has already provisioned the venv. Passing `requirements=` without `python=` is an error. + +**Layer 3 — auto-provision from declared deps** (opt-in `Model.deploy(..., build_env=True)`). A convenience helper that detects the charm's build plugin, extracts the dependency list, and calls into layer 2 with the extracted values. Charmers who want ergonomics use this; charmers whose plugin detection fails, or who want control, drop to layer 2 without losing anything. + + Plugin detection is `charmcraft.yaml` first; file-based heuristic second (`uv.lock` → `uv`; `poetry.lock` → `poetry`; `pyproject.toml` → `python`; else → `charm`). Extraction per plugin: + + | Plugin | Extraction method | + | :---- | :---- | + | charm | reads `charm-requirements` files \+ `charm-python-packages` \+ AST-harvested `PYDEPS` from `lib/**/*.py` | + | python | reads `python-requirements` / `python-packages` \+ PYDEPS (included even though charmcraft's python plugin does not auto-install them: the isolation worker needs the lib importable) | + | uv | shells to `uv export --frozen --no-emit-project --no-hashes --format requirements-txt` with forwarded `--extra` / `--group` flags | + | poetry | shells to `poetry export --without-hashes --format requirements.txt` with forwarded `--with` flags | + + All forms normalise to a flat pip specifier list, then install via `uv pip install` into the per-charm venv. The venv is cached on disk, keyed by SHA-256 of (plugin, extracted requirements, lockfile/pyproject content, interpreter version, parent-process `ops` version): cold build \~3s for a test charm; warm cache hit \<0.05s (measured). Detection-and-extraction was validated against two real charms (loki-k8s-operator via the uv plugin; postgresql-k8s-operator via the poetry plugin). Layer 3 emits a clear, typed error when an extracted requirement cannot be resolved, when a lockfile is stale relative to the pyproject, or when a required external tool (`uv` / `poetry`) is missing — the failure mode is "drop to layer 2", not "fall through silently". + +**Tool dependency (layer 3 only):** `uv` on PATH for all install steps; `poetry` \+ `poetry-plugin-export` for the poetry-plugin path. Layers 1 and 2 depend on nothing beyond the Python standard library. This scopes the external-tool concern to the convenience layer where charmers opted in to the ergonomics. + +**Future:** a charmcraft-managed flow for charms already packed locally, sitting at layer 3 alongside the plugin-based extraction. + +## State (de)serialisation + +A typed schema-aware encoder/decoder pair (not `json.dumps(dataclasses.asdict(state))`) covers all `ops.testing.State` leaf types: + +| Type | Encoding | +| :---- | :---- | +| `frozenset` (on `relations`, `containers`, `secrets`, etc.) | list on encode; re-frozen on decode (dataclass `__post_init__`s re-freeze) | +| `datetime` / `timedelta` | ISO-8601 string / fractional seconds | +| `pathlib.Path` / `PurePosixPath` | string with type tag | +| Pebble enums (`ServiceStatus`, `NoticeType`, `SecretRotate`, …) | enum name | +| `pebble.Layer` (`Container.layers`) | `to_dict()` / `Layer(dict)` round-trip (not a dataclass; `asdict` won't descend) | +| Int-keyed dicts (`Secret.remote_grants: Mapping[int, set[str]]`) | keys stringified on encode; restored to `int`, inner `set` restored on decode | + +`StoredState.content` and `Container._base_plan` are `Mapping[str, Any]` of marshallable types, a superset of JSON: `bytes`, `tuple`\-vs-`list` distinction, and arbitrary `set`s. The spec commits to: + +1. **A typed escape hatch, applied always**: tag-and-encode `bytes` as base64 and `tuple` as a `["__tuple__", ...]` envelope so they round-trip losslessly. **No flag.** An opt-in would create a "passed locally because today's stored-state had no `bytes`, failed in CI when it did" failure mode; the cost (a few extra bytes per typed value) doesn't justify it. +2. **Loud error on unrecognised types**: the encoder raises `TypeError` with the type name and the path through `State`. The decoder mirrors: unknown `__type__` tags raise `TypeError`. Never silent lossy coercion. + +A small follow-up survey of in-tree charm `StoredState` usage is left as a separate work item, to confirm no charm relies on round-tripping more exotic marshallable types than `bytes`/`tuple`/`set`. The survey does not gate the implementation; it informs whether the encoder's type set needs to be extended. + +The JSON `State` representation is **part of `ops.testing`'s public API**: a per-charm venv may have a different `ops` build from the parent, and both sides must agree on the encoded form. + +1. **Explicit `state_schema_version` integer field** is embedded in every encoded `State` payload. Cheaper and more legible than overloading `ops`'s SemVer: we don't have to ship an `ops` major just because we added one field to `State` serialisation. +2. **Per-version interpretation is locked at the source-code level**: a dispatch table keyed by version. Adding a new version is ordinary work; changing the *meaning* of an existing version is not. +3. **Decoder rejects unsupported versions with a clear typed error.** A v1 producer and a v1+v2 consumer work fine. A version the decoder doesn't recognise surfaces as an explicit error, not silent corruption. +4. **`ops.testing` API-stability documentation commits to (2).** "We may add new versions; we don't change the meaning of an existing version." + +### **Serialisation examples** + +#### Container with a Pebble layer, notices, check infos, service statuses, and mounts + +This example shows the most type-envelope diversity in one place: `pebble.Layer` → **t**: "layer", `pebble.ServiceStatus` enum → **t**: "enum", `pebble.NoticeType` enum, `datetime` fields, `timedelta` fields, `PurePosixPath` and `Path` in a `Mount`, and `CheckInfo` containing multiple enum types. + +```py + state = State( + containers=frozenset([ + Container( + name='myapp', + can_connect=True, + layers={ + 'base': pebble.Layer({ + 'services': { + 'app': { + 'command': '/bin/app --port 8080', + 'startup': 'enabled', + 'override': 'replace', + }, + }, + }), + }, + service_statuses={'app': pebble.ServiceStatus.ACTIVE}, + notices=[ + Notice( + key='example.com/event', + type=pebble.NoticeType.CUSTOM, + repeat_after=datetime.timedelta(minutes=30), + first_occurred=datetime.datetime( + 2025, 6, 15, 10, 0, 0, tzinfo=datetime.timezone.utc + ), + last_occurred=datetime.datetime( + 2025, 6, 15, 10, 30, 0, tzinfo=datetime.timezone.utc + ), + last_repeated=datetime.datetime( + 2025, 6, 15, 10, 30, 0, tzinfo=datetime.timezone.utc + ), + ), + ], + check_infos=frozenset([ + CheckInfo( + name='http', + level=pebble.CheckLevel.ALIVE, + status=pebble.CheckStatus.UP, + ), + ]), + mounts={ + 'cfg': Mount( + location=pathlib.PurePosixPath('/etc/config'), + source=pathlib.Path('/tmp/cfg'), + ), + }, + ), + ]), + ) +``` + +```json + { + "state_schema_version": 1, + "state": { + "__t__": "dc", + "cls": "State", + "f": { + "config": {}, + "relations": {"__t__": "frozenset", "v": []}, + "networks": {"__t__": "frozenset", "v": []}, + "containers": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Container", + "f": { + "name": "myapp", + "can_connect": true, + "_base_plan": {}, + "layers": { + "base": { + "__t__": "layer", + "v": { + "services": { + "app": { + "startup": "enabled", + "override": "replace", + "command": "/bin/app --port 8080" + } + } + } + } + }, + "service_statuses": { + "app": {"__t__": "enum", "cls": "ServiceStatus", "name": +"ACTIVE"} + }, + "mounts": { + "cfg": { + "__t__": "dc", + "cls": "Mount", + "f": { + "location": {"__t__": "PurePosixPath", "v": +"/etc/config"}, + "source": {"__t__": "Path", "v": "/tmp/cfg"} + } + } + }, + "execs": {"__t__": "frozenset", "v": []}, + "notices": [ + { + "__t__": "dc", + "cls": "Notice", + "f": { + "key": "example.com/event", + "id": "1", + "user_id": null, + "type": {"__t__": "enum", "cls": "NoticeType", "name": +"CUSTOM"}, + "first_occurred": {"__t__": "datetime", "v": +"2025-06-15T10:00:00+00:00"}, + "last_occurred": {"__t__": "datetime", "v": +"2025-06-15T10:30:00+00:00"}, + "last_repeated": {"__t__": "datetime", "v": +"2025-06-15T10:30:00+00:00"}, + "occurrences": 1, + "last_data": {}, + "repeat_after": {"__t__": "timedelta", "v": 1800.0}, + "expire_after": null + } + } + ], + "check_infos": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "CheckInfo", + "f": { + "name": "http", + "level": {"__t__": "enum", "cls": "CheckLevel", "name": +"ALIVE"}, + "startup": {"__t__": "enum", "cls": "CheckStartup", +"name": "ENABLED"}, + "status": {"__t__": "enum", "cls": "CheckStatus", "name": +"UP"}, + "successes": 0, + "failures": 0, + "threshold": 3, + "change_id": "1" + } + } + ]} + } + } + ]}, + "storages": {"__t__": "frozenset", "v": []}, + "opened_ports": {"__t__": "frozenset", "v": []}, + "leader": false, + "model": {"__t__": "dc", "cls": "Model", "f": {"name": "...", "uuid": +"...", "type": "kubernetes", "cloud_spec": null}}, + "secrets": {"__t__": "frozenset", "v": []}, + "resources": {"__t__": "frozenset", "v": []}, + "planned_units": 1, + "deferred": [], + "stored_states": {"__t__": "frozenset", "v": []}, + "app_status": {"__t__": "status", "name": "unknown", "msg": ""}, + "unit_status": {"__t__": "status", "name": "unknown", "msg": ""}, + "workload_version": "" + } + } + } +``` + +#### Secrets with `int`\-keyed `remote_grants`, rotation policy, and Relations with `int`\-keyed `remote_units_data` + +This example shows the "idict" envelope (for `dict[int, ...]` which JSON can’t represent natively), the "set" envelope inside an `idict`, and `SecretRotate` enum. + +```py + state = State( + secrets=frozenset([ + Secret( + tracked_content={'password': 'x2', 'cert': '---BEGIN CERT---'}, + latest_content={'password': 'x3', 'cert': '---BEGIN CERT---'}, + owner='app', + label='db-creds', + description='Database credentials', + expire=datetime.datetime( + 2030, 12, 31, 23, 59, 59, tzinfo=datetime.timezone.utc + ), + rotate=SecretRotate.DAILY, + remote_grants={0: {'related-app'}, 2: {'other-app/0'}}, + ), + ]), + relations=frozenset([ + Relation( + endpoint='db', + interface='postgresql_client', + remote_app_name='postgresql', + remote_units_data={ + 0: {'host': '10.0.0.1', 'port': '5432'}, + 1: {'host': '10.0.0.2', 'port': '5432'}, + }, + remote_app_data={'version': '14', 'db': 'mydb'}, + ), + PeerRelation( + endpoint='peers', + peers_data={1: {'leader': 'true', 'ready': 'yes'}}, + ), + ]), + ) +``` + +```json + { + "relations": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Relation", + "f": { + "endpoint": "db", + "interface": "postgresql_client", + "id": 1, + "local_app_data": {}, + "local_unit_data": { + "egress-subnets": "192.0.2.0", + "ingress-address": "192.0.2.0", + "private-address": "192.0.2.0" + }, + "remote_app_name": "postgresql", + "limit": 1, + "remote_app_data": {"version": "14", "db": "mydb"}, + "remote_units_data": { + "__t__": "idict", + "v": { + "0": {"host": "10.0.0.1", "port": "5432"}, + "1": {"host": "10.0.0.2", "port": "5432"} + } + }, + "remote_model_uuid": null + } + }, + { + "__t__": "dc", + "cls": "PeerRelation", + "f": { + "endpoint": "peers", + "interface": null, + "id": 2, + "local_app_data": {}, + "local_unit_data": {"egress-subnets": "192.0.2.0", +"ingress-address": "192.0.2.0", "private-address": "192.0.2.0"}, + "peers_data": { + "__t__": "idict", + "v": {"1": {"leader": "true", "ready": "yes"}} + } + } + } + ]}, + "secrets": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Secret", + "f": { + "tracked_content": {"password": "hunter2", "cert": "---BEGIN +CERT---"}, + "latest_content": {"password": "hunter3", "cert": "---BEGIN +CERT---"}, + "id": "secret:oup6t3bz6wzkkjylya5u", + "owner": "app", + "remote_grants": { + "__t__": "idict", + "v": { + "0": {"__t__": "set", "v": ["related-app"]}, + "2": {"__t__": "set", "v": ["other-app/0"]} + } + }, + "label": "db-creds", + "description": "Database credentials", + "expire": {"__t__": "datetime", "v": "2030-12-31T23:59:59+00:00"}, + "rotate": {"__t__": "enum", "cls": "SecretRotate", "name": "DAILY"}, + "_tracked_revision": 1, + "_latest_revision": 1 + } + } + ]} + } +``` + +#### `StoredState` escape hatch (`bytes`, `tuple`, `set`) \+ statuses, resources, ports, and deferred events + +This example shows the escape-hatch types that `StoredState.content` can contain: `bytes` (base64-encoded), `tuple` (`list`\-based envelope), and `set`. It also shows the "status" envelope for non-trivial status values, "Path" in a Resource, and `DeferredEvent`. + +```py + state = State( + config={'log_level': 'debug', 'port': 8080, 'enabled': True}, + leader=True, + unit_status=ActiveStatus('ready'), + app_status=BlockedStatus('needs database'), + workload_version='1.2.3', + planned_units=3, + stored_states=frozenset([ + StoredState( + name='_stored', + owner_path='MyCharm', + content={ + 'count': 42, + 'raw': b'\xde\xad\xbe\xef', + 'coords': (1, 2, 3), + 'tags': {'alpha', 'beta'}, + 'mapping': {'nested': True}, + }, + ), + ]), + resources=frozenset([ + Resource(name='oci-image', path=pathlib.Path('/tmp/image.tar')), + ]), + opened_ports=frozenset([TCPPort(8080), UDPPort(53)]), + deferred=[ + DeferredEvent( + handle_path='MyCharm/on/config_changed[1]', + owner='MyCharm', + observer='_on_config_changed', + ), + ], + ) +``` + +```json + { + "config": {"log_level": "debug", "port": 8080, "enabled": true}, + "opened_ports": {"__t__": "frozenset", "v": [ + {"__t__": "dc", "cls": "UDPPort", "f": {"port": 53, "protocol": "udp"}}, + {"__t__": "dc", "cls": "TCPPort", "f": {"port": 8080, "protocol": +"tcp"}} + ]}, + "leader": true, + "resources": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "Resource", + "f": { + "name": "oci-image", + "path": {"__t__": "Path", "v": "/tmp/image.tar"} + } + } + ]}, + "planned_units": 3, + "deferred": [ + { + "__t__": "dc", + "cls": "DeferredEvent", + "f": { + "handle_path": "MyCharm/on/config_changed[1]", + "owner": "MyCharm", + "observer": "_on_config_changed", + "snapshot_data": {} + } + } + ], + "stored_states": {"__t__": "frozenset", "v": [ + { + "__t__": "dc", + "cls": "StoredState", + "f": { + "name": "_stored", + "owner_path": "MyCharm", + "content": { + "count": 42, + "raw": {"__t__": "bytes", "v": "3q2+7w=="}, + "coords": ["__tuple__", 1, 2, 3], + "tags": {"__t__": "set", "v": ["alpha", "beta"]}, + "mapping": {"nested": true} + }, + "_data_type_name": "StoredStateData" + } + } + ]}, + "app_status": {"__t__": "status", "name": "blocked", "msg": "needs +database"}, + "unit_status": {"__t__": "status", "name": "active", "msg": "ready"}, + "workload_version": "1.2.3" + } +``` + +## Metadata resolution + +The harness reads `metadata.yaml` / `charmcraft.yaml` from the charm source path without importing the charm in the test process. Less common cases are left as future work: + +- Resolve unified-vs-legacy metadata variants and the merged `charmcraft.yaml` \+ `metadata.yaml` form charms commonly produce +- Resolve charm libraries the charm declares but that live elsewhere on disk +- Inline charm classes (`from_type` charms defined in the test): kept as in-process, same-environment only; isolation requires charms on disk (`from_isolated_path`). See Further Information. + +At least one passing test for each of these charm shapes must be included in the implementation: + +1. Unified `charmcraft.yaml` only (no `metadata.yaml`) — the current default-modern shape. +2. Legacy split: `metadata.yaml` \+ `actions.yaml` \+ `config.yaml` — older charms still in production. +3. Mixed / transitional: `charmcraft.yaml` \+ `metadata.yaml` (the form charmcraft merges) — charms mid-migration. +4. Multi-bases (`bases:` / `platforms:` variants per supported platform). +5. Charm with libs outside `src/` (e.g. installed via `pip` from PyPI for testing) — exercises the "where did this lib come from" resolution. +6. Inline `from_type` (no metadata file; class declares everything) — documented as `inline: ok` per the docstring convention in Further Information. + +## Migration from Catan / ops-scenario\~=6.0 + +The existing `PietroPasotti/catan` package pins `ops-scenario~=6.0` (superseded by `ops.testing`). There is **no compatibility shim**: tests written against Catan must be rewritten against the new API surface. The `ops-scenario~=6.0` form is throwaway regardless of whether this spec is adopted. Porting off it is net-new work either way. + +What the spec carries forward from Catan: + +- The model: Apps \+ integration graph \+ convergence/settle +- The Juju-like operation vocabulary: `deploy` / `integrate` / `add-unit` / … + +## Incremental delivery plan + +Sequenced so each merge is **independently useful** and the area can pause cleanly after any step. This is the handle on the priority reservation: agentic coding lowers per-step build cost, and the sequencing means a pause never strands a half-built feature. + +| Step | Deliverable | Value without later steps | +| :---- | :---- | :---- | +| 1 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/18)) | Lift isolation into `ops.testing`: `IsolatedEnv`, persistent worker, `from_isolated_path` | Immediately useful: run one charm in an isolated venv from a Scenario test; no `Model`, no convergence | +| 2 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/20)) | Typed `State` (de)serialiser with full leaf-type coverage and the StoredState escape hatch | Snapshot-style tests and debugging; required by every later step | +| 3 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/22)) | Persistent-worker mode replacing spawn-per-event (benchmarked in the same PR). **Acceptance:** within 2× of in-process Scenario on the 4-charm/20-event workload. See *Isolation model* | Performance baseline; unblocks realistic convergence-run testing | +| 4 | `Model` \+ `App` \+ non-relating operations: `deploy`, `add-unit`, `remove-unit`, `update-config`, `run_action`, single-app convergence | Drive a single charm through Juju-shaped operations without writing event sequences by hand | +| 5 | `integrate` \+ cross-app event propagation \+ settle policy locked. **Acceptance:** see *Settle policy considerations* (Further Information), at least one deterministic policy passes a 100×-soak; user-override hook exposed | Unlocks the multi-charm pain points; steps 1–4 stand alone if priorities pull us off here | +| 6 | `build_env=True` environment provisioning (all four plugins, caching, lockfile handling) \+ full metadata-resolution coverage. **Acceptance:** the six-shape metadata test set. See *Metadata resolution* | Productisation tail; not on the critical path for any earlier step | + +# Further Information + +### **Settle policy considerations** {#settle-policy-considerations} + +The determinism requirement is locked in *Automatic event triggering and convergence*: byte-for-byte reproducible across runs and machines; no wall-clock, PID, or dict-iteration-order leakage. + +The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is picked in step 5 against that requirement, with the following acceptance: + +- At least one deterministic policy ships and passes a **"run 100×, all final `State`s identical"** soak test. +- The API exposes a **user-override hook** for cases the default policy doesn't fit. The hook may ship empty initially. + +Picked here rather than locked up front because the choice benefits from being able to play with the convergence loop in working code. + +## **Inline charm classes vs. on-disk charms** + +`from_type` charms defined inline in a test cannot be trivially transferred to a fresh subprocess (the class object is not serialisable). Isolation realistically requires charms on disk. The API keeps both paths: inline classes run in-process with no isolation (like Scenario today); on-disk charms run isolated by default. + +The matrix of ‘works inline but not isolated’ surprises is closed by a **documentation convention** rather than a runtime check. Every operation beyond `IsolatedContext.run()` (`deploy`, `add-unit`, `integrate`, `update-config`, `run_action`, …) declares in its docstring two explicit fields: + +- inline: ok | error | n/a +- isolated: ok | error | n/a + +Where `ok` means works as documented, `error` means raises a clear typed error explaining why it's not supported in this mode, and `n/a` means doesn't make sense in this mode. + +This is enforced by API-doc review at PR time. It’s trivial to grep for. The matrix becomes readable in the docs rather than discoverable only by trial. There is no separate validation layer. + +### **Real-world charm coverage** + +The isolation and provisioning proof-of-concept implementations were validated end-to-end on synthetic test charms with intentionally conflicting deps, and the provisioning detection/extraction path was validated against two real charms (loki-k8s-operator, postgresql-k8s-operator) via a read-only `describe` CLI. Full end-to-end "provision venv, run the charm's code" against a real production charm remains unproven at spec writing time. Likely first regression class: Python-version mismatch (charms targeting `requires-python = "~=3.12"` in a 3.11 harness). + +### **Out of scope** + +- Changes to Scenario's existing single-charm API or behaviour +- charmcraft `reactive` plugin support + +### **Prior art and references** + +- [Catan](https://github.com/PietroPasotti/catan) — the abstraction this spec draws from; spike branch `explore/per-charm-dependency-isolation` at commit `1f4c357` +- [operator\#1929](https://github.com/canonical/operator/issues/1929) — origin of the `ops.testing`\-integrated direction + +# Spec History and Changelog + +Please be thorough when recording changes and progress with the spec itself and the work resulting from it. Record every meeting, attendees and conclusions from the meeting. + +| Author(s) | Status | Date | Comment | +| :---- | :---- | :---- | :---- | +| [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 8, 2026 | Initial draft | +| [Ben Hoyt](mailto:ben.hoyt@canonical.com) | Drafting | Jun 15, 2026 | Initial review, comments | +| [James Garner](mailto:james.garner@canonical.com) | Drafting | Jun 15, 2026 | Initial review, comments | +| [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 24, 2026 | Addressed comments. Added serialisation example section. | +| Person | Braindump | Date | | +| | | | | + From e9dd12e73847fe33db73f722cd15161dbafcf3dc Mon Sep 17 00:00:00 2001 From: Tony Meyer Date: Fri, 24 Jul 2026 16:52:03 +1200 Subject: [PATCH 4/4] docs: remove old-path copy of spec (renamed to OP089-multiple-charm-state-transition-tests.md) --- ...multi-charm-unit-testing-in-ops-testing.md | 697 ------------------ 1 file changed, 697 deletions(-) delete mode 100644 specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md diff --git a/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md b/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md deleted file mode 100644 index daa1cd9..0000000 --- a/specs/OPxxx-saddle-multi-charm-unit-testing-in-ops-testing.md +++ /dev/null @@ -1,697 +0,0 @@ -| Index | OP089 | | | -| :---- | :---- | :---- | :---- | -| Title | Multiple-charm state transition tests | | | -| **[Type](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Author(s)** | **[Status](https://docs.google.com/document/d/1lStJjBGW7lyojgBhxGLUNnliUocYWjAZ1VEbbVduX54/edit?usp=sharing)** | **Created** | -| Implementation | [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 8, 2026 | -| | **Reviewer(s)** | **Status** | **Date** | -| | Person | Pending Review | Date | - -# Abstract - -This spec defines a multi-charm unit (state-transition) test surface folded into `ops.testing`. It provides `deploy`, `integrate`, `add-unit`, `remove-unit`, and `update-config` as test operations, with the corresponding Juju events triggered automatically as the model converges. Each charm runs in its own isolated subprocess (its own venv), so charms with mutually incompatible dependency sets can be tested together in a single test. This spec targets `ops.testing` in `canonical/operator`; it is not a companion package. It draws the abstraction from [Catan](https://github.com/PietroPasotti/catan) (Apps \+ convergence) but is not a direct port. - -# Rationale - -### **The gap in current tooling** - -`ops.testing` (Scenario) runs one charm at a time: the developer writes an event and a `State`, calls `Context.run`, and asserts on the output. This is efficient for single-charm logic but leaves four pain points unaddressed: - -- **Integration tests are slow.** A real Juju controller bootstrap and deploy round-trip takes minutes; unit tests should be measured in seconds. -- **Scenario is unlike the `juju` commands charmers think in.** Charmers reason about `juju deploy`, `juju integrate`, `juju add-unit`. Translating that mental model into hand-crafted event sequences and `State`s is friction the tool should absorb. This has been evident in the success of Jubilant (which itself further strengthens the issue, because charmers are now also used to thinking that way in integration tests too). -- **Charm libraries don't yet ship mocks.** Cross-charm relation data is hand-crafted per test. Running the other charm's own code against the relation removes that hand-crafting: the other charm's actual reaction becomes the fixture. This may change with the acceptance of [OP077](https://docs.google.com/document/d/1A50MlUpNz6YPVvIFxFZYTDBgxNboVYlpfZB8uy8o58c/edit?usp=sharing) (Charm library testing). -- **Bundle (or solution) and topology assertions are unreachable from unit tests.** Asserting "if I deploy A and B and integrate them, A's leader ends up with config X" requires real Juju or fragile mocks today. - -The shape charmers ask Charm Tech for is ‘a multi-charm state-transition test tool that feels like Juju’. This is Catan’s abstraction (Apps, `integrate`, convergence / settle). - -Why ops.testing rather than a companion package? Catan today pins `ops-scenario~=6.0`, a superseded API that has been merged into `ops.testing` as Scenario 7 (Scenario 8 is identical to 7, other than requiring Python 3.10+). Porting off that API is **net-new work regardless of the destination**; there is no path that productises the companion-package form without a full re-targeting. Since we have to re-target anyway, we target `ops.testing` directly. This is a non-trivial addition to ops-scenario, but not so large that it merits a new ops-{something} package, with the packaging overhead that comes from that. - -# Specification - -### **Location and scope** - -The new surface lives in `ops.testing` (`canonical/operator`), the same module that provides `Context` and `State`. The single-charm primitives are **unchanged and remain the substrate**. The new surface adds a model-level layer above `Context` that owns a set of `App`s, an integration graph, and a convergence loop. Single-charm Scenario (state transition) tests continue to work exactly as today and do not require the new surface. - -### **Operation surface** - -The test API uses Juju-shaped operations (`deploy`, `integrate`, `add-unit`, `remove-unit`, `update-config`, `run_action`) that describe **intents, not events**. The model generates the correct Juju-faithful event sequence and delivers it to the affected units. - -# - -```py -from ops import testing - -model = testing.Model() # Already exists, but *not* the same as ops.Model! - -db = model.deploy( - charm="./charms/postgresql", - app="postgresql", - config={"profile": "testing"}, - num_units=1, -) -web = model.deploy( - charm="./charms/myapp", - app="myapp", - num_units=2, -) - -model.integrate(db, web) -# Or, when the endpoints need to be disambiguated: -# model.integrate(db.get_endpoint("db"), web.get_endpoint("pg")) -model.add_unit(web) -model.update_config(web, {"log_level": "debug"}) -model.remove_unit(web, unit=1) -model.run_action(db.leader, "create-backup", params={}) -``` - -# - -`Model.deploy(...)` returns an `App` handle. An `App` owns its charm reference, the metadata resolved from the charm on disk, the environment it runs in (see Isolation model below), and its per-unit `State`s. Exact type names are illustrative and will be settled in code review. - -`Model.integrate(...)` follows Jubilant's shape but takes `App` objects rather than strings — we have the objects here, and there is no live Juju model to look them up in. The default form `integrate(db, web)` works whenever the endpoint pair is unambiguous from the two charms' metadata. When it isn't, endpoints are named explicitly via `App.get_endpoint(name)`: `integrate(db.get_endpoint("db"), web.get_endpoint("pg"))`. This mirrors Jubilant's `integrate("postgresql:db", "myapp:pg")` while keeping the API object-typed. - -### **Charm source: local paths only in v1** - -`charm=` in v1 is **a path to charm source on the local filesystem**. Type is `pathlib.Path | str`; a `str` must be a relative path, matching `juju deploy` behaviour. This covers test charms and monorepos out of the box. - -**Not in v1:** - -- **Deploy from git** (with or without patches, as Catan supports) is out of scope for v1 and possibly ever — the complexity is high and the value versus a conftest-managed clone is low. -- **Deploy from Charmhub** — feasible (download → unzip → treat the unpacked source as a local path), but not v1. If added later, the same `charm=` argument would grow a Charmhub-URL form; no prototype yet. - -**Recommendation for charms defined outside your repository:** the test infra (typically a `conftest.py` fixture) should source the charm from somewhere — `git clone` is the obvious choice — and pass the resulting local path to `deploy(charm=...)`. This keeps the harness surface small and puts the "where does this charm come from" decision in the test author's hands, where it can be tailored to CI, caching, and gitignore rules per project. - -### **Automatic event triggering and convergence** - -Each operation enqueues the appropriate events (`relation-created`, `relation-joined`, `relation-changed`, `config-changed`, and so on) on the affected units. A **convergence loop** drains the queue: it picks an event, dispatches it through that unit's `Context.run(...)`, observes any new relation-data writes, status changes, or secret operations, and enqueues the downstream events those produce on peers and related units. Convergence terminates when the queue is empty and no unit's last run produced new observable changes. - -```py -model.settle() # explicit convergence point -assert web.leader.state.unit_status == ActiveStatus("ready") -assert db.leader.state.get_relation("db").local_app_data["database"] == "myapp" -``` - -Catan shows that this can be done with suitable fidelity to Juju without *actually reimplementing Juju*. We are able to take advantage of the non-specific ordering of some Juju events, and also of the access to the Juju code to understand its behaviour. - -`settle()` is also called implicitly before reading any unit’s `State` if there are pending events, so most tests are a sequence of operations followed by assertions on `app.leader.state` or `app.units[i].state`. The `State` exposed on each unit is the same `ops.testing.State` dataclass that single-charm Scenario tests use today; all existing assertions are available unchanged. - -For tests that need to assert on **intermediate** state — partway through convergence, or on the exact sequence of events dispatched — two complementary APIs are exposed: - -- **Step-at-a-time context** — mirrors Scenario's `with ctx(...) as mgr: mgr.run(...)` shape. Inside the block, the implicit-settle-on-read is suspended so assertions between steps do not drain the queue: - - ```py - with model.stepping() as stepper: - stepper.step() # dispatch exactly one queued event - assert web.leader.state.unit_status == MaintenanceStatus("configuring") - stepper.step() - assert ... - # outside the block, implicit-settle-on-read is back - ``` - - `stepper.step()` returns the `(event, unit)` it just dispatched so the loop can be predicate-driven (`while stepper.step() != ("relation-changed", web.leader): ...`) without a separate peek API. Exiting the block re-enables implicit settling. - -- **Recorded event trace** — `settle()` returns the deterministic sequence of `(event, unit, state_snapshot)` tuples it dispatched, so post-hoc assertions can walk what happened without needing to steer the loop: - - ```py - trace = model.settle() - assert [e.name for e, _, _ in trace] == ["relation-created", "relation-joined", "relation-changed", "config-changed"] - ``` - - No behaviour change to the default settle path — the return value is additive. - -Mid-flight assertions with `stepping()`, post-hoc assertions with the returned trace; neither leaks into the default settle path a simple test uses. - -Given the same starting `Model` and the same input event sequence, `settle()` **must** produce the same final `State` per unit and the same sequence of intermediate events, byte-for-byte, across runs and across machines. **No wall-clock dependence, no PID-derived ordering, no Python-dict-iteration-order leakage** is permitted in the policy. The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is decided against this requirement, when there is working code to validate it. See [Further Information](#settle-policy-considerations). - -Real Juju runs charms concurrently and per-unit event ordering is non-deterministic. Unit tests must be deterministic, so v1 ships a single ordering policy — but the design keeps the door open for exercising alternative orderings against the *same* starting sequence of events. Two mechanisms are anticipated (concrete API picked in step 5 alongside the default policy): - -- **Alternative strategies via `settle(...)` arguments** — e.g. `model.settle(strategy="data-flow")` vs `strategy="round-robin"` vs a caller-supplied deterministic function. Running the same test under two strategies is one way to smoke out charms that accidentally depend on one particular ordering. -- **Per-app / per-unit scheduling hints** — e.g. mark an `App` as "running late" so its events are held until the queue is otherwise empty, or bias one unit to drain first. This lets charmers reproduce specific ordering bugs seen in production. - -Both are optional layers on top of the default deterministic policy; the default remains what `settle()` with no arguments does. - -### **Isolation model: subprocess \+ per-charm venv** - -Each `App` runs in its own process with its own interpreter (an existing venv's `bin/python`, or one the harness provisioned; see Environment provisioning below). For each event dispatch the harness serialises `(meta, config, actions, event, State)` as JSON, sends it to the charm's worker process, the worker runs `testing.Context(...).run(...)`, and serialises the resulting `State` back. Two `App`s in the same `Model` may use mutually incompatible dependency sets (`cryptography==41` vs `==3`, different `pydantic` majors, conflicting charm-lib versions, …). - -**Required invariant:** each per-charm venv contains an `ops` / `ops-scenario` build whose JSON `State` wire format is compatible with the parent process's. Only the charm's own runtime deps differ between venvs. This is realistic, since all charms (that matter here) build on `ops`. - -**Cross-`ops`-version compatibility policy.** Running each charm in its own venv means charms may sit on different `ops` versions than the parent process — a real ongoing cost. The policy has three parts: - -- **Version negotiation at worker handshake.** On startup, the worker declares the wire-format version it supports. The parent process picks the intersection or raises `StateSchemaVersionError` (already introduced by the typed-`State` serialiser, step 2 of the incremental delivery plan). Mismatches fail fast at handshake, not silently mid-test. -- **Directionality: unidirectional, parent-authoritative.** The parent process (running `ops.testing`) is the authority; it supports schema versions `1..N` where `N` is its own current version. A charm worker on any version in `1..N` is fine — the parent decodes at the worker's declared version. A worker on `>N` (a newer `ops` than the parent) raises at handshake, because the parent cannot invent decoder logic for a future schema. **Practical invariant: the orchestrating test suite's `ops` must be ≥ every charm-under-test's `ops`.** This matches the natural test-suite topology (bump `ops.testing` first, then adopt in charms) and keeps the compat matrix strictly triangular rather than a full N×N grid. -- **Documented compatibility window.** Wire-format changes are treated as `ops` release-note items with an explicit "supported charm-side `ops` versions" range published against each `ops.testing` release — this is the *floor* (how far back a charm can be); the *ceiling* is the parent's own version (directionality, above). Charms outside the window raise at handshake with a message pointing at the range. Bounded surface, small compat matrix, testable. -- **Escape hatch: pin the charm's `ops` to the parent process's.** When `build_env=True`, the provisioner can override the charm's declared `ops` pin with the parent process's exact version via `deploy(..., pin_ops=True)`. Cuts through the compat matrix at the cost of not exercising the charm on its own declared `ops`. Off by default; opt-in per `deploy(...)`. - -The isolation worker is persistent: one long-lived process per `App`, spawned lazily on first dispatch and torn down at `Model` teardown or after an idle timeout. A framed JSON protocol over stdin/stdout carries requests and responses. Spawn-per-event remains available as an explicit debug mode (easier to attach a debugger; no shared interpreter state between events). Worker crashes propagate as a typed `IsolationError` and fail the test cleanly; the harness never silently re-spawns mid-test. - -Acceptance criteria: - -- **Workload:** the performance yardstick is a **4-charm bundle with 20 events per `settle()`** (80 dispatches total). Lives in the test suite and is re-run by every later step to catch regressions. -- **Bar:** **persistent-worker mode within 2× the in-process Scenario baseline** on that workload. 2× is the threshold where "slow but acceptable for a CI unit-test suite" tips into "too slow, charmers will avoid it and just use integration tests"; subprocess \+ JSON serialisation has a real floor below that. -- **Benchmark shape:** a benchmark suite measuring (a) in-process Scenario baseline, (b) spawn-per-event isolated, and (c) persistent-worker isolated, all on the same 4-charm/20-event workload. -- **Measured absolute numbers (step 3 draft implementation, Python 3.11.15, median of 5 reps):** - - | mode | total (80 dispatches) | per event | vs baseline | - | :---- | :---- | :---- | :---- | - | (a) in-process Scenario baseline | 0.332 s | 4.15 ms | 1.00× | - | (c) persistent-worker isolated | 1.671 s | 20.89 ms | **5.04×** | - | (b) spawn-per-event isolated | 22.94 s | 286.77 ms | 69× | - - Persistent-worker mode does **not** currently meet the 2× bar (see step 3 log in [`step3-persistent-worker-log.md`](./step3-persistent-worker-log.md)). Dominant cost is `import ops` per worker (~243 ms), paid once per isolated environment and structurally unshareable across environments; JSON/IPC is sub-millisecond per event. On this 4-worker workload the ~1.1 s of worker spawns alone already exceeds the 2× budget (~0.66 s). Persistent is nonetheless ~13.7× faster than spawn-per-event and is the correct foundation for step 5's convergence loop, where more events per worker amortise the spawn cost further. **The 2× bar itself is up for revision** in light of the numbers — the working assumption is that the threshold is comfortably higher than 2× in practice, and step 5 will re-measure with real bundle-shaped workloads before locking it in. - -### **Subinterpreters are explicitly ruled out** - -PEP 734 subinterpreters are **not** the mechanism, even assuming a Python 3.14+ requirement would be acceptable: - -- **Binary-dependency conflicts remain unsolved.** Many C extensions (`cryptography`, `pydantic-core`, `cffi`\-based packages) are not multi-interpreter-safe; two different versions of the same extension cannot coexist in one process at any Python version. These are the exact libraries motivating the work. -- **State boundary still costs a serialisation roundtrip**. The same JSON encode/decode as a subprocess, taken without the isolation benefit. -- **Maintenance cost is high** on a bleeding-edge API for no payoff over the subprocess design. - -This will be revisited only if the C-extension multi-interpreter story matures enough to load conflicting binary versions in one process, not merely because the Python version floor moves. - -## Environment provisioning - -The environment API is deliberately layered so the core primitive has a small surface and no external-tool dependencies, and the ergonomic auto-detection form is a convenience helper on top rather than the load-bearing default. - -**Layer 1 — inherit the test process** (default; no `python=` argument). No isolation, no venv work, no external tools. Runs the charm in the same interpreter as the test. Unblocks Scenario-style tests with locally-defined test charms and monorepo cases where dependencies are known compatible. - -**Layer 2 — explicit interpreter + resolved requirements** (`Model.deploy(..., python="/path/to/venv/bin/python", requirements=[...])`). The core primitive: a Python interpreter path plus a fully-resolved list of pip specifiers. Zero magic; no charmcraft-plugin knowledge; no dependency on `uv` or `poetry`. The caller has already done whatever resolution they want to do (their own build script, their own lockfile export, whatever). This form unlocks the conflicting-dependency case without pulling any external-tool complexity into the core. - - Passing `python=` without `requirements=` means "use this interpreter as-is" — the caller has already provisioned the venv. Passing `requirements=` without `python=` is an error. - -**Layer 3 — auto-provision from declared deps** (opt-in `Model.deploy(..., build_env=True)`). A convenience helper that detects the charm's build plugin, extracts the dependency list, and calls into layer 2 with the extracted values. Charmers who want ergonomics use this; charmers whose plugin detection fails, or who want control, drop to layer 2 without losing anything. - - Plugin detection is `charmcraft.yaml` first; file-based heuristic second (`uv.lock` → `uv`; `poetry.lock` → `poetry`; `pyproject.toml` → `python`; else → `charm`). Extraction per plugin: - - | Plugin | Extraction method | - | :---- | :---- | - | charm | reads `charm-requirements` files \+ `charm-python-packages` \+ AST-harvested `PYDEPS` from `lib/**/*.py` | - | python | reads `python-requirements` / `python-packages` \+ PYDEPS (included even though charmcraft's python plugin does not auto-install them: the isolation worker needs the lib importable) | - | uv | shells to `uv export --frozen --no-emit-project --no-hashes --format requirements-txt` with forwarded `--extra` / `--group` flags | - | poetry | shells to `poetry export --without-hashes --format requirements.txt` with forwarded `--with` flags | - - All forms normalise to a flat pip specifier list, then install via `uv pip install` into the per-charm venv. The venv is cached on disk, keyed by SHA-256 of (plugin, extracted requirements, lockfile/pyproject content, interpreter version, parent-process `ops` version): cold build \~3s for a test charm; warm cache hit \<0.05s (measured). Detection-and-extraction was validated against two real charms (loki-k8s-operator via the uv plugin; postgresql-k8s-operator via the poetry plugin). Layer 3 emits a clear, typed error when an extracted requirement cannot be resolved, when a lockfile is stale relative to the pyproject, or when a required external tool (`uv` / `poetry`) is missing — the failure mode is "drop to layer 2", not "fall through silently". - -**Tool dependency (layer 3 only):** `uv` on PATH for all install steps; `poetry` \+ `poetry-plugin-export` for the poetry-plugin path. Layers 1 and 2 depend on nothing beyond the Python standard library. This scopes the external-tool concern to the convenience layer where charmers opted in to the ergonomics. - -**Future:** a charmcraft-managed flow for charms already packed locally, sitting at layer 3 alongside the plugin-based extraction. - -## State (de)serialisation - -A typed schema-aware encoder/decoder pair (not `json.dumps(dataclasses.asdict(state))`) covers all `ops.testing.State` leaf types: - -| Type | Encoding | -| :---- | :---- | -| `frozenset` (on `relations`, `containers`, `secrets`, etc.) | list on encode; re-frozen on decode (dataclass `__post_init__`s re-freeze) | -| `datetime` / `timedelta` | ISO-8601 string / fractional seconds | -| `pathlib.Path` / `PurePosixPath` | string with type tag | -| Pebble enums (`ServiceStatus`, `NoticeType`, `SecretRotate`, …) | enum name | -| `pebble.Layer` (`Container.layers`) | `to_dict()` / `Layer(dict)` round-trip (not a dataclass; `asdict` won't descend) | -| Int-keyed dicts (`Secret.remote_grants: Mapping[int, set[str]]`) | keys stringified on encode; restored to `int`, inner `set` restored on decode | - -`StoredState.content` and `Container._base_plan` are `Mapping[str, Any]` of marshallable types, a superset of JSON: `bytes`, `tuple`\-vs-`list` distinction, and arbitrary `set`s. The spec commits to: - -1. **A typed escape hatch, applied always**: tag-and-encode `bytes` as base64 and `tuple` as a `["__tuple__", ...]` envelope so they round-trip losslessly. **No flag.** An opt-in would create a "passed locally because today's stored-state had no `bytes`, failed in CI when it did" failure mode; the cost (a few extra bytes per typed value) doesn't justify it. -2. **Loud error on unrecognised types**: the encoder raises `TypeError` with the type name and the path through `State`. The decoder mirrors: unknown `__type__` tags raise `TypeError`. Never silent lossy coercion. - -A small follow-up survey of in-tree charm `StoredState` usage is left as a separate work item, to confirm no charm relies on round-tripping more exotic marshallable types than `bytes`/`tuple`/`set`. The survey does not gate the implementation; it informs whether the encoder's type set needs to be extended. - -The JSON `State` representation is **part of `ops.testing`'s public API**: a per-charm venv may have a different `ops` build from the parent, and both sides must agree on the encoded form. - -1. **Explicit `state_schema_version` integer field** is embedded in every encoded `State` payload. Cheaper and more legible than overloading `ops`'s SemVer: we don't have to ship an `ops` major just because we added one field to `State` serialisation. -2. **Per-version interpretation is locked at the source-code level**: a dispatch table keyed by version. Adding a new version is ordinary work; changing the *meaning* of an existing version is not. -3. **Decoder rejects unsupported versions with a clear typed error.** A v1 producer and a v1+v2 consumer work fine. A version the decoder doesn't recognise surfaces as an explicit error, not silent corruption. -4. **`ops.testing` API-stability documentation commits to (2).** "We may add new versions; we don't change the meaning of an existing version." - -### **Serialisation examples** - -#### Container with a Pebble layer, notices, check infos, service statuses, and mounts - -This example shows the most type-envelope diversity in one place: `pebble.Layer` → **t**: "layer", `pebble.ServiceStatus` enum → **t**: "enum", `pebble.NoticeType` enum, `datetime` fields, `timedelta` fields, `PurePosixPath` and `Path` in a `Mount`, and `CheckInfo` containing multiple enum types. - -```py - state = State( - containers=frozenset([ - Container( - name='myapp', - can_connect=True, - layers={ - 'base': pebble.Layer({ - 'services': { - 'app': { - 'command': '/bin/app --port 8080', - 'startup': 'enabled', - 'override': 'replace', - }, - }, - }), - }, - service_statuses={'app': pebble.ServiceStatus.ACTIVE}, - notices=[ - Notice( - key='example.com/event', - type=pebble.NoticeType.CUSTOM, - repeat_after=datetime.timedelta(minutes=30), - first_occurred=datetime.datetime( - 2025, 6, 15, 10, 0, 0, tzinfo=datetime.timezone.utc - ), - last_occurred=datetime.datetime( - 2025, 6, 15, 10, 30, 0, tzinfo=datetime.timezone.utc - ), - last_repeated=datetime.datetime( - 2025, 6, 15, 10, 30, 0, tzinfo=datetime.timezone.utc - ), - ), - ], - check_infos=frozenset([ - CheckInfo( - name='http', - level=pebble.CheckLevel.ALIVE, - status=pebble.CheckStatus.UP, - ), - ]), - mounts={ - 'cfg': Mount( - location=pathlib.PurePosixPath('/etc/config'), - source=pathlib.Path('/tmp/cfg'), - ), - }, - ), - ]), - ) -``` - -```json - { - "state_schema_version": 1, - "state": { - "__t__": "dc", - "cls": "State", - "f": { - "config": {}, - "relations": {"__t__": "frozenset", "v": []}, - "networks": {"__t__": "frozenset", "v": []}, - "containers": {"__t__": "frozenset", "v": [ - { - "__t__": "dc", - "cls": "Container", - "f": { - "name": "myapp", - "can_connect": true, - "_base_plan": {}, - "layers": { - "base": { - "__t__": "layer", - "v": { - "services": { - "app": { - "startup": "enabled", - "override": "replace", - "command": "/bin/app --port 8080" - } - } - } - } - }, - "service_statuses": { - "app": {"__t__": "enum", "cls": "ServiceStatus", "name": -"ACTIVE"} - }, - "mounts": { - "cfg": { - "__t__": "dc", - "cls": "Mount", - "f": { - "location": {"__t__": "PurePosixPath", "v": -"/etc/config"}, - "source": {"__t__": "Path", "v": "/tmp/cfg"} - } - } - }, - "execs": {"__t__": "frozenset", "v": []}, - "notices": [ - { - "__t__": "dc", - "cls": "Notice", - "f": { - "key": "example.com/event", - "id": "1", - "user_id": null, - "type": {"__t__": "enum", "cls": "NoticeType", "name": -"CUSTOM"}, - "first_occurred": {"__t__": "datetime", "v": -"2025-06-15T10:00:00+00:00"}, - "last_occurred": {"__t__": "datetime", "v": -"2025-06-15T10:30:00+00:00"}, - "last_repeated": {"__t__": "datetime", "v": -"2025-06-15T10:30:00+00:00"}, - "occurrences": 1, - "last_data": {}, - "repeat_after": {"__t__": "timedelta", "v": 1800.0}, - "expire_after": null - } - } - ], - "check_infos": {"__t__": "frozenset", "v": [ - { - "__t__": "dc", - "cls": "CheckInfo", - "f": { - "name": "http", - "level": {"__t__": "enum", "cls": "CheckLevel", "name": -"ALIVE"}, - "startup": {"__t__": "enum", "cls": "CheckStartup", -"name": "ENABLED"}, - "status": {"__t__": "enum", "cls": "CheckStatus", "name": -"UP"}, - "successes": 0, - "failures": 0, - "threshold": 3, - "change_id": "1" - } - } - ]} - } - } - ]}, - "storages": {"__t__": "frozenset", "v": []}, - "opened_ports": {"__t__": "frozenset", "v": []}, - "leader": false, - "model": {"__t__": "dc", "cls": "Model", "f": {"name": "...", "uuid": -"...", "type": "kubernetes", "cloud_spec": null}}, - "secrets": {"__t__": "frozenset", "v": []}, - "resources": {"__t__": "frozenset", "v": []}, - "planned_units": 1, - "deferred": [], - "stored_states": {"__t__": "frozenset", "v": []}, - "app_status": {"__t__": "status", "name": "unknown", "msg": ""}, - "unit_status": {"__t__": "status", "name": "unknown", "msg": ""}, - "workload_version": "" - } - } - } -``` - -#### Secrets with `int`\-keyed `remote_grants`, rotation policy, and Relations with `int`\-keyed `remote_units_data` - -This example shows the "idict" envelope (for `dict[int, ...]` which JSON can’t represent natively), the "set" envelope inside an `idict`, and `SecretRotate` enum. - -```py - state = State( - secrets=frozenset([ - Secret( - tracked_content={'password': 'x2', 'cert': '---BEGIN CERT---'}, - latest_content={'password': 'x3', 'cert': '---BEGIN CERT---'}, - owner='app', - label='db-creds', - description='Database credentials', - expire=datetime.datetime( - 2030, 12, 31, 23, 59, 59, tzinfo=datetime.timezone.utc - ), - rotate=SecretRotate.DAILY, - remote_grants={0: {'related-app'}, 2: {'other-app/0'}}, - ), - ]), - relations=frozenset([ - Relation( - endpoint='db', - interface='postgresql_client', - remote_app_name='postgresql', - remote_units_data={ - 0: {'host': '10.0.0.1', 'port': '5432'}, - 1: {'host': '10.0.0.2', 'port': '5432'}, - }, - remote_app_data={'version': '14', 'db': 'mydb'}, - ), - PeerRelation( - endpoint='peers', - peers_data={1: {'leader': 'true', 'ready': 'yes'}}, - ), - ]), - ) -``` - -```json - { - "relations": {"__t__": "frozenset", "v": [ - { - "__t__": "dc", - "cls": "Relation", - "f": { - "endpoint": "db", - "interface": "postgresql_client", - "id": 1, - "local_app_data": {}, - "local_unit_data": { - "egress-subnets": "192.0.2.0", - "ingress-address": "192.0.2.0", - "private-address": "192.0.2.0" - }, - "remote_app_name": "postgresql", - "limit": 1, - "remote_app_data": {"version": "14", "db": "mydb"}, - "remote_units_data": { - "__t__": "idict", - "v": { - "0": {"host": "10.0.0.1", "port": "5432"}, - "1": {"host": "10.0.0.2", "port": "5432"} - } - }, - "remote_model_uuid": null - } - }, - { - "__t__": "dc", - "cls": "PeerRelation", - "f": { - "endpoint": "peers", - "interface": null, - "id": 2, - "local_app_data": {}, - "local_unit_data": {"egress-subnets": "192.0.2.0", -"ingress-address": "192.0.2.0", "private-address": "192.0.2.0"}, - "peers_data": { - "__t__": "idict", - "v": {"1": {"leader": "true", "ready": "yes"}} - } - } - } - ]}, - "secrets": {"__t__": "frozenset", "v": [ - { - "__t__": "dc", - "cls": "Secret", - "f": { - "tracked_content": {"password": "hunter2", "cert": "---BEGIN -CERT---"}, - "latest_content": {"password": "hunter3", "cert": "---BEGIN -CERT---"}, - "id": "secret:oup6t3bz6wzkkjylya5u", - "owner": "app", - "remote_grants": { - "__t__": "idict", - "v": { - "0": {"__t__": "set", "v": ["related-app"]}, - "2": {"__t__": "set", "v": ["other-app/0"]} - } - }, - "label": "db-creds", - "description": "Database credentials", - "expire": {"__t__": "datetime", "v": "2030-12-31T23:59:59+00:00"}, - "rotate": {"__t__": "enum", "cls": "SecretRotate", "name": "DAILY"}, - "_tracked_revision": 1, - "_latest_revision": 1 - } - } - ]} - } -``` - -#### `StoredState` escape hatch (`bytes`, `tuple`, `set`) \+ statuses, resources, ports, and deferred events - -This example shows the escape-hatch types that `StoredState.content` can contain: `bytes` (base64-encoded), `tuple` (`list`\-based envelope), and `set`. It also shows the "status" envelope for non-trivial status values, "Path" in a Resource, and `DeferredEvent`. - -```py - state = State( - config={'log_level': 'debug', 'port': 8080, 'enabled': True}, - leader=True, - unit_status=ActiveStatus('ready'), - app_status=BlockedStatus('needs database'), - workload_version='1.2.3', - planned_units=3, - stored_states=frozenset([ - StoredState( - name='_stored', - owner_path='MyCharm', - content={ - 'count': 42, - 'raw': b'\xde\xad\xbe\xef', - 'coords': (1, 2, 3), - 'tags': {'alpha', 'beta'}, - 'mapping': {'nested': True}, - }, - ), - ]), - resources=frozenset([ - Resource(name='oci-image', path=pathlib.Path('/tmp/image.tar')), - ]), - opened_ports=frozenset([TCPPort(8080), UDPPort(53)]), - deferred=[ - DeferredEvent( - handle_path='MyCharm/on/config_changed[1]', - owner='MyCharm', - observer='_on_config_changed', - ), - ], - ) -``` - -```json - { - "config": {"log_level": "debug", "port": 8080, "enabled": true}, - "opened_ports": {"__t__": "frozenset", "v": [ - {"__t__": "dc", "cls": "UDPPort", "f": {"port": 53, "protocol": "udp"}}, - {"__t__": "dc", "cls": "TCPPort", "f": {"port": 8080, "protocol": -"tcp"}} - ]}, - "leader": true, - "resources": {"__t__": "frozenset", "v": [ - { - "__t__": "dc", - "cls": "Resource", - "f": { - "name": "oci-image", - "path": {"__t__": "Path", "v": "/tmp/image.tar"} - } - } - ]}, - "planned_units": 3, - "deferred": [ - { - "__t__": "dc", - "cls": "DeferredEvent", - "f": { - "handle_path": "MyCharm/on/config_changed[1]", - "owner": "MyCharm", - "observer": "_on_config_changed", - "snapshot_data": {} - } - } - ], - "stored_states": {"__t__": "frozenset", "v": [ - { - "__t__": "dc", - "cls": "StoredState", - "f": { - "name": "_stored", - "owner_path": "MyCharm", - "content": { - "count": 42, - "raw": {"__t__": "bytes", "v": "3q2+7w=="}, - "coords": ["__tuple__", 1, 2, 3], - "tags": {"__t__": "set", "v": ["alpha", "beta"]}, - "mapping": {"nested": true} - }, - "_data_type_name": "StoredStateData" - } - } - ]}, - "app_status": {"__t__": "status", "name": "blocked", "msg": "needs -database"}, - "unit_status": {"__t__": "status", "name": "active", "msg": "ready"}, - "workload_version": "1.2.3" - } -``` - -## Metadata resolution - -The harness reads `metadata.yaml` / `charmcraft.yaml` from the charm source path without importing the charm in the test process. Less common cases are left as future work: - -- Resolve unified-vs-legacy metadata variants and the merged `charmcraft.yaml` \+ `metadata.yaml` form charms commonly produce -- Resolve charm libraries the charm declares but that live elsewhere on disk -- Inline charm classes (`from_type` charms defined in the test): kept as in-process, same-environment only; isolation requires charms on disk (`from_isolated_path`). See Further Information. - -At least one passing test for each of these charm shapes must be included in the implementation: - -1. Unified `charmcraft.yaml` only (no `metadata.yaml`) — the current default-modern shape. -2. Legacy split: `metadata.yaml` \+ `actions.yaml` \+ `config.yaml` — older charms still in production. -3. Mixed / transitional: `charmcraft.yaml` \+ `metadata.yaml` (the form charmcraft merges) — charms mid-migration. -4. Multi-bases (`bases:` / `platforms:` variants per supported platform). -5. Charm with libs outside `src/` (e.g. installed via `pip` from PyPI for testing) — exercises the "where did this lib come from" resolution. -6. Inline `from_type` (no metadata file; class declares everything) — documented as `inline: ok` per the docstring convention in Further Information. - -## Migration from Catan / ops-scenario\~=6.0 - -The existing `PietroPasotti/catan` package pins `ops-scenario~=6.0` (superseded by `ops.testing`). There is **no compatibility shim**: tests written against Catan must be rewritten against the new API surface. The `ops-scenario~=6.0` form is throwaway regardless of whether this spec is adopted. Porting off it is net-new work either way. - -What the spec carries forward from Catan: - -- The model: Apps \+ integration graph \+ convergence/settle -- The Juju-like operation vocabulary: `deploy` / `integrate` / `add-unit` / … - -## Incremental delivery plan - -Sequenced so each merge is **independently useful** and the area can pause cleanly after any step. This is the handle on the priority reservation: agentic coding lowers per-step build cost, and the sequencing means a pause never strands a half-built feature. - -| Step | Deliverable | Value without later steps | -| :---- | :---- | :---- | -| 1 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/18)) | Lift isolation into `ops.testing`: `IsolatedEnv`, persistent worker, `from_isolated_path` | Immediately useful: run one charm in an isolated venv from a Scenario test; no `Model`, no convergence | -| 2 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/20)) | Typed `State` (de)serialiser with full leaf-type coverage and the StoredState escape hatch | Snapshot-style tests and debugging; required by every later step | -| 3 ([proof of concept PR](https://github.com/tonyandrewmeyer/operator/pull/22)) | Persistent-worker mode replacing spawn-per-event (benchmarked in the same PR). **Acceptance:** within 2× of in-process Scenario on the 4-charm/20-event workload. See *Isolation model* | Performance baseline; unblocks realistic convergence-run testing | -| 4 | `Model` \+ `App` \+ non-relating operations: `deploy`, `add-unit`, `remove-unit`, `update-config`, `run_action`, single-app convergence | Drive a single charm through Juju-shaped operations without writing event sequences by hand | -| 5 | `integrate` \+ cross-app event propagation \+ settle policy locked. **Acceptance:** see *Settle policy considerations* (Further Information), at least one deterministic policy passes a 100×-soak; user-override hook exposed | Unlocks the multi-charm pain points; steps 1–4 stand alone if priorities pull us off here | -| 6 | `build_env=True` environment provisioning (all four plugins, caching, lockfile handling) \+ full metadata-resolution coverage. **Acceptance:** the six-shape metadata test set. See *Metadata resolution* | Productisation tail; not on the critical path for any earlier step | - -# Further Information - -### **Settle policy considerations** {#settle-policy-considerations} - -The determinism requirement is locked in *Automatic event triggering and convergence*: byte-for-byte reproducible across runs and machines; no wall-clock, PID, or dict-iteration-order leakage. - -The *implementation* of the convergence policy (round-robin, data-flow order, deterministic seed, user-overridable hook) is picked in step 5 against that requirement, with the following acceptance: - -- At least one deterministic policy ships and passes a **"run 100×, all final `State`s identical"** soak test. -- The API exposes a **user-override hook** for cases the default policy doesn't fit. The hook may ship empty initially. - -Picked here rather than locked up front because the choice benefits from being able to play with the convergence loop in working code. - -## **Inline charm classes vs. on-disk charms** - -`from_type` charms defined inline in a test cannot be trivially transferred to a fresh subprocess (the class object is not serialisable). Isolation realistically requires charms on disk. The API keeps both paths: inline classes run in-process with no isolation (like Scenario today); on-disk charms run isolated by default. - -The matrix of ‘works inline but not isolated’ surprises is closed by a **documentation convention** rather than a runtime check. Every operation beyond `IsolatedContext.run()` (`deploy`, `add-unit`, `integrate`, `update-config`, `run_action`, …) declares in its docstring two explicit fields: - -- inline: ok | error | n/a -- isolated: ok | error | n/a - -Where `ok` means works as documented, `error` means raises a clear typed error explaining why it's not supported in this mode, and `n/a` means doesn't make sense in this mode. - -This is enforced by API-doc review at PR time. It’s trivial to grep for. The matrix becomes readable in the docs rather than discoverable only by trial. There is no separate validation layer. - -### **Real-world charm coverage** - -The isolation and provisioning proof-of-concept implementations were validated end-to-end on synthetic test charms with intentionally conflicting deps, and the provisioning detection/extraction path was validated against two real charms (loki-k8s-operator, postgresql-k8s-operator) via a read-only `describe` CLI. Full end-to-end "provision venv, run the charm's code" against a real production charm remains unproven at spec writing time. Likely first regression class: Python-version mismatch (charms targeting `requires-python = "~=3.12"` in a 3.11 harness). - -### **Out of scope** - -- Changes to Scenario's existing single-charm API or behaviour -- charmcraft `reactive` plugin support - -### **Prior art and references** - -- [Catan](https://github.com/PietroPasotti/catan) — the abstraction this spec draws from; spike branch `explore/per-charm-dependency-isolation` at commit `1f4c357` -- [operator\#1929](https://github.com/canonical/operator/issues/1929) — origin of the `ops.testing`\-integrated direction - -# Spec History and Changelog - -Please be thorough when recording changes and progress with the spec itself and the work resulting from it. Record every meeting, attendees and conclusions from the meeting. - -| Author(s) | Status | Date | Comment | -| :---- | :---- | :---- | :---- | -| [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 8, 2026 | Initial draft | -| [Ben Hoyt](mailto:ben.hoyt@canonical.com) | Drafting | Jun 15, 2026 | Initial review, comments | -| [James Garner](mailto:james.garner@canonical.com) | Drafting | Jun 15, 2026 | Initial review, comments | -| [Tony Meyer](mailto:tony.meyer@canonical.com) | Drafting | Jun 24, 2026 | Addressed comments. Added serialisation example section. | -| Person | Braindump | Date | | -| | | | | -