Repository navigation
test: retire the old e2e test system onto testkit #5281
Description
User Story
I maintain OpenShell's end-to-end tests and CI. I need one place that shows how far the move from the old test system to testkit has come, what is in flight, and what is next, so reviewers and other contributors can follow it without reading every PR.
Description
OpenShell's end-to-end tests are moving from the old system to the new one:
- Old: the
openshell-conformancetest library and its CLI suite (both now deleted), tmachine tests run through Ansible playbooks, and the ~70 test files ine2e/rust. - New: tests written with testkit under
tests/suites/, run inside a tmachine VM by the runner, or against any existing gateway with theopenshell-conformanceprogram.
This issue tracks the remaining work to retire the old system, plus the flake and CI-time fixes made alongside it. Each piece of work keeps its own issue and PR. Sub-issues are listed below this description.
This issue covers part of the scope of #3954 (testing strategy, closed by the stale bot). RFC 0016 and the broader testing strategy stay out of scope here.
Done
- test(testkit): add the testkit library for end-to-end suites #4724 (testkit), refactor(tests): move tmachine support files under tests/tmachine #4726 (tmachine layout), feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine #4773 (runner)
- test(features): run feature tests against environment services, starting with Keycloak #4891 (features pattern, Keycloak example)
- test(suites): port the conformance suites onto testkit and run conformance-integration on the runner #4774 (conformance tests on testkit, required on 5 VM setups)
- test(e2e): run the testkit conformance suite in the host lanes #5078 (host lanes run the new tests; old CLI suite deleted)
- test(testkit): add cleanup hooks that run after pass, failure, or panic #5219 and test(e2e): migrate the features and drivers suites to testkit #5137 (testkit cleanup hooks; OCI image, Podman user namespace and Keycloak tests moved; old library deleted)
- test(testkit): add the openshell-conformance test program #5249 and test(testkit): name unsupported gateway auth modes in the error #5248 (the
openshell-conformanceprogram; the host lanes run it, built once per architecture in Branch E2E) - Fixes found by the new suites: fix(kubernetes): keep a late stop snapshot from reviving a stopped sandbox #5196 (Kubernetes stop race, bug(kubernetes): stopped sandbox regresses to Provisioning after stop completes #5195), fix(tests): pin the k3s installer to a release #5223 (k3s installer pin)
What's left of the old system: e2e/rust and the tmachine e2e-podman testsuite.
Migrating e2e/rust
Rules for every batch:
- Each batch deletes the
e2e/rustfiles it ports, soe2e/rustshrinks as batches land. - Ported tests use unique names and can run in parallel against one gateway, so lanes don't get slower as tests move.
- Overlaps with tests already on testkit are resolved by reading both, not by porting twice.
| Step | What moves | Destination | Depends on |
|---|---|---|---|
| Decide | A keep, drop or merge decision for every file in e2e/rust/tests/, with a destination and a batch (#5297) |
This issue's table, updated | — |
| Prepare | host.openshell.internal on k3s and the Python workload image in every VM (#5275, PR #5277); per-family timeouts, serial groups and a thread count on runner rows (#5295) |
tmachine environments, runner | — |
| Pilot | credential_gating, after it's split into independent tests (#5263, PR #5264). Adds only the harness helpers it needs (#5296) |
conformance |
#5264, #5277 |
| Podman | The 8 files the e2e-podman lane runs |
drivers/podman |
#5297, #5277 |
| Docker | The 7 files the e2e-docker lane runs, plus config_push, driver_config_volume and otlp_relay if they fit |
New drivers/docker |
#5297, #5277 |
| Common | About 30 driver-agnostic files (the e2e and e2e-host-gateway lanes), in 2–3 PRs by size |
conformance |
#5297, pilot |
| Gateway reconfiguration | additional_ca, oidc_pkce, local_driver_token_restart and other tests that restart or reconfigure the gateway |
New gateway-control family |
#5297 |
| Kubernetes | Corporate proxy, runtime reconcile, HA, shared supervisor (3 files), workspace namespace (2), credential drivers | New kubernetes support crate and family |
Common started |
| Host-only | VM, GPU, performance and edge tunnel: ported to testkit, kept on host lanes | Matching family | Late |
| Delete | e2e/rust, the Rust steps in the e2e wrapper scripts, the tmachine e2e-podman testsuite. Each remaining host-only lane moves to a tmachine environment or keeps a written reason |
— | All of the above |
Podman, Docker and Common touch different crates, so they can run in parallel once the decision and preparation steps land. Kubernetes and the slow shared-supervisor and HA tests go last, because Kubernetes lifecycle races are still being fixed.
Using the openshell-conformance program everywhere
- Runner rows run the prebuilt program (
ready, thenrun) instead of building a nextest archive per row. - Release Dev, release qualification and GPU lanes use the prebuilt program.
- Rename the runner to a tmachine-specific name.
Flakes and CI time
- Flakes: test(e2e): split credential_gating into independent parallel tests #5264 (
credential_gatingtimeouts on Kubernetes, test(e2e): credential_gating end-to-end test times out on the Kubernetes Rust smoke lanes #5263), fix(server): skip the reconcile poll of a sandbox whose start is in progress #5265 (Podman restart race, bug(podman): polled snapshot moves a restarting sandbox from Starting to Error (ContainerExited) #5258). - CI time: ci(e2e): start Linux and tmachine jobs without waiting on macOS or package builds #5270 (ci: start Branch E2E Linux and tmachine jobs without waiting on macOS builds or package builds #5268, start Linux and tmachine jobs without waiting on macOS or package builds), then ci: start Release Dev Linux jobs without waiting on macOS builds #5269 (the same change for Release Dev). After that, build the runner and the suite archives once per run instead of once per tmachine row (about 5 min per row).
Not needed for retirement, planned after it: generate CI matrices from a list of tests and environments, and check that every expected test ran.
Definition of Done
-
e2e/rustis deleted, and its tests run on testkit. - The tmachine
e2e-podmantestsuite and the Rust steps in the e2e wrapper scripts are deleted. - CI, Release Dev and release qualification run only testkit suites.
- Each remaining host-only CI lane has a written reason it can't run on tmachine.
Activity