test(suites): port the conformance suites onto testkit and run conformance-integration on the runner by matthewgrossman · Pull Request #4774 · NVIDIA/OpenShell

matthewgrossman · 2026-10-20T23:38:28Z

Summary

Ports the universal conformance suites (CLI, lifecycle, file transfer, persistent workspace, policy advisor) onto testkit and runs them with openshell-test-runner on Docker, Podman (rootful and rootless) and k3s (single and HA). Then it switches the required conformance-integration job to the runner and deletes the old tmachine conformance and policy-advisor testsuites.

The lifecycle, persistent-workspace, file-transfer and policy-advisor ports start from @elezar's draft #4629.

Context

Builds on the runner merged in #4773. It is independent of #4891 (the Keycloak features example), which adds its own row and job to e2e-suites.yml and Branch E2E.

Later: ship openshell-conformance as one program and delete the old CLI suite; migrate the features and drivers suites; bring back test plans.

Related Issue

Part of #3954

Changes

tests/suites/conformance/sandbox: test files lifecycle, cli, persistent_workspace, file_transfer and policy_advisor next to smoke. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start.
cx.track and cx.adopt in testkit, for sandboxes a test creates through the CLI.
e2e-suites.yml: a row for each conformance environment (ubuntu-docker-rootful/deb, fedora-podman-rootful/rpm, fedora-podman-rootless/rpm, ubuntu-k3s/k3s, ubuntu-k3s/k3s-ha), each running --family conformance --package sandbox. A gateway field passes --gateway (openshell, or tmachine for k3s). A family input runs the rows of one family. The disk cache is keyed per environment, and k3s skips it, as integration-runner.yml does; its history doesn't say why, so that is still to find out.
conformance-integration in Branch E2E, Release Dev and Release Qualification now calls e2e-suites.yml with family: conformance. The job ID is unchanged. Branch E2E's separate e2e-suites job is removed (its only rows are now in conformance-integration), along with its core-e2e-result entry; every caller passes CACHIX_AUTH_TOKEN.
The tmachine conformance and policy-advisor testsuites, their playbooks and the old conformance test archive are deleted. The integration-runner and integration-test defaults no longer name conformance.
The k3s installers wait for the Agent Sandbox controller to reconcile (k3s-controller-ready.yaml, moved out of the old conformance playbook). Tests don't wait for it.
Git is installed in every tmachine test environment (test-tools.yaml), for the .gitignore filtering test.
TESTING.md says what a conformance test may assert; CI.md and the watch-github-actions skill point at the runner's e2e-results-* artifacts.

Old test to new test

Old	New	Notes
`smoke` (scenario: status, create, get Ready, list, exec, delete, list empty)	`cli.rs::cli_create_exec_list_delete`	Same CLI steps. `smoke.rs::create_exec_delete` is the SDK variant (in #4773).
`sandbox_lifecycle` stop_start_preserves_workspace	`persistent_workspace.rs::persistent_workspace_survives_stop_start` (SDK) and `cli.rs::cli_stop_start_delete` (CLI)	Also `lifecycle.rs::restart_stopped_sandbox`.
`sandbox_lifecycle` stopped_can_be_deleted	`lifecycle.rs::stopped_sandbox_can_be_deleted`	Also at the end of `cli_stop_start_delete`.
`file_transfer::round_trip`	`file_transfer.rs::upload_download_round_trip`
`file_transfer::git_filtering`	`file_transfer.rs::upload_respects_gitignore`	Needs `git` where the tests run.
`file_transfer::path_safety`	`file_transfer.rs::download_stays_inside_workspace`
`policy_local`	`policy_advisor.rs::policy_local_accepts_proposals`
`mechanistic_proposal`	`policy_advisor.rs::denied_address_produces_draft_rule`
`new_hostname_proposal`	`policy_advisor.rs::denied_hostname_produces_draft_rule`

New with no old counterpart (from #4629): lifecycle.rs::long_running_exec_delete, run_to_completion_exit_delete, default_command_sandbox_is_listed_ready.

Not in this PR

The old CLI suite (tests/suites/conformance/cli, crates/openshell-conformance, e2e/support/conformance.sh) stays: the host lanes and mise run e2e:* still use it.
The features and drivers suites (oci-image, Podman userns, e2e-podman).
arm64: needs an arm64 KVM runner and arm64 build inputs.

Testing

Checks appropriate to the affected code and behavior pass
- Locally: fmt (runner, tmachine, testkit), clippy -D warnings, runner and tmachine tests, lockfile alignment, license headers, and Nix evaluation of the runner, tmachine config and test archives.
- On CI: Branch E2E passed on an earlier head of this PR (run 40402056078, 270c660f0, green on attempt 2): all five conformance rows in tmachine VMs. This head is rebased onto main and has not run yet.
Unit tests added/updated
E2E tests added/updated

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (TESTING.md, CI.md)

copy-pr-bot · 2026-10-20T23:38:32Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-10-20T23:39:04Z

🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell

copy-pr-bot · 2026-10-21T20:37:55Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

matthewgrossman · 2026-10-26T21:10:03Z

/ok to test 270c660

github-actions · 2026-10-26T21:10:27Z

Label test:e2e applied for 270c660. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

matthewgrossman · 2026-10-27T04:53:48Z

/ok to test ff09b71

Add cx.track and cx.adopt, so a test that creates a sandbox through the CLI still cleans it up after a failure: track the name before the command, adopt it afterwards for an SDK handle. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…ent-workspace suites to testkit Add the lifecycle, cli, persistent_workspace and file_transfer test files to the sandbox crate. The lifecycle, persistent-workspace and file-transfer tests are adapted from #4629. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start, and the one that can't yet (MXC) isn't conformant until it can. The policy-advisor tests are ported in the next commit but one. The OCI image, Keycloak and Podman userns suites are ported separately, in the features and drivers families. The old conformance/cli crate stays until the host lanes move off it. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The file-transfer suite builds a Git repository on the machine that runs the tests, and the old conformance playbook installed Git itself. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The k3s installer now waits until the Agent Sandbox controller reconciles a gated probe Sandbox, so a test that starts right after boot doesn't race it. The check moves out of the old conformance playbook into tests/tmachine/ansible/playbooks/k3s-controller-ready.yaml, which the installer and the old playbook both include. Tests themselves don't wait for the controller. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Port the old suite's policy.local, denied-address and denied-hostname scenarios as the policy-advisor suite, so they survive the deletion of the old CLI suite. Adapted from #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add a row for each conformance environment (Docker, Podman rootful and rootless, k3s and HA k3s) next to the Docker and Keycloak rows. Each row names the gateway its installer registers (`openshell`, or `tmachine` for k3s) and the runner passes it with --gateway. A `family` input runs the rows of one family, so a caller can run the conformance rows alone. The disk cache is keyed per environment, and k3s skips it as integration-runner.yml does. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Switch the required conformance-integration job in Branch E2E, Release Dev and Release Qualification from the tmachine conformance testsuite to the E2E Suites workflow, selecting the conformance family. The job ID is unchanged. Delete the tmachine conformance and policy-advisor testsuites, their playbooks and the old conformance test archive: the sandbox package covers the same tests, including the ported policy-advisor ones. The Branch E2E E2E Suites job now runs the features family only, so conformance does not run twice. Point the integration runner defaults and the docs at what remains. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman added this pull request to stack #4776 October 20, 2026 23:38

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch from 0b87fa4 to 0025ee3 Compare October 21, 2026 04:40

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch 2 times, most recently from 6d64703 to e3d9f01 Compare October 21, 2026 20:40

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch 2 times, most recently from e06a0eb to a23e140 Compare October 21, 2026 20:41

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from e3d9f01 to 5e2cbc5 Compare October 21, 2026 20:49

This was referenced Oct 21, 2026

test(testkit): add the testkit library for end-to-end suites #4724

Merged

feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine #4773

Merged

feat(runner): add the test runner, plans and tmachine Session API #4775

Closed

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 5e2cbc5 to cda205f Compare October 21, 2026 23:45

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch from 936e1fd to be4af48 Compare October 21, 2026 23:45

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from cda205f to 7c75fbf Compare October 22, 2026 04:51

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch from be4af48 to 63f4141 Compare October 22, 2026 04:51

matthewgrossman removed this pull request from stack #4776 October 22, 2026 04:51

matthewgrossman added this pull request to stack #4824 October 22, 2026 04:51

matthewgrossman mentioned this pull request Oct 22, 2026

test(sandbox): run a testkit suite in Branch E2E #4822

Closed

5 tasks done

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch from 63f4141 to df44e02 Compare October 22, 2026 05:03

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 7c75fbf to bb39723 Compare October 22, 2026 05:29

matthewgrossman changed the title ~~test(suites): move the conformance, oci-image, provider-refresh and podman suites to testkit~~ test(suites): port the old suites onto testkit and run them from the plan Oct 22, 2026

matthewgrossman removed this pull request from stack #4824 October 22, 2026 05:29

matthewgrossman added this pull request to stack #4825 October 22, 2026 05:30

matthewgrossman removed this pull request from stack #4825 October 22, 2026 16:01

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch from df44e02 to 4d19336 Compare October 22, 2026 16:01

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from bb39723 to 1cb1dc8 Compare October 22, 2026 16:01

matthewgrossman added this pull request to stack #4836 October 22, 2026 16:01

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 1cb1dc8 to fbcd389 Compare October 22, 2026 16:37

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch from 2b879b0 to c4f944f Compare October 23, 2026 04:46

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 6f5c621 to b9e6bfe Compare October 23, 2026 04:46

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch from c4f944f to be103d6 Compare October 23, 2026 05:29

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from b9e6bfe to 919f767 Compare October 23, 2026 05:30

matthewgrossman force-pushed the test/3954-first-testkit-lane/matthewgrossman branch 2 times, most recently from f244394 to 6106cd0 Compare October 26, 2026 20:12

matthewgrossman mentioned this pull request Oct 26, 2026

test(features): run feature tests against environment services, starting with Keycloak #4891

Open

6 tasks done

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 919f767 to 270c660 Compare October 26, 2026 21:08

matthewgrossman changed the title ~~test(suites): port the universal conformance suites onto testkit and run them from the plan~~ test(suites): port the conformance suites onto testkit and run conformance-integration on the runner Oct 26, 2026

matthewgrossman removed this pull request from stack #4836 October 26, 2026 21:09

matthewgrossman changed the base branch from test/3954-first-testkit-lane/matthewgrossman to test/3954-keycloak-features-example/matthewgrossman October 26, 2026 21:09

matthewgrossman added this pull request to stack #4938 October 26, 2026 21:09

matthewgrossman added the test:e2e Requires end-to-end coverage label Oct 26, 2026

matthewgrossman force-pushed the test/3954-keycloak-features-example/matthewgrossman branch from 3e9b04c to 38e728c Compare October 26, 2026 23:48

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from 270c660 to ff09b71 Compare October 27, 2026 04:53

matthewgrossman force-pushed the test/3954-keycloak-features-example/matthewgrossman branch from 38e728c to 4c29f9e Compare October 28, 2026 19:29

matthewgrossman and others added 8 commits October 28, 2026 13:30

test(tmachine): install Git in every test environment

03cd875

The file-transfer suite builds a Git repository on the machine that runs the tests, and the old conformance playbook installed Git itself. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

docs(testing): say what a conformance test may assert

dbdea20

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman force-pushed the test/3954-migrate-suites-to-testkit/matthewgrossman branch from ff09b71 to 4e6a9c4 Compare October 28, 2026 20:37

matthewgrossman removed this pull request from stack #4938 October 28, 2026 20:40

matthewgrossman changed the base branch from test/3954-keycloak-features-example/matthewgrossman to main October 28, 2026 20:40

Navigation Menu

Repository navigation

test(suites): port the conformance suites onto testkit and run conformance-integration on the runner - #4774

test(suites): port the conformance suites onto testkit and run conformance-integration on the runner#4774
matthewgrossman wants to merge 8 commits into
mainfrom
test/3954-migrate-suites-to-testkit/matthewgrossman

matthewgrossman commented Oct 20, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

copy-pr-bot Bot commented Oct 21, 2026

Uh oh!

matthewgrossman commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

matthewgrossman commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

matthewgrossman commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

Related Issue

Changes

Old test to new test

Not in this PR

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

copy-pr-bot Bot commented Oct 21, 2026

Uh oh!

matthewgrossman commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

matthewgrossman commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

matthewgrossman commented Oct 20, 2026 •

edited

Loading