Skip to content

test(runner): apply per-family nextest config and a thread count to runner rows #5295

Open

Description

@matthewgrossman

User Story

I maintain OpenShell's end-to-end tests. When a family of tests moves from e2e/rust onto testkit, the tmachine runner has to give it the timeouts and concurrency the old lanes gave it, so a ported test doesn't pass in the old lane and time out in the runner.

Description

The runner runs every family with one fixed nextest profile: 30 s slow-timeout × 4 and one test thread. The old e2e/rust lanes get more from the repository's .config/nextest.toml: per-test slow-timeout overrides and test groups such as kubernetes-ha and kubernetes-shared-supervisor-pool. The runner passes its own --config-file, so that file doesn't apply to runner rows.

Add the runner side of this when the first batch that needs it is ready:

  • Pass the runner's openshell profile as a nextest tool config (--tool-config-file openshell-test-runner:<path>) instead of --config-file, so a family's own config is read.
  • Have the bundle carry tests/suites/<family>/.config/nextest.toml into the remapped workspace, so a family can set per-test timeouts and test groups, or override the profile.
  • Add a --test-threads N runner flag. The default of 1 keeps today's serial --no-capture run. Parallel runs lose streamed output, so document that.
  • Add a test_threads column to the e2e-suites.yml rows so CI can set it.

An implementation was written and reviewed in #5277 (commit 8441388e5, reverted before merge to keep that PR small). Start from it.

Context

Likely first users:

  • The Kubernetes batch: shared-supervisor, HA and rebalancing tests need the existing test groups and longer timeouts.
  • credential_gating after test(e2e): split credential_gating into independent parallel tests #5264 splits it: independent tests with unique names can run in parallel.
  • Any common batch that wants to run in parallel so lanes don't get slower as tests move in.

Not needed by the first conformance, Podman or Docker batches if they run serially under the default profile.

Part of #5281.

Definition of Done

  • A family can set per-test timeouts and test groups through tests/suites/<family>/.config/nextest.toml, and they apply under the runner.
  • --test-threads is configurable from the runner and from the e2e-suites.yml rows; the default is unchanged.
  • Runner unit tests cover the nextest arguments and the family config copy.
  • TESTING.md documents the behavior.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions