Adopt Vitest fixtures as the E2E scenario execution model

[cv]

Problem Statement

The E2E scenario migration currently has competing execution models:

• Implement layered E2E scenario model #3588 describes a layered E2E scenario model.
• Phase 1: Environment, Manifest, Fixture, and Runtime Action Primitives (E2E audit-coverage) #4347 through Phase 11: Final Audit Reconciliation (E2E audit-coverage) #4357 break that model into implementation phases.
• test(e2e): execute real shell assertions; delete dry-run, --validate-only, and the bash runner #4380 moves toward a TypeScript scenario runner as the live source of truth.
• test(e2e-scenario): delete obsolete run.ts framework, switch CI to run-scenario.sh #4379 points the other direction by removing the TypeScript runner and keeping the shell runner path.
• test(e2e): run onboarding assertions in scenario runner #4657 strengthens the YAML/bash runner by executing declared onboarding assertions.
• docs(e2e): clarify single-runner scenario target #4939 documents that the current hybrid state is transitional and that the end state should be a single runner with minimal shell.

After reviewing the current repo state, we should decide whether the “single runner” should be a custom NemoClaw E2E runner or an existing test runner with NemoClaw-specific fixtures.

The current repo already has Vitest projects in vitest.config.ts, including e2e-scenario-framework and e2e-branch-validation. Vitest also already provides the lifecycle primitives we would otherwise have to rebuild: test discovery, filtering, reporters, timeouts, skip behavior, per-test context, fixture setup/cleanup, scoped fixtures, and abort signals.

This issue proposes that we align on Vitest as the E2E scenario execution runner, with NemoClaw providing typed fixtures, clients, assertions, and migration inventory.

cc @jyaunches

Proposed Design

Use Vitest + NemoClaw E2E fixtures as the final E2E scenario framework.

In this model:

• Vitest owns test execution, lifecycle, filtering, reporters, timeout handling, and CI integration.
• NemoClaw owns the domain layer: scenario fixtures, sandbox/gateway/provider clients, redacted process execution, artifacts, secret handling, cleanup, external flake classification, and typed assertion helpers.
• Existing scenario metadata and assertion modules become migration inputs and fixture libraries, not a second runner.
• Shell scripts are retained only as temporary bridge probes or true system-boundary fixtures where shell is the natural interface.
• Legacy test/e2e/test-*.sh scripts are deleted only after equivalent Vitest scenarios are wired into the same CI lane with matching required secrets, skips, artifacts, and failure semantics.

A final-state scenario should look more like this:

import { test } from "../framework/e2e-test.ts";

test("ubuntu repo cloud OpenClaw", async ({
  repo,
  openclaw,
  gateway,
  sandbox,
  inference,
}) => {
  await repo.installCurrent();

  const instance = await openclaw.onboard({
    agent: "openclaw",
    provider: "nvidia",
  });

  await gateway.expectHealthy(instance);
  await sandbox.expectRunning(instance);
  await inference.expectLocalChat(instance, { prompt: "Say ok.", expect: /ok/i });
});

The shared fixture layer would live under test/e2e-scenario/framework/ or a similar path and expose fixtures such as:

• artifacts
• secrets
• host
• repo
• openclaw
• hermes
• gateway
• sandbox
• inference
• providers
• networkPolicy
• cleanup

The current TypeScript scenario code should be salvaged where useful:

• test/e2e-scenario/scenarios/types.ts provides useful vocabulary.
• test/e2e-scenario/scenarios/builder.ts can become typed test data or matrix helpers.
• test/e2e-scenario/scenarios/registry.ts and scenario definitions can continue to drive matrix generation.
• test/e2e-scenario/scenarios/clients/* and assertions/* can seed the fixture/domain helper layer.

The current YAML/bash execution path should not be expanded as the durable architecture:

• test/e2e-scenario/runtime/run-scenario.sh
• test/e2e-scenario/runtime/run-suites.sh
• test/e2e-scenario/nemoclaw_scenarios/scenarios.yaml
• test/e2e-scenario/nemoclaw_scenarios/expected-states.yaml
• test/e2e-scenario/validation_suites/suites.yaml

Those files can remain during migration, but new architectural work should have an explicit path into Vitest fixtures/scenarios.

Migration Plan

1. Decision PR / docs alignment

Update #3588 and test/e2e-scenario/docs/ to state that the target is:

one E2E execution runner: Vitest, extended by NemoClaw fixtures and domain helpers.

Then clarify that #4347-#4357 are acceptance coverage phases, not requirements to build a YAML/bash runner.

2. Fixture skeleton

Add a new opt-in Vitest project, for example e2e-scenarios-live, gated by an environment variable so normal npm test remains fast and local-friendly.

Add:

• test/e2e-scenario/framework/e2e-test.ts
• fixture definitions using test.extend
• artifact capture and cleanup hooks
• secret loading/skipping helpers
• redacted command execution
• one migrated smoke scenario

3. Transitional bridge with a fuse

Allow a narrow helper such as shellProbe.run(...) for assertions that are expensive to port immediately. This should be treated as a bridge, not as a reason to keep authoring new shell suites.

Bridge helpers must:

• capture stdout/stderr/artifacts in Vitest output
• redact secrets
• support cleanup
• be associated with an owner/migration note
• be removed once the corresponding TypeScript helper exists

4. Family-by-family migration

Migrate behavior families, not files mechanically:

• Smoke/onboarding: test-full-e2e.sh, test-cloud-onboard-e2e.sh, test-onboard-*
• Inference: cloud inference, routing, OpenAI-compatible, Kimi, Bedrock, provider switching
• Messaging: split test-messaging-providers.sh into Telegram, Discord, Slack, fake-provider, and token-rotation fixtures/scenarios
• Sandbox lifecycle: rebuild, upgrade, backup/restore, crash-loop recovery, sandbox survival
• Security: credential sanitization, Telegram injection, network policy, shields
• Platform: Brev, WSL, macOS, GPU/Ollama stay special but still run through Vitest projects

5. Delete legacy entrypoints only after parity is proven

A legacy shell script is deletion-ready only when:

• equivalent Vitest coverage exists,
• it runs in the same relevant CI lane,
• required secrets/skips/runner requirements are preserved,
• artifacts are at least as useful as before,
• failure classification is not weaker,
• the migration inventory marks the old assertions as covered or intentionally retired.

Alternatives Considered

Build a custom TypeScript runner

This preserves the direction of #4380, but it duplicates core test-runner behavior that Vitest already provides. We would need to own discovery, filtering, reporters, cleanup semantics, skip behavior, concurrency, timeouts, CI output, and eventually fixture scopes.

Keep improving the YAML/bash runner

This is the direction #4657 strengthens. It is useful as a bridge and as requirements evidence, but it leaves us with a hybrid framework and makes shell/YAML the live execution authority. That conflicts with the desired end state of one runner and tests that rely on shell scripts as little as possible.

Keep both forever

This is the current accidental state. It makes the phase issues harder to interpret, encourages duplicate implementations, and makes it unclear when legacy E2E coverage can safely be retired.

Proposed Decisions

• Agree that Vitest is the final E2E scenario execution runner.
• Agree that NemoClaw builds fixtures/domain helpers, not a full custom runner.
• Agree that test(e2e): run onboarding assertions in scenario runner #4657 is unnecessary as durable architecture, though its assertion requirements can be ported.
• Agree that Phase 1: Environment, Manifest, Fixture, and Runtime Action Primitives (E2E audit-coverage) #4347-Phase 11: Final Audit Reconciliation (E2E audit-coverage) #4357 should be reinterpreted as acceptance/migration phases for Vitest fixtures and scenarios.
• Agree that new E2E scenario work should target Vitest fixtures unless it is explicitly marked as temporary bridge work.

Acceptance Criteria

• Implement layered E2E scenario model #3588 is updated or superseded to reflect the Vitest fixture-based target.
• Phase 1: Environment, Manifest, Fixture, and Runtime Action Primitives (E2E audit-coverage) #4347-Phase 11: Final Audit Reconciliation (E2E audit-coverage) #4357 include comments or edits clarifying that YAML/bash deliverables are not required as the final architecture.
• test(e2e): execute real shell assertions; delete dry-run, --validate-only, and the bash runner #4380, test(e2e-scenario): delete obsolete run.ts framework, switch CI to run-scenario.sh #4379, and test(e2e): run onboarding assertions in scenario runner #4657 have a clear disposition against this decision.
• A fixture skeleton PR adds the first live Vitest scenario using NemoClaw E2E fixtures.
• Migration docs define when a legacy shell test can be deleted.
• New scenario work has one obvious place to land.

Category

Testing

Checklist

• I searched existing issues and this is not a duplicate
• This is a design proposal, not a "please build this" request

[cv]

Here is how I would implement this as a PR stack.

The stack should start with a linear foundation, then fan out into domain migrations. The goal is to avoid one huge PR that tries to settle the architecture, add fixture infrastructure, wire CI, and port the whole legacy suite at once.

Foundation stack

1. docs(e2e): choose Vitest fixtures as the scenario runner

   Update test/e2e-scenario/docs/README.md and test/e2e-scenario/docs/MIGRATION.md from “single runner eventually” to “Vitest is the runner; NemoClaw provides fixtures.”

   This should also record the intended disposition for test(e2e): execute real shell assertions; delete dry-run, --validate-only, and the bash runner #4380, test(e2e-scenario): delete obsolete run.ts framework, switch CI to run-scenario.sh #4379, and test(e2e): run onboarding assertions in scenario runner #4657. docs(e2e): clarify single-runner scenario target #4939 is a good base, but this PR should name Vitest explicitly.

2. test(e2e): add live Vitest scenario project

   Add an opt-in e2e-scenarios-live project to vitest.config.ts, disabled unless something like NEMOCLAW_RUN_E2E_SCENARIOS=1 is set.

   Keep e2e-scenario-framework for framework/unit tests.

3. test(e2e): add scenario fixture context

   Add test/e2e-scenario/framework/e2e-test.ts with test.extend fixtures such as:

   • artifacts
   • secrets
   • cleanup
   • host
   • temporary shellProbe

   This PR should be mostly hermetic: redaction tests, cleanup tests, artifact path tests, and skip-on-missing-secret tests.

4. test(e2e): add NemoClaw E2E clients

   Port or salvage the current test/e2e-scenario/scenarios/clients/* into fixture-friendly clients:

   • host CLI
   • gateway
   • sandbox
   • provider
   • state

   No scenario migration yet. This PR should just give tests a typed API.

5. test(e2e): add first Vitest live smoke scenario

   Add the smallest real scenario, probably either:

   • ubuntu repo cli smoke, or
   • ubuntu repo cloud OpenClaw smoke, if we want to prove real onboarding immediately.

   It should prove install, artifact capture, cleanup, secret gating, and at least one real assertion.

6. ci(e2e): wire opt-in Vitest scenario workflow

   Add or update workflow wiring so maintainers can run selected Vitest scenarios manually. Keep existing shell workflows untouched for now.

   The important part is that artifacts and summaries are useful from day one.

7. test(e2e): add migration inventory and deletion gates

   Define the rule for when a legacy test/e2e/test-*.sh script is deletion-ready.

   Add a machine-readable inventory mapping legacy scripts/assertions to one of:

   • covered by a Vitest scenario
   • temporarily covered by a bridge probe
   • intentionally retired
   • not yet migrated

Migration fan-out

After the foundation stack lands, these should branch from the foundation and can mostly proceed in parallel by domain.

8. Smoke/onboarding migration

   Port test-full-e2e.sh, test-cloud-onboard-e2e.sh, test-onboard-*, double-onboard, repair, resume, and negative paths.

9. Inference migration

   Port cloud inference, routing, provider switch, OpenAI-compatible, Kimi, Bedrock, and Ollama auth proxy coverage.

10. Messaging fixture migration

    Convert fake Telegram/Discord/Slack helpers into TypeScript fixtures/providers. This should avoid exploding into all messaging scenarios immediately.

11. Messaging scenario migration

    Break test-messaging-providers.sh into separate Vitest scenarios for Telegram, Discord, Slack, token rotation, and Hermes variants.

12. Security/policy migration

    Port credential sanitization, Telegram injection, network policy, shields config, and secret-boundary checks.

13. Sandbox lifecycle migration

    Port rebuild, upgrade, backup/restore, sandbox survival, crash-loop recovery, diagnostics, and snapshot commands.

14. Platform migration

    Bring Brev, WSL, macOS, and GPU/Ollama under the same Vitest fixture model.

    test/e2e/brev-e2e.test.ts is already Vitest, so this is more about replacing remote shell-suite dispatch with shared fixtures.

Final cutover

15. ci(e2e): move nightly/regression lanes to Vitest scenarios

    Change the high-value CI lanes one by one, preserving secrets, skips, runner labels, timeouts, and artifacts.

16. test(e2e): deprecate YAML/bash scenario runner

    Mark run-scenario.sh, run-suites.sh, scenarios.yaml, expected-states.yaml, and suite YAML as frozen/legacy.

    Add tests that prevent new durable coverage from landing there.

17. test(e2e): remove retired legacy E2E runner paths

    Delete the YAML/bash scenario runner and any legacy shell scripts whose inventory rows are covered or intentionally retired.

Suggested merge discipline

PRs 1-7 should be stacked and merged in order.

PRs 8-14 can run in parallel once the fixture API is stable.

PRs 15-17 should wait until the migration inventory says we are not silently dropping coverage.