test(e2e): credential_gating end-to-end test times out on the Kubernetes Rust smoke lanes #5263

User Story

I maintain OpenShell and run the Kubernetes Rust smoke lanes on my PRs. One test, credential_gating, regularly turns those lanes red on PRs that don't touch credentials, providers or the Kubernetes driver, so I have to rerun jobs to get a green result.

Description

e2e/rust/tests/credential_gating.rs credentialed_endpoint_gates_work_end_to_end keeps hitting the nextest 300 s cap on the Kubernetes Rust smoke lanes (Agent Sandbox v1alpha1 and v1beta1). The test isn't hanging. Its normal runtime on Kubernetes is already close to the cap:

• Passing runs take more than 180 s, and most take more than 240 s.
• On Docker it takes about 50 s, and on Podman about 75 s.

It is the most common cause of red Kubernetes Rust smoke jobs.

Where the time goes

The test runs these steps one after another, in a single test function:

• six full sandbox lifecycles (create, wait for Ready, exec, delete)
• two policy set --wait calls

On Kubernetes each lifecycle costs a pod schedule, an image pull or cache check, and supervisor startup. The test uses a fixed provider name and profile ID (PROVIDER_NAME = PROFILE_ID = "e2e-credential-gating"), so its steps can't simply run in parallel.

The cap is in .config/nextest.toml:

[profile.e2e-kubernetes]
status-level = "slow"
# Flag slow tests after 60s and terminate them after five minutes.
slow-timeout = { period = "60s", terminate-after = 5 }

Context

Evidence

Scan scope:

• Every Branch E2E Checks run created on or after 2026-10-19: 626 runs.
• Coverage is partial: 18 runs couldn't be fetched because of repeated GitHub API 502s.

Kubernetes Rust smoke jobs that completed (success or failure):

• Total: 28 timeouts in 554 Agent Sandbox jobs (5%). 28 of the 280 run attempts (10%) had at least one.
• Spread: 24 PRs, from 2026-10-19 to 2026-11-04.
• Share of failures: 28 of the 38 failed Kubernetes smoke jobs were this test. The other 10 were seven failures of six other tests (in sandbox_lifecycle, ssh_host_identity, ephemeral_cleanup and sandbox_labels) and three infrastructure failures. They are out of scope here.
• Duration: every timeout was at 300.005–300.009 s, which is the cap and not a crash.
• Docker and Podman Rust E2E lanes: no failures of this test in an earlier scan.
• VM lanes don't run this test, and Release Dev has no Kubernetes smoke lane.

Passing durations on Kubernetes come from an earlier scan of 50 passing runs:

• All 50 took over 180 s, and 45 took over 240 s.
• Docker took 47–53 s and Podman 68–83 s.
• The latest data point: on test(e2e): migrate the features and drivers suites to testkit #5137 57496136c (run 41752143964, job 126324084818, v1alpha1), the test passed but was flagged SLOW [>240.000s].

Proposed fix

1. Split the test into independent tests, one per gated-endpoint scenario. Give each test its own provider name and profile ID, for example suffixed with a run ID, so nextest can run them in parallel and each one fits easily under the cap.
2. Progress output: print a progress line to stderr for each lifecycle step. A future timeout then shows which step was slow; today a TIMEOUT gives no hint.

A per-test nextest override that raises the cap for this test would hide the problem rather than fix it, so it isn't part of this proposal.

Reproduce / verify

• Reproduce: run the Kubernetes Rust smoke lane (kubernetes-e2e, Agent Sandbox v1alpha1/v1beta1) a few times. About 1 in 10 run attempts hits the timeout. Passing runs show SLOW [>240.000s] for this test.
• Verify a fix: on the Kubernetes lanes each split test finishes well under the cap with no SLOW [>240s] flag. A repeat scan like the one above then shows no credential_gating TIMEOUTs.

Definition of Done

• credential_gating coverage is split into independent tests with unique provider and profile names, and each cleans up its own resources on failure.
• Each split test prints per-step progress to stderr.
• Each split test finishes well under the 300 s cap on both Agent Sandbox Kubernetes smoke lanes, with coverage unchanged.
• No per-test timeout override is added for credential_gating.