Skip to content

Apply dotnet test zero-tests verdict at the whole-run level (microsoft/testfx#7457)#55167

Draft
Evangelink wants to merge 2 commits into
dotnet:mainfrom
Evangelink:zero-tests-whole-run-7457
Draft

Apply dotnet test zero-tests verdict at the whole-run level (microsoft/testfx#7457)#55167
Evangelink wants to merge 2 commits into
dotnet:mainfrom
Evangelink:zero-tests-whole-run-7457

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Summary

Implements the dotnet test (Microsoft.Testing.Platform path) side of microsoft/testfx#7457: make the "zero tests ran" (exit code 8) verdict apply to the whole run instead of failing once per test module.

Today TestApplicationActionQueue aggregates the raw per-module exit code. When a run uses --test-modules or a global --filter, a module that matches no tests exits with 8 (ZeroTests), which poisons the aggregate exit code even though other modules ran fine. There is no way to get a single run-level "zero tests" result short of --ignore-exit-code 8 (which suppresses it everywhere).

Change

  • TestApplicationActionQueue: normalize a per-module ExitCode.ZeroTests (8) to Success before aggregating. An individual module that matched no tests is no longer treated as a whole-run failure.
  • MicrosoftTestingPlatformTestCommand.Run: decide the whole-run zero-tests verdict once, from output.TotalTests. When no global --minimum-expected-tests is set and the run executed zero tests in total, the command returns ExitCode.ZeroTests (8). When a global --minimum-expected-tests N is set, the existing aggregate check against output.TotalTests governs.

Behavior

  • Single project / all modules empty (e.g. a typo in --filter): total tests 0 → still exit code 8 (unchanged).
  • Some modules have tests, others match nothing (--test-modules / global filter): run succeeds instead of failing with 8 — this is the requested fix.
  • dotnet test --minimum-expected-tests N: global check on the aggregate total (unchanged).
  • dotnet test -- --minimum-expected-tests N (forwarded per-module): a module that runs fewer than N tests fails with ExitCode.MinimumExpectedTestsPolicyViolation (9), which is not normalized here and is preserved. Both a global and a forwarded minimum can be combined and are honored independently.

Why normalize 8 rather than flow --minimum-expected-tests 0 down

Flowing --minimum-expected-tests 0 to each module was considered but rejected: older Microsoft.Testing.Platform versions reject --minimum-expected-tests 0 (it required a positive value), and the SDK does not know a module's MTP version at argument-build time, so flowing it would break those modules with an invalid-command-line error. Normalizing the child 8 is an SDK-only interpretation with no MTP-version dependency.

Depends on / pairs with

microsoft/testfx PR #9709: it makes an explicit --minimum-expected-tests N violation return exit code 9 (not 8) even when zero tests ran. That is what lets this PR safely normalize 8 without swallowing a forwarded per-module minimum. With an older MTP that still returns 8 for a forwarded minimum + zero tests, that narrow edge would be normalized to success; with the paired testfx change it is reported correctly as 9.

Notes

  • ⚠️ Draft: this was authored without a local dotnet/sdk build. Needs CI to compile and run, and SDK-side tests should be added (e.g. a multi-module scenario where one module matches no tests asserts a successful run, and an all-empty run asserts exit code 8).

Fixes the dotnet test side of microsoft/testfx#7457.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant