Apply dotnet test zero-tests verdict at the whole-run level (microsoft/testfx#7457)#55167
Draft
Evangelink wants to merge 2 commits into
Draft
Apply dotnet test zero-tests verdict at the whole-run level (microsoft/testfx#7457)#55167Evangelink wants to merge 2 commits into
Evangelink wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the dotnet test (Microsoft.Testing.Platform path) side of microsoft/testfx#7457: make the "zero tests ran" (exit code 8) verdict apply to the whole run instead of failing once per test module.
Today
TestApplicationActionQueueaggregates the raw per-module exit code. When a run uses--test-modulesor a global--filter, a module that matches no tests exits with8(ZeroTests), which poisons the aggregate exit code even though other modules ran fine. There is no way to get a single run-level "zero tests" result short of--ignore-exit-code 8(which suppresses it everywhere).Change
TestApplicationActionQueue: normalize a per-moduleExitCode.ZeroTests(8) toSuccessbefore aggregating. An individual module that matched no tests is no longer treated as a whole-run failure.MicrosoftTestingPlatformTestCommand.Run: decide the whole-run zero-tests verdict once, fromoutput.TotalTests. When no global--minimum-expected-testsis set and the run executed zero tests in total, the command returnsExitCode.ZeroTests(8). When a global--minimum-expected-tests Nis set, the existing aggregate check againstoutput.TotalTestsgoverns.Behavior
--filter): total tests0→ still exit code8(unchanged).--test-modules/ global filter): run succeeds instead of failing with8— this is the requested fix.dotnet test --minimum-expected-tests N: global check on the aggregate total (unchanged).dotnet test -- --minimum-expected-tests N(forwarded per-module): a module that runs fewer thanNtests fails withExitCode.MinimumExpectedTestsPolicyViolation(9), which is not normalized here and is preserved. Both a global and a forwarded minimum can be combined and are honored independently.Why normalize
8rather than flow--minimum-expected-tests 0downFlowing
--minimum-expected-tests 0to each module was considered but rejected: older Microsoft.Testing.Platform versions reject--minimum-expected-tests 0(it required a positive value), and the SDK does not know a module's MTP version at argument-build time, so flowing it would break those modules with an invalid-command-line error. Normalizing the child8is an SDK-only interpretation with no MTP-version dependency.Depends on / pairs with
microsoft/testfx PR #9709: it makes an explicit
--minimum-expected-tests Nviolation return exit code9(not8) even when zero tests ran. That is what lets this PR safely normalize8without swallowing a forwarded per-module minimum. With an older MTP that still returns8for a forwarded minimum + zero tests, that narrow edge would be normalized to success; with the paired testfx change it is reported correctly as9.Notes
Fixes the
dotnet testside of microsoft/testfx#7457.