Conditional test filtering framework for PR builds

[MichaelSimons]

Summary

Introduces a framework for conditionally running tests on PR builds based on what source files changed. Tests that are unrelated to the PR's changes can be skipped, reducing Helix compute consumption and improving PR validation times. All tests continue to run unconditionally on CI builds (main, release branches).

This is posted as a draft to seek feedback on the approach. The implementation is included to illustrate the direction — it is intentionally simple and small despite addressing a complex problem.

Design

See documentation/project-docs/pr-test-filtering.md for the full motivation, mechanisms, and instructions for adding new scopes.

What's included

• Design document with motivation, mechanisms, and instructions for adding scopes
• C# evaluation script
• Central config file (ConditionalTests.props) with TemplateEngine as the first scope
• MSBuild + pipeline integration
• Agent instructions update

[joeloff]

If we refactor, is this at risk of breaking? Can't you pass the repo root from the .csproj and generate a .cs file that's added to <Compile Include="..."/> That way the code always tracks the real root?

[MichaelSimons]

Good point on the refactor fragility. Given this is used as a pipeline script the generated file feels more complex than necessary. I think it would be sufficient to pass in the path to the props file as an arg.

[nagilson]

Overall, I'm in favor of the proposal.

However, I want to poke on two points:

1. How will we define TriggerPaths?
   I think this should be more than the folder containing relevant source code for that repo - e.g. dotnet-watch project should not be conditioned on just dotnet-watch but also include the ProjectReference(s) such as ..\..\Cli\Microsoft.DotNet.Cli.Definitions\Microsoft.DotNet.Cli.Definitions.csproj. Changes to PackageReference(s) such as Microsoft.Build.Locator" may also result in a break.

dotnetup already uses conditional test filtering to a degree but we've seen more breaks than expected when running all of the tests. So, there is a risk involved in disabling these tests though on average I think it's the right approach. We should try to minimize that risk.

There have been times where we caught product bugs (e.g. system command line, roslyn, and others) because tests in other areas started failing even when roslyn's tests were all passing. But we shouldn't rely on test coverage of others like that.

2. When can we ignore 'CoditionalTests' - I think we should be able to ignore that. We can still get a lot of the time saving on codeflow or certain dev PRs... but I think for example in dotnet/dotnet flow we will still want to run all of the tests so we get full filtering.

I'd propose to have the conditional tests condition test runs out on code flow for specific repos - e.g. a change to dotnet/templating code flow shouldn't need to run the watch tests. If we are all encompassing on dependencies with 1 then maybe we can be looser on that?

[MichaelSimons]

@noahfalk, thanks for the insightful feedback.

1. How will we define TriggerPaths?

You are correct that defining just the src path for the affected area is often not sufficient. I did not elaborate on this in the documentation but should call it out in order to help devs and AI make better decisions when defining new test scopes.

So, there is a risk involved in disabling these tests though on average I think it's the right approach. We should try to minimize that risk.

Yes this is in ways a big hammer so it must be used correctly. Having good documentation calling these gotchas is imperative.

2. When can we ignore 'CoditionalTests'

I do see it valuable to be able to specify a RunAlways setting to be able to force tests to run on code flows. Given the vmr, it might not provide substantive value in providing fined grained support.

I updated the doc in d518adb in response to this feedback.

[marcpopMSFT]

You're proposing starting with these five but potentially expanding it later to more? I assume if there were any more obvious buckets, they'd be here. Is the main remaining problem that we have too many really large or generic test assemblies (like dotnet-tests)?

[MichaelSimons]

Yes, these are some initial scopes to consider mostly used to illustrate the size impact of filtering. I updated the doc to specify some initial aspirational goals which are to define scopes for 1/3 to 1/2 of the total test time. Anything beyond that is additional investment that must be weighed against the ROI.

Also note that tests do not necessarily have to be separated into separate assemblies to be filtered. The document does call out how future support could be added to condition tests at the TestClass level. The SDK currently has some classes that are rather expensive to run where splitting them into a stand alone assembly may not be the best decision.