Make runtime test R2R incompatibility properties granular (libraries vs tests, composite vs non-composite) and merged-runner aware

[jtschuster]

Summary

We currently have a coarse, somewhat overloaded set of properties for controlling how CoreCLR tests interact with ReadyToRun (R2R)/crossgen2. After the recent PR that added R2RIncompatible (#129460), it's clear we don't have a clean way to express which kind of R2R a given test is incompatible with. We should redesign these flags so a test can declare incompatibility with a specific R2R configuration, and ensure the skip logic works correctly with merged test runners.

Background / current state

There are at least three distinct "R2R surfaces" a test can interact with, and today they get conflated:

1. Runtime libraries R2R config — the framework / System.Private.CoreLib and friends being crossgen'd (including composite). The test itself isn't compiled with R2R, but it runs against an R2R'd framework, possibly in composite mode.
2. Test R2R config — the test assembly (and its dependencies) being crossgen'd, controlled per-test by the crossgen execution-script generation in src/tests/Common/CLRTest.CrossGen.targets.
3. Composite vs. non-composite — orthogonal to the above; a test can be fine in standard (per-assembly) R2R but fail in composite mode (or vice versa).

The existing knobs don't map cleanly onto these axes:

• <CrossGenTest>false</CrossGenTest> only disables crossgen compilation of the test; the framework can still be R2R, and the test still runs in R2R legs. It does not prevent execution in an R2R pipeline.
• <AlwaysUseCrossGen2>true</AlwaysUseCrossGen2> forces crossgen2 compilation in all modes.
• <R2RIncompatible>true</R2RIncompatible> (proposed in PR Add support for new R2RIncompatible flag for runtime tests #129460) makes the test's generated execution script bail out whenever RunCrossGen2 is set — i.e. it skips in all R2R legs regardless of composite/non-composite.

The concrete motivating case (#129046): Loader/SystemCoreLibDirectory fails only in R2R composite mode (the composite image isn't found beside the redirected SYSTEM_CORELIB_DIRECTORY assembly), but passes in non-composite R2R. The R2RIncompatible fix disables it for all R2R legs, which is broader than necessary and reduces coverage. We don't currently have a way to say "incompatible with composite R2R only."

Problems to solve

1. Granularity — A test should be able to declare incompatibility with a specific R2R configuration rather than all of R2R. At minimum we want to distinguish:
   • incompatible with composite R2R (framework and/or test composite image),
   • incompatible with non-composite / standard R2R,
   • incompatible with test crossgen vs. incompatible with running against an R2R'd framework/libraries.
2. Correct flag semantics — Clarify and document the difference between "don't compile the test with R2R" (CrossGenTest=false) and "don't run this test at all in R2R leg." These are different intents and should not be conflated.
3. Merged test runner support — The skip logic in CLRTest.CrossGen.targets is emitted into each test's individual execution script. We need to confirm/define behavior when many tests are bundled into a single merged test runner (e.g. Methodical_d1.csproj): a single incompatible test should be skippable without disabling the whole merged runner, and the pipeline flag (RunCrossGen2, composite mode) is evaluated at the runner level. Today an individual test's R2RIncompatible may either be ignored inside a merged runner or incorrectly skip the entire bundle — this needs to be designed explicitly.

Proposal

• Introduce a small, well-named family of properties that map to the axes above.
• Drive the execution-script skip off the specific infra flags (RunCrossGen2, composite indicator) rather than the single RunCrossGen2 check, so composite-only skips are possible.
• Define and implement merged-test-runner behavior so per-test incompatibility is honored at the granularity of the individual test, not the whole runner (or, if that's infeasible, document the constraint and fail the build when an incompatible test is placed in a merged runner).
• Update docs in docs/workflow/testing/coreclr/test-configuration.md and docs/workflow/ci/disabling-tests.md with a clear table mapping each property to the axis it controls and the leg(s) it affects.

Related

• PR Add support for new R2RIncompatible flag for runtime tests #129460 — adds R2RIncompatible (the broad version this issue would refine): Add support for new R2RIncompatible flag for runtime tests #129460
• Issue [ci-scan] Test failure: SystemCoreLibDirectory crashes in R2R_Composite (crossgen2 outerloop) #129046 — composite-only failure that motivated this: [ci-scan] Test failure: SystemCoreLibDirectory crashes in R2R_Composite (crossgen2 outerloop) #129046
• PR Disable SystemCoreLibDirectory on ReadyToRun #129178 — earlier incorrect attempt that misinterpreted CrossGenTest.
• Key file: src/tests/Common/CLRTest.CrossGen.targets.