Skip to content

[tests] Remove the 'UseSystem' option.#25649

Draft
rolfbjarne wants to merge 1 commit into
mainfrom
dev/rolf/no-system-use
Draft

[tests] Remove the 'UseSystem' option.#25649
rolfbjarne wants to merge 1 commit into
mainfrom
dev/rolf/no-system-use

Conversation

@rolfbjarne

Copy link
Copy Markdown
Member

It doesn't do anything different, so there's no need for it.

It doesn't do anything different, so there's no need for it.
@rolfbjarne

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes the unused “UseSystem” test option and associated TESTS_USE_SYSTEM plumbing across CI scripts, Makefiles, xharness, and test configuration parsing, simplifying the test execution path to always use test.config and the locally produced layout.

Changes:

  • Removed TESTS_USE_SYSTEM variable setup from Azure DevOps templates and Windows generator test script.
  • Removed --use-system option from xharness and eliminated test-system.config generation/usage.
  • Simplified test path resolution/config parsing and project references to no longer branch on “system vs local” behavior.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tools/devops/automation/templates/tests/run-tests.yml Stops exporting TESTS_USE_SYSTEM in CI test jobs.
tools/devops/automation/scripts/run-generator-tests-on-windows.ps1 Removes unused TESTS_USE_SYSTEM env var for Windows generator tests.
tests/xtro-sharpie/Makefile Removes conditional assembly path selection; always uses $(DOTNET_DESTDIR) layout.
tests/xharness/Program.cs Removes the --use-system command-line option.
tests/xharness/Harness.cs Drops UseSystemXamarinIOSMac handling and always loads test.config.
tests/mmptest/mmptest.csproj Removes conditional HintPath; always points to local _mac-build mmp.
tests/Makefile Removes test-system.config target/dependency and stops passing --use-system to xharness.
tests/common/Configuration.cs Removes UseSystem logic and always parses test.config / uses unversioned dotnet root layout.
tests/.gitignore Stops ignoring test-system.config (no longer generated).

}
}

static bool? is_vsts; // if the system-installed XI/XM should be used instead of the local one.
@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ [PR Build #c7bfab7] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: c7bfab77ede9711e1398223e6ea77822edfa6c9d [PR build]

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ [PR Build #c7bfab7] Build passed (Build packages) ✅

Pipeline on Agent
Hash: c7bfab77ede9711e1398223e6ea77822edfa6c9d [PR build]

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ API diff for current PR / commit

NET (empty diffs)

✅ API diff vs stable

NET (empty diffs)

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: c7bfab77ede9711e1398223e6ea77822edfa6c9d [PR build]

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ [PR Build #c7bfab7] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: c7bfab77ede9711e1398223e6ea77822edfa6c9d [PR build]

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

🔥 [CI Build #c7bfab7] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

4 tests crashed, 20 tests failed, 163 tests passed.

Failures

❌ cecil tests

🔥 Failed catastrophically on VSTS: test results - cecil (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (MacCatalyst)

🔥 Failed catastrophically on VSTS: test results - dotnettests_maccatalyst (no summary found).

Html Report (VSDrops) Download

❌ generator tests

🔥 Failed catastrophically on VSTS: test results - generator (no summary found).

Html Report (VSDrops) Download

❌ introspection tests

4 tests failed, 2 tests passed.

Failed tests

  • introspection/iOS - simulator/Debug: LaunchTimedOut
  • introspection/iOS - simulator/Debug (ARM64): LaunchTimedOut
  • introspection/tvOS - simulator/Debug: LaunchTimedOut
  • introspection/tvOS - simulator/Debug (ARM64): LaunchTimedOut

Html Report (VSDrops) Download

❌ linker tests (tvOS)

11 tests failed, 0 tests passed.

Failed tests

  • dont link/tvOS - simulator/Debug: LaunchTimedOut
  • dont link/tvOS - simulator/Release: LaunchTimedOut
  • link sdk/tvOS - simulator/Debug: LaunchTimedOut
  • link sdk/tvOS - simulator/Release: LaunchTimedOut
  • link all/tvOS - simulator/Debug: LaunchTimedOut
  • link all/tvOS - simulator/Release: LaunchTimedOut
  • link all/tvOS - simulator/Debug (don't bundle original resources): LaunchTimedOut
  • trimmode copy/tvOS - simulator/Debug: LaunchTimedOut
  • trimmode copy/tvOS - simulator/Release: LaunchTimedOut
  • trimmode link/tvOS - simulator/Debug: LaunchTimedOut
  • trimmode link/tvOS - simulator/Release: LaunchTimedOut

Html Report (VSDrops) Download

❌ monotouch tests (macOS)

2 tests failed, 21 tests passed.

Failed tests

  • monotouch-test/macOS/Release (trimmable static registrar, all optimizations): TimedOut (Execution timed out after 1200 seconds.
    Test run crashed)
  • monotouch-test/macOS/Release (NativeAOT): TimedOut (Execution timed out after 1200 seconds.
    Test run crashed)

Html Report (VSDrops) Download

❌ sharpie tests

1 tests failed, 0 tests passed.

Failed tests

  • Sharpie tests: Failed (Execution failed with exit code 1)
    • Sharpie.Bind.Tests.OnDiskTests.ParseNoRunTest("Category.h",""): Expected success
      Assert.That(bindings.ExitCode, Is.EqualTo (0))
      Expected: 0
      But was: 1

    • Sharpie.Bind.Tests.OnDiskTests.ParseNoRunTest("CFRelatedType.h",...: Expected success
      Assert.That(bindings.ExitCode, Is.EqualTo (0))
      Expected: 0
      But was: 1

    • Sharpie.Bind.Tests.OnDiskTests.ParseNoRunTest("CodeAuditedAttrib...: Expected success
      Assert.That(bindings.ExitCode, Is.EqualTo (0))
      Expected: 0
      But was: 1

    • ... and 108 more

Html Report (VSDrops) Download

❌ windows tests

🔥 Failed catastrophically on VSTS: test results - windows (no summary found).

Html Report (VSDrops) Download

❌ xtro tests

1 tests failed, 0 tests passed.

Failed tests

  • Xtro: Failed (Execution failed with exit code 1)
    • Xamarin.Tests.Xtro.RunTest: ExitCode
      Assert.That(rv, Is.EqualTo (0))
      Expected: 0
      But was: 2

Html Report (VSDrops) Download

❌ Tests on macOS Tahoe (26) tests

1 tests failed, 4 tests passed.

Failed tests

  • monotouch-test: Failed (exit code 2)
    • No test failure details available. stderr output:
      • 2026-06-05 00:29:46.797 monotouchtest[1347:22577330] [PASS] VeryGeneric
      • 2026-06-05 00:29:46.797 monotouchtest[1347:22577330] [PASS] WrapperTypeLookupTest
      • 2026-06-05 00:29:46.798 monotouchtest[1347:22577330] Xamarin.Tests.RuntimeTest : 500.4139 ms
      • 2026-06-05 00:29:46.798 monotouchtest[1347:22577330] Xamarin.Tests : 500.464 ms
      • 2026-06-05 00:29:46.798 monotouchtest[1347:22577330] Xamarin : 507.7559 ms
      • 2026-06-05 00:29:46.798 monotouchtest[1347:22577330] bindings-test : 507.8578 ms
      • 2026-06-05 00:29:46.803 monotouchtest[1347:22576492] Tests run: 3590 Passed: 3580 Inconclusive: 10 Failed: 0 Ignored: 180
      • 2026-06-05 00:29:46.803 monotouchtest[1347:22576492] AutoRun (): completed test run on main thread
      • 2026-06-05 00:29:46.803 monotouchtest[1347:22576492] Exiting test run with success
      • make: *** [exec-monotouch-test] Error 1

Html Report (VSDrops) Download

Successes

✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ linker (iOS): All 11 tests passed. Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 11 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 11 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 20 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 23 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 20 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Ventura (13): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

Linux build succeeded

Pipeline on Agent
Hash: c7bfab77ede9711e1398223e6ea77822edfa6c9d [PR build]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants