CI Improvements for stability

[ryanskeith]

Description

This PR seeks to reduce CI flakiness by fixing issue #5078 due to the fact that tests marked by serial were not running serially. This is done by using the py-test marker to run a slightly different pytest command. Also, marked a few more tests as serial during testing that were causing issues on parallel. Went more conservative thinking Stability > Speed.

Changes

1. Fixed CI workflow (.github/workflows/tests.yml)

• Serial tests now run without pytest-xdist (single process)
• Parallel tests continue using -n auto
• Cross-platform compatible using PYTEST_XDIST env var

2. Marked 5 flaky tests as serial

• test_subpackage_recipes - repodata cache locking
• test_inner_python_loop_with_output - repodata cache locking
• test_remove_workdir_default - package cache corruption
• test_overlinking_detection - package cache corruption + tool crashes
• test_conda_pkg_format - Windows file locking

3. Suppressed deprecation warning (pyproject.toml)

• Ignore CONDA_PACKAGE_EXTENSIONS warning from conda

Impact

• ✅ Serial tests now actually run serially
• ✅ Expected significant improvement in CI stability
• ⚠️ Some tests run slower (no parallelism) but reliably (hopefully)

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[codspeed-hq]

CodSpeed Performance Report

Merging this PR will not alter performance

_{Comparing ryanskeith:fix/serial-test-configuration (401f654) with main (0ade301)}

Summary

✅ 5 untouched benchmarks

[dholth]

Suggested change

	# Serial tests run without xdist (-n auto) to avoid race conditions
	# Serial tests run without xdist worker processes (-n auto) to avoid race conditions

Is there a higher-level place that a non-repeated env can go?

[ryanskeith]

Sadly, it cannot. Matrix.test-type is on the job level.

[beckermr]

I am a strong -1 on this PR. My understanding of the original issue is that changes were made so that things that had been marked as serial no longer needed to be serial. Further the conda-build tests run fine after a few retries.

OTOH this pr is producing CI times in excess of 1.5 hours and the CI is still failing.

We need to go through each test and determine how it needs to be run, removing serial markers as much as possible. Then we can merge this PR for the remaining tests that actually need to be serial.

[beckermr]

Indeed see the original issue text:

I did not change/fix that in the recent pull requests (since tests run somewhat fine and otherwise it's a waste of resources).
We actually should run most of those tests in parallel since the reasons they were marked as serial are most likely connected to reasons handled by the pull requests above.

[ryanskeith]

I feel that the code has a few issues with running in parallel. Thus moving the ones necessary to really run in serial will overall provide less restarts and thus saved time. There currently is a high chance of having to restart jobs due to parallelization. I would love to be able to run in parallel as well but I think we actually save time if we don't have to restart jobs.

Note that the serial jobs have not failed, they are only the parallel ones. I also think we could move some of the bigger tests to another nightly thread so we don't have to build pytorch every single push BUT this PR was about reducing run failures.

I think we should seek stability over speed. The claim here is restarting a failed jobs over and over again costs more than waiting for it to pass when run serially.

[jezdez]

There is also #5904, could we combine these PRs so they get merged at some point?

[beckermr]

We won't see real improvements here unless we spend the effort to understand the code and why things break. If we don't have time, that's fine, we can move on to other things. If we are going to put effort in, let's spend it on fixing bugs and understanding the code.

[beckermr]

@jezdez I suggest we work on and merge your PR #5904 before this one. Effort on that PR will help make further debugging of flaky testing failures faster.