Repeating a test causes file not found regression error

[akaszynski]

When repeating a test using a flaky test fixture within PyVista, the incremented test run resulted in pytest-pyvista being unable to find the image in the cache:

from https://github.com/pyvista/pyvista/actions/runs/17310223268/job/49142681675?pr=7832

FAILED tests/plotting/test_plotting.py::test_box_axes - pytest_pyvista.pytest_pyvista.RegressionFileNotFoundError: tests/plotting/image_cache/box_axes_1.png does not exist in image cache
= 1 failed, 1727 passed, 27 skipped, 2 xfailed, 5 warnings in 1609.45s (0:26:49) =
py3.10-numpy_1.26-vtk_9.2.2-cov-plotting: exit 1 (1615.59 seconds) /home/runner/_work/pyvista/pyvista> pytest tests/plotting --disallow_unused_cache --cov pyvista --cov-config /home/runner/_work/pyvista/pyvista/pyproject.toml --cov-context test --cov-report term:skip-covered --cov-report xml:coverage_py3.10-numpy_1.26-vtk_9.2.2-cov-plotting.xml --cov-report html:coverage_py3.10-numpy_1.26-vtk_9.2.2-cov-plotting pid=30205

[user27182]

Probably just need to reset the call count at some point. We can move that fixture to this repo, add some tests, and inject a reset somewhere.

Though I think a flaky test for image regression errors should probably be dealt with in other ways other than retrying. E.g. with #197 or with a high variance flag

[akaszynski]

Though I think a flaky test for image regression errors should probably be dealt with in other ways other than retrying. E.g. with #197 or with a high variance flag

Having a flaky directory would be helpful, and so would being able to set the threshold for a specific test.

[user27182]

#198 brings in the flaky test code for the documentation tests. Can probably refactor parts of it and reuse it with the unit tests.

[user27182]

#198 brings in the flaky test code for the documentation tests. Can probably refactor parts of it and reuse it with the unit tests.

The same "multiple cache images" feature from #198 for the doc image tests is implemented for unit test images in #211. Although the API and generated outputs are pretty much identical for both use cases, the way the code is implemented for both uses is sufficiently different for it to be somewhat impractical to refactor and reuse most of the code.

Any refactoring to reuse code from these two PRs will likely need to be done in a separate PR after both are merged.