Dynamically set the validation image directory

[banesullivan]

We ought to allow for a mechanism to automatically create a directory called validation_images adjacent to the current module being tested. This would help better organize the image cache in large projects

I'm currently working on an internal project with 100s (approaching 1000) validation images. This is spread across ~20 different sub-packages each with their own tests/ directory. Rather than having one global validation image directory, I'd like to organize images so that they are adjacent to the module being tested.

I think we could configure the pytest-pyvista fixture to automatically determine that path based on the current module and set up that directory.

My ideal structure would be:

package/
├── module_a/
│   ├── ...
│   └── tests/
│       ├── validation_images/
│       │   └── image_a.png
│       └── test_module_a.py
├── module_b/
│   ├── ...
│   └── tests/
│       ├── validation_images/
│       │   └── image_b.png
│       └── test_module_b.py
└── ...

[MatthewFlamm]

Would this mean we'd also have dynamic failed_image_dir and so on?

It might be hard to handle test directories that are nested versus separated modules. There can also be cases where someone could have nested and separated directories.

A thought is that the folder structure of the image cache could follow the folder structure from the parent. So in your example it would look like:

package/
├── image_cache/
    └── module_a/
    └── module_b/
    └── ...
├── module_a/
│   ├── ...
│   └── tests/
│       ├── validation_images/
│       │   └── image_a.png
│       └── test_module_a.py
├── module_b/
│   ├── ...
│   └── tests/
│       ├── validation_images/
│       │   └── image_b.png
│       └── test_module_b.py
└── ...

One way to implement this on a per-project basis today: VerifyImageCache.cache_dir could be used in a fixture, assuming there is a suitable parent directory that a conftest.py could reside.

[user27182]

Is it possible to simply include a separate pytest.ini config file for each directory (with image_cache_dir = validation_images for this example), and invoke pytest on a per-directory basis? That way each subdirectory can be treated like its own mini project, without any need to change the core functionality of pytest-pyvista.

[MatthewFlamm]

Is it possible to simply include a separate pytest.ini config file for each directory (with image_cache_dir = validation_images for this example), and invoke pytest on a per-directory basis? That way each subdirectory can be treated like its own mini project, without any need to change the core functionality of pytest-pyvista.

I like this idea the best so far.

[user27182]

Another idea: keep the single, flat image_cache structure, but add a --prepend_module_name option

E.g. if we have def test_add_mesh() in test_plotting.py, the image's file name is currently add_mesh.png. With --prepend_module_name, it would become plotting-add_mesh.png. We could include or exclude test from the module name.

---

If you want the name of the package (i.e. tests parent), you could also have --prepend_package_name, where the name of the parent dir to the tests dir is included regardless of how deeply nested the current test module is.

E.g. if we have def test_add_mesh() in tests/plotting/plotting.py, then --prepend_package_name, the name changes from add_mesh.png to pyvista-add_mesh.png.

---

Or combine both features and include all directories in between, maybe with a --prepend_path option.

E.g. if we have def test_add_mesh() in tests/plotting/test_plotting.py, then with --prepend_path, add_mesh.png becomes pyvista-plotting-test_plotting-add_mesh.png

---

IMO simply renaming the saved file is a lot simpler to implement and maintain than managing multiple directories