pytest-pyvista already lets regular image tests control comparison thresholds through VerifyImageCache(error_value=..., warning_value=...), but --doc_mode still uses hard-coded defaults of 500.0 and 200.0.
In practice, this makes downstream docs builds flaky. In PyVista's CI we are seeing doc image comparisons fail just over the line, for example:
sphx_glr_openfoam_cooling_001_vtksz Exceeded image regression error of 500.0 with an image error equal to: 513.5843137255087
see pyvista/trame-pyvista#62
Current behavior
doc_mode.py compares images using the module defaults:
|
DEFAULT_ERROR_THRESHOLD: float = 500.0 |
|
DEFAULT_WARNING_THRESHOLD: float = 200.0 |
And they are used directly in doc mode here:
|
allowed_error=DEFAULT_ERROR_THRESHOLD, |
|
allowed_warning=DEFAULT_WARNING_THRESHOLD, |
As far as I can tell, there is no CLI or ini option for overriding these values in doc mode today and we cannot set the module level vars since doc_mode.py imports its own copies of those name
Proposed behavior
Expose doc-mode thresholds through config, ideally using the same config patterns the plugin already uses elsewhere.
Any of these would solve the problem:
- Add common threshold options and allow
doc_-prefixed overrides, for example error_threshold / warning_threshold plus doc_error_threshold / doc_warning_threshold.
- Add doc-specific CLI flags such as
--doc_error_threshold and --doc_warning_threshold.
- Keep the current defaults, but let downstream projects opt into slightly higher thresholds when their docs images are known to be noisier in CI.
It looks like doc_mode.py could read the thresholds from config via _get_option_from_config_or_ini(...) instead of using the imported defaults directly. That would line up with the rest of the plugin's doc-mode configuration.
pytest-pyvistaalready lets regular image tests control comparison thresholds throughVerifyImageCache(error_value=..., warning_value=...), but--doc_modestill uses hard-coded defaults of500.0and200.0.In practice, this makes downstream docs builds flaky. In PyVista's CI we are seeing doc image comparisons fail just over the line, for example:
see pyvista/trame-pyvista#62
Current behavior
doc_mode.pycompares images using the module defaults:pytest-pyvista/pytest_pyvista/pytest_pyvista.py
Lines 47 to 48 in 144eafb
And they are used directly in doc mode here:
pytest-pyvista/pytest_pyvista/doc_mode.py
Lines 553 to 554 in 144eafb
As far as I can tell, there is no CLI or ini option for overriding these values in doc mode today and we cannot set the module level vars since
doc_mode.pyimports its own copies of those nameProposed behavior
Expose doc-mode thresholds through config, ideally using the same config patterns the plugin already uses elsewhere.
Any of these would solve the problem:
doc_-prefixed overrides, for exampleerror_threshold/warning_thresholdplusdoc_error_threshold/doc_warning_threshold.--doc_error_thresholdand--doc_warning_threshold.It looks like
doc_mode.pycould read the thresholds from config via_get_option_from_config_or_ini(...)instead of using the imported defaults directly. That would line up with the rest of the plugin's doc-mode configuration.