Skip to content

Make --doc_mode image regression thresholds configurable #285

Description

@banesullivan

pytest-pyvista already lets regular image tests control comparison thresholds through VerifyImageCache(error_value=..., warning_value=...), but --doc_mode still uses hard-coded defaults of 500.0 and 200.0.

In practice, this makes downstream docs builds flaky. In PyVista's CI we are seeing doc image comparisons fail just over the line, for example:

sphx_glr_openfoam_cooling_001_vtksz Exceeded image regression error of 500.0 with an image error equal to: 513.5843137255087

see pyvista/trame-pyvista#62

Current behavior

doc_mode.py compares images using the module defaults:

DEFAULT_ERROR_THRESHOLD: float = 500.0
DEFAULT_WARNING_THRESHOLD: float = 200.0

And they are used directly in doc mode here:

allowed_error=DEFAULT_ERROR_THRESHOLD,
allowed_warning=DEFAULT_WARNING_THRESHOLD,

As far as I can tell, there is no CLI or ini option for overriding these values in doc mode today and we cannot set the module level vars since doc_mode.py imports its own copies of those name

Proposed behavior

Expose doc-mode thresholds through config, ideally using the same config patterns the plugin already uses elsewhere.

Any of these would solve the problem:

  1. Add common threshold options and allow doc_-prefixed overrides, for example error_threshold / warning_threshold plus doc_error_threshold / doc_warning_threshold.
  2. Add doc-specific CLI flags such as --doc_error_threshold and --doc_warning_threshold.
  3. Keep the current defaults, but let downstream projects opt into slightly higher thresholds when their docs images are known to be noisier in CI.

It looks like doc_mode.py could read the thresholds from config via _get_option_from_config_or_ini(...) instead of using the imported defaults directly. That would line up with the rest of the plugin's doc-mode configuration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions