Repro:
Step 1) lighthouserc assertion config: {"image-alt": ["error", {"maxLength": 1}]}
Step 2) LHR before: 1 failing item in image-alt, node.selector = img.header-logo.
Step 3) Fix img.header-logo, but introduce a new failing image-alt item on a different element, node.selector = img.footer-icon.
Step 4) Run the assertion via getAllAssertionResults.
Actual: {"name":"maxLength","expected":1,"actual":1,"passed":true,"auditId":"image-alt",...} -- identical result before and after, no selector in the assertion output.
Expected: an assertion meant to cap known accessibility debt per audit shouldn't silently pass when the specific elements failing have changed.
Why: maxLength (packages/utils/src/assertions.js) evaluates result.details.items.length -- a raw count of failing items for that audit -- without comparing which items (by node.selector) they are. It's documented as a general threshold option usable on any audit, including accessibility audits like image-alt, aria-*, color-contrast, label.
Suggest: at minimum, document that maxLength only bounds count, not identity. Ideally, an assertion mode that also flags when the current failing selectors differ from a recorded baseline, even if maxLength is still satisfied.
Reproduced locally using the real getAllAssertionResults function from this repo, fed synthetic LHRs.
Background, optional: https://doi.org/10.5281/zenodo.21908527
Repro:
Step 1) lighthouserc assertion config: {"image-alt": ["error", {"maxLength": 1}]}
Step 2) LHR before: 1 failing item in image-alt, node.selector = img.header-logo.
Step 3) Fix img.header-logo, but introduce a new failing image-alt item on a different element, node.selector = img.footer-icon.
Step 4) Run the assertion via getAllAssertionResults.
Actual: {"name":"maxLength","expected":1,"actual":1,"passed":true,"auditId":"image-alt",...} -- identical result before and after, no selector in the assertion output.
Expected: an assertion meant to cap known accessibility debt per audit shouldn't silently pass when the specific elements failing have changed.
Why: maxLength (packages/utils/src/assertions.js) evaluates result.details.items.length -- a raw count of failing items for that audit -- without comparing which items (by node.selector) they are. It's documented as a general threshold option usable on any audit, including accessibility audits like image-alt, aria-*, color-contrast, label.
Suggest: at minimum, document that maxLength only bounds count, not identity. Ideally, an assertion mode that also flags when the current failing selectors differ from a recorded baseline, even if maxLength is still satisfied.
Reproduced locally using the real getAllAssertionResults function from this repo, fed synthetic LHRs.
Background, optional: https://doi.org/10.5281/zenodo.21908527