Skip to content

Fix misleading Validation Summary numbers in HTML report #20

Description

@dragosprisaca

Problem

The "Validation Summary" section in the HTML validation report displays numbers that don't add up, confusing users:

Total Requirements:        65
Applicable Requirements:   56
Requirements Passed:       54 of 54   ← denominator (54) ≠ Applicable (56)
Requirements Failed:        0 of 54   ← same mismatched denominator
Tests Failed:              39         ← 39 tests fail but 0 requirements fail?
Overall Result:            Pass

Three issues:

  1. WARNING counted as PASS — WARNING requirements are silently included in "Requirements Passed" (count(PASS or WARNING)), giving users no visibility into warnings.
  2. Denominator mismatch — the "of N" denominator only counts PASS + FAIL + WARNING, excluding INFORMATIONAL requirements. This doesn't match "Applicable Requirements" which includes them.
  3. No breakdown — users cannot tell how many requirements had warnings vs. informational results without scrolling through the full report body.

Root Cause

In decima-xml/src/main/resources/xsl/result.xsl (lines 338–353):

  • "Requirements Passed" numerator: count(PASS or WARNING) — lumps WARNING into passed
  • Denominator for both Passed/Failed: count(PASS or FAIL or WARNING) — excludes INFORMATIONAL
  • No separate lines for WARNING or INFORMATIONAL counts

Proposed Solution

Update the XSLT summary section to:

  • Count only PASS (not WARNING) for "Requirements Passed"
  • Use the Applicable Requirements count as the denominator for all lines (consistent with the "Applicable Requirements" row)
  • Add conditional "Requirements Warning" line (shown only when warnings exist)
  • Add conditional "Requirements Informational" line (shown only when informational results exist)

Expected result after fix:

Total Requirements:              65
Applicable Requirements:         56
Requirements Passed:             51 of 56
Requirements Warning:             3 of 56
Requirements Informational:       2 of 56
Requirements Failed:              0 of 56
Tests Failed:                    39
Overall Result:                  Pass

Now 51 + 3 + 2 + 0 = 56 = Applicable Requirements. All numbers add up and each status type is visible.

Files to Modify

  • decima-xml/src/main/resources/xsl/result.xsl — update the Validation Summary <dl> section (~25 insertions, 3 deletions)

Notes

  • The pie chart already separates PASS, WARNING, and INFORMATIONAL as distinct categories — this fix makes the text summary consistent with the chart.
  • The NOT_TESTED and NOT_IN_SCOPE exclusion logic (controlled by $ignore-not-tested-results and $ignore-outofscope-results parameters) is preserved.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions