Skip to content

Explore SARIF/JUnit result surfacing for the test tiers #20

Description

@perdrizat

Once the UAT tier is otherwise settled (scenarios stable, tooling mature — see the companion "UAT backlog scenarios" issue), explore standards-based result surfacing so findings show up where reviewers already look:

  • SARIF (Static Analysis Results Interchange Format) — severity-leveled findings that render natively in GitHub code scanning / IDE integrations.
  • JUnit XML — a de facto standard for CI test-result reporting, would let UAT's report.json output plug into any CI dashboard that already parses JUnit.

Open question (the hard part)

UAT's assertions[] include LLM visual judgments (contrast, occlusion, layering, "looks right"), which are not reproducible run-to-run by design — see TESTING.md "UAT is not run in CI." Before adopting either format:

  • Decide how to represent a visual-judgment assertion's severity/confidence in a format designed for deterministic pass/fail results.
  • Decide whether non-deterministic findings should even be mixed into the same report as deterministic ones, or need a separate advisory-only stream.
  • Consider whether this is worth doing before or after UAT ever runs in any automated (even manually-triggered workflow_dispatch) capacity — see TESTING.md's CI section for why UAT isn't in CI today.

Scope

Research/design spike first — don't commit to a format until the calibration question above has an answer. observations[] (non-blocking, informational findings) likely need different treatment than assertions[] (pass/fail) in either output format.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions