Once the UAT tier is otherwise settled (scenarios stable, tooling mature — see the companion "UAT backlog scenarios" issue), explore standards-based result surfacing so findings show up where reviewers already look:
- SARIF (Static Analysis Results Interchange Format) — severity-leveled findings that render natively in GitHub code scanning / IDE integrations.
- JUnit XML — a de facto standard for CI test-result reporting, would let UAT's
report.json output plug into any CI dashboard that already parses JUnit.
Open question (the hard part)
UAT's assertions[] include LLM visual judgments (contrast, occlusion, layering, "looks right"), which are not reproducible run-to-run by design — see TESTING.md "UAT is not run in CI." Before adopting either format:
- Decide how to represent a visual-judgment assertion's severity/confidence in a format designed for deterministic pass/fail results.
- Decide whether non-deterministic findings should even be mixed into the same report as deterministic ones, or need a separate advisory-only stream.
- Consider whether this is worth doing before or after UAT ever runs in any automated (even manually-triggered
workflow_dispatch) capacity — see TESTING.md's CI section for why UAT isn't in CI today.
Scope
Research/design spike first — don't commit to a format until the calibration question above has an answer. observations[] (non-blocking, informational findings) likely need different treatment than assertions[] (pass/fail) in either output format.
Once the UAT tier is otherwise settled (scenarios stable, tooling mature — see the companion "UAT backlog scenarios" issue), explore standards-based result surfacing so findings show up where reviewers already look:
report.jsonoutput plug into any CI dashboard that already parses JUnit.Open question (the hard part)
UAT's
assertions[]include LLM visual judgments (contrast, occlusion, layering, "looks right"), which are not reproducible run-to-run by design — seeTESTING.md"UAT is not run in CI." Before adopting either format:workflow_dispatch) capacity — seeTESTING.md's CI section for why UAT isn't in CI today.Scope
Research/design spike first — don't commit to a format until the calibration question above has an answer.
observations[](non-blocking, informational findings) likely need different treatment thanassertions[](pass/fail) in either output format.