Skip to content

ci: fix the screenshot comment pipeline and post device render captures #154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
l2hyunwoo merged 3 commits into from
Jul 11, 2026
Merged

ci: fix the screenshot comment pipeline and post device render captures #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f75b4aa
ci: upload the roborazzi summary from its real path and survive a mis…
l2hyunwoo Jul 11, 2026
67303ad
ci: post device render captures in the screenshot comment gallery
l2hyunwoo Jul 11, 2026
f6c695d
ci: always show the verified golden screenshots in the PR comment
l2hyunwoo Jul 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 187 additions & 23 deletions .github/workflows/screenshot-comment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,45 +84,100 @@ jobs:
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}

# --- Are there any valid *_compare.png diff images? ---
# Instrumented render captures from the device-screenshots matrix (screenshot-test.yml). One
# artifact per API band; `pattern` + `merge-multiple` folds both into ./device-screenshots.
# continue-on-error: the emulator job may not have run (older producer) or a band may have
# produced nothing -- a missing/empty artifact must NOT fail this job. The device gallery step
# below simply renders no rows when the dir is empty.
- name: Download device screenshot artifacts
continue-on-error: true
uses: actions/download-artifact@v4
with:
pattern: device-screenshots-api*
merge-multiple: true
path: device-screenshots
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}

# The committed goldens, uploaded on every run. continue-on-error: an older producer run (before
# this artifact existed) or a download hiccup must NOT fail the job -- the goldens gallery step
# below simply renders nothing when the dir is empty.
- name: Download screenshot goldens artifact
continue-on-error: true
uses: actions/download-artifact@v4
with:
name: screenshot-goldens
path: goldens
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}

# --- Is there anything to push to the companion branch? ---
# `diff` = valid *_compare.png Roborazzi diff images exist (a regression).
# `device` = instrumented render captures exist (present on every emulator run, not just diffs).
# `golden` = committed goldens downloaded (present on every run -> gallery is always-on).
# `any` = any of the three -- the push + companion branch run when ANY is true.
- id: check
name: Check for diff images
name: Check for images to publish
shell: bash
run: |
mapfile -t files < <(find . -type f -name "*_compare.png")
exist="false"
for f in "${files[@]}"; do
# Roborazzi diff images (only exist on a mismatch).
diff="false"
while IFS= read -r f; do
# Reject any path with characters outside the safe set before we touch git / build URLs.
if [[ "$f" =~ ^[a-zA-Z0-9_./-]+$ ]]; then
exist="true"
break
fi
done
echo "exist=$exist" >> "$GITHUB_OUTPUT"
[[ "$f" =~ ^[a-zA-Z0-9_./-]+$ ]] && { diff="true"; break; }
done < <(find . -type f -name "*_compare.png")

# Device render captures (band_api<SDK>_<case>_{actual,expected}.png) under device-screenshots/.
device="false"
if [[ -d device-screenshots ]]; then
while IFS= read -r f; do
[[ "$f" =~ ^[a-zA-Z0-9_./-]+$ ]] && { device="true"; break; }
done < <(find device-screenshots -type f -name "*.png")
fi

# --- Push diff images to an orphan companion branch (only when there are diffs) ---
# Committed goldens under goldens/.
golden="false"
if [[ -d goldens ]]; then
while IFS= read -r f; do
[[ "$f" =~ ^[a-zA-Z0-9_./-]+$ ]] && { golden="true"; break; }
done < <(find goldens -type f -name "*.png")
fi

any="false"
[[ "$diff" == "true" || "$device" == "true" || "$golden" == "true" ]] && any="true"
{
echo "diff=$diff"
echo "device=$device"
echo "golden=$golden"
echo "any=$any"
} >> "$GITHUB_OUTPUT"
Comment on lines 119 to +153

@coderabbitai coderabbitai Bot Jul 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# 1) Confirm the diff gate and the diff gallery operate on the same file set.
rg -n -e '_compare\.png' -e 'roborazzi/' -e 'device-screenshots' .github/workflows/screenshot-comment.yml

# 2) Confirm Roborazzi actually nests *_compare.png under a 'roborazzi/' path segment
#    (otherwise `grep roborazzi/` in the gallery never matches).
rg -n --iglob '*.gradle*' -e 'roborazzi' -e 'outputDir' -e 'compareOutputDir' -e 'resultsDir'

# 3) Show how device captures are named to validate the regex contract with PR `#152`.
rg -n -e 'band_api' -e 'device-screenshots' -g '!**/build/**'

Repository: skydoves/Cloudy

Length of output: 1422

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant workflow sections around the gate and render logic.
sed -n '100,320p' .github/workflows/screenshot-comment.yml | cat -n

# Show the exact snippets that decide whether rows are emitted for each gallery.
rg -n -A6 -B6 'grep "roborazzi/"|device_gallery|band_api\([0-9]+\)_\(.+\)_(actual|expected)' .github/workflows/screenshot-comment.yml

Repository: skydoves/Cloudy

Length of output: 15902

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Find Roborazzi configuration and screenshot producers.
git ls-files | rg -n '(^|/)(build\.gradle(\.kts)?|settings\.gradle(\.kts)?|.*(gradle|kts|kt|java|xml|yml|yaml)$)' >/tmp/allfiles.txt || true
rg -n --glob '!**/build/**' -e 'roborazzi' -e 'compareOutputDir' -e 'outputDir' -e 'device-screenshots' -e 'band_api' .

# Narrow to likely producer/test files if present.
fd -H -I '.*(Roborazzi|Screenshot|MirageBand).*' .

Repository: skydoves/Cloudy

Length of output: 3166

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read the Roborazzi setup and screenshot test file names.
sed -n '240,340p' cloudy/build.gradle.kts | cat -n
printf '\n---\n'
sed -n '1,220p' cloudy/src/androidHostTest/kotlin/com/skydoves/cloudy/ScreenshotTestSupport.kt | cat -n
printf '\n---\n'
sed -n '1,220p' cloudy/src/androidDeviceTest/kotlin/com/skydoves/cloudy/LiquidGlassScreenshotTest.kt | cat -n
printf '\n---\n'
sed -n '1,220p' cloudy/src/androidDeviceTest/kotlin/com/skydoves/cloudy/SkyBackdropScreenshotTest.kt | cat -n

Repository: skydoves/Cloudy

Length of output: 22772

🌐 Web query:

Roborazzi default compare output directory roborazzi compare png path

💡 Result:

The default output directory for Roborazzi—used for reference images and comparison artifacts—is module/build/outputs/roborazzi [1][2]. You can customize where these files are generated using the following methods: 1. Gradle Configuration: You can set the global output directory in your build.gradle file using the roborazzi.outputDir parameter [3]: roborazzi { outputDir = "src/your/screenshot/folder" } 2. Comparison-Specific Output: For specifically managing the output directory of comparison images (e.g., _compare.png and _actual.png files), you can use the roborazzi.compare.outputDir property (introduced in version 1.36.0) or the outputDirectoryPath option within CompareOptions [4][5]. Note that Roborazzi distinguishes between where reference (golden) images are stored and where comparison artifacts are generated [4]. If you need to separate your workflow (e.g., recording vs. verification), you can dynamically adjust these paths in your Gradle configuration based on the task being executed [5].

Citations:

🌐 Web query:

site:github.com takahirom roborazzi compareOutputDir roborazzi png path default

💡 Result:

In Roborazzi, the default directory for both reference images and comparison results is module/build/outputs/roborazzi [1][2]. Regarding your query about the output path, note that Roborazzi distinguishes between how it handles recorded images (golden snapshots) and comparison results (diffs/actual images). 1. Default Path: By default, screenshots captured via captureRoboImage are often saved relative to the current module directory (e.g., module/image.png), though this depends on your configured filePathStrategy [2][3]. The library's internal comparison artifacts are stored in the module/build/outputs/roborazzi directory [1][2]. 2. Customizing Paths: - To customize the global output directory for Roborazzi (which affects where comparison files and potentially recorded images are stored, depending on your strategy), you can configure it in your build.gradle file: roborazzi { outputDir = file("src/your/custom/path") } [4][2] - To control where recorded images are saved, you can set the file path strategy in your gradle.properties file: roborazzi.record.filePathStrategy=relativePathFromRoborazziContextOutputDirectory [2] When this strategy is enabled, images are saved in the directory specified by roborazzi.outputDir instead of the default module root [2]. 3. Comparison Options: While there is an outputDirectoryPath parameter available within CompareOptions, it is specifically intended for managing where comparison-related output files are generated, rather than overriding the primary golden snapshot path for regular capture methods [5].

Citations:

Align the publish gate with the gallery filters. check.diff and check.device can turn true for files that the gallery steps later drop, so the comment still posts a table header with zero rows. Use the same path/filename predicates in both places (roborazzi/ for diff, band_api<SDK>_<case>_{actual,expected}.png for device).

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/screenshot-comment.yml around lines 106 - 131, Update the
Check for images to publish step’s diff and device detection predicates to match
the gallery filters exactly: only accept Roborazzi files under roborazzi/ for
diff images, and only accept device-screenshots files matching
band_api<SDK>_<case>_{actual,expected}.png for device images. Keep the existing
safe-path validation and GITHUB_OUTPUT values, but ensure check.diff and
check.device cannot become true for files the gallery later excludes.

Source: Linters/SAST tools


# --- Push images to an orphan companion branch (diff images and/or device captures) ---
- id: push
name: Push diffs to companion branch
if: steps.check.outputs.exist == 'true'
name: Push images to companion branch
if: steps.check.outputs.any == 'true'
shell: bash
env:
# Keyed by PR number (integer-guarded above), not head_branch -- two fork PRs can share a
# branch name and would otherwise overwrite each other's companion branch / gallery.
BRANCH_NAME: companion_pr-${{ steps.pr.outputs.number }}
run: |
# Orphan branch: no history, just the diff images for this PR head.
# Orphan branch: no history, just the images for this PR head.
git branch -D "$BRANCH_NAME" || true
git checkout --orphan "$BRANCH_NAME"
git rm -rf . > /dev/null

for f in $(find . -type f -name "*_compare.png"); do
# Roborazzi diff images (roborazzi/...) + device render captures (device-screenshots/...) +
# committed goldens (goldens/...). Same safe-set path filter as everywhere else before
# staging fork-run-produced files.
while IFS= read -r f; do
if [[ "$f" =~ ^[a-zA-Z0-9_./-]+$ ]]; then
git add "$f"
fi
done
done < <(find . -type f \( -name "*_compare.png" -o -path "./device-screenshots/*.png" -o -path "./goldens/*.png" \))
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git commit -m "Add screenshot diff for PR #${{ steps.pr.outputs.number }}"
git commit -m "Add screenshots for PR #${{ steps.pr.outputs.number }}"
git push origin "HEAD:$BRANCH_NAME" -f

# --- Summary report (ALWAYS): parse results-summary.json into a counts table + verdict line ---
Expand All @@ -138,7 +193,10 @@ jobs:
# unzipped HTML cannot be deep-linked; the run page exposes the artifact downloads).
RUN_URL: https://github.com/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }}
run: |
summary_json="$(find screenshot-summary -type f -name 'results-summary.json' 2>/dev/null | head -n1)"
# `|| true`: the artifact is absent whenever the producing run failed before writing the
# summary; under `set -e -o pipefail` a bare failing find kills this step before the
# "summary unavailable" fallback below can run.
summary_json="$(find screenshot-summary -type f -name 'results-summary.json' 2>/dev/null | head -n1 || true)"
delimiter="$(openssl rand -hex 8)"
{
echo "body<<${delimiter}"
Expand Down Expand Up @@ -186,12 +244,44 @@ jobs:
} >> "$GITHUB_OUTPUT"
echo "${delimiter}" >> "$GITHUB_OUTPUT"

# --- Goldens gallery (ALWAYS, whenever goldens were downloaded): the committed reference PNGs each
# spec asserts against, inlined from the companion branch. This is NOT a diff -- it shows the
# current expected state so a reviewer sees the actual screenshots in the PR even on a clean pass
# (the whole point of this section). goldens/ came from our own upload-artifact, but apply the
# same ^[a-zA-Z0-9_./-]+$ safe-set filter as the other galleries before building any URL.
- id: goldens
name: Generate goldens gallery
if: steps.check.outputs.golden == 'true'
shell: bash
env:
# Must match the push step's branch key (PR number, not head_branch).
BRANCH_NAME: companion_pr-${{ steps.pr.outputs.number }}
run: |
files=$(find goldens -type f -name "*.png" | grep -E "^[a-zA-Z0-9_./-]+$" || true)
delimiter="$(openssl rand -hex 8)"
{
echo "body<<${delimiter}"
echo ""
echo "### Screenshots"
echo ""
echo "Committed reference screenshots each spec verifies against (current expected state)."
echo ""
echo "| File name | Image |"
echo "|-----------|-------|"
} >> "$GITHUB_OUTPUT"
for f in $files; do
name=$(basename "$f")
url="https://github.com/${{ github.repository }}/blob/$BRANCH_NAME/$f"
echo "| \`$name\` | ![]($url?raw=true) |" >> "$GITHUB_OUTPUT"
done
echo "${delimiter}" >> "$GITHUB_OUTPUT"

# --- Gallery (ONLY when there are diffs): inline before/after/diff images from companion branch.
# Emits a markdown fragment that the comment step appends below the summary, so a regression
# comment shows BOTH the counts AND the images in one sticky comment.
- id: gallery
name: Generate diff gallery
if: steps.check.outputs.exist == 'true'
if: steps.check.outputs.diff == 'true'
shell: bash
env:
# Must match the push step's branch key (PR number, not head_branch).
Expand Down Expand Up @@ -219,6 +309,77 @@ jobs:
done
echo "${delimiter}" >> "$GITHUB_OUTPUT"

# --- Device render gallery (ALWAYS, whenever captures exist): the actual pixels each API band
# drew on the emulator. This is NOT a regression diff -- it is render evidence, so it shows on
# every run that produced captures, independent of the Roborazzi diff above. One row per (band,
# case): actual, and expected beside it if the spec wrote one. Filenames are
# band_api<SDK>_<case>_{actual,expected}.png (see MirageBandScreenshotTest); they arrive from a
# fork-triggered run's artifact, so parse them as UNTRUSTED -- same ^[a-zA-Z0-9_./-]+$ safe-set
# filter as the diff gallery, and pull SDK/case/kind out with pure bash (no eval).
- id: device_gallery
name: Generate device render gallery
if: steps.check.outputs.device == 'true'
shell: bash
env:
# Must match the push step's branch key (PR number, not head_branch).
BRANCH_NAME: companion_pr-${{ steps.pr.outputs.number }}
run: |
delimiter="$(openssl rand -hex 8)"
{
echo "body<<${delimiter}"
echo ""
echo "### Device rendering"
echo ""
echo "Actual pixels captured on the emulator per API band (API 30 = GLES mirage + legacy blur, API 34 = AGSL + RenderEffect). Render evidence, not a golden diff."
echo ""
echo "| Band | Case | Actual | Expected |"
echo "|------|------|--------|----------|"
} >> "$GITHUB_OUTPUT"

# Collect one logical row per api+case, then emit actual|expected columns for each.
# Assoc arrays keyed by "<api>|<case>"; value is the safe repo-relative path to that PNG.
declare -A actual expected
declare -A seen
while IFS= read -r f; do
# Safe-set gate FIRST (path is fork-run-produced). Skip anything with odd chars.
[[ "$f" =~ ^[a-zA-Z0-9_./-]+$ ]] || continue
bn=$(basename "$f" .png)
# Expect band_api<SDK>_<case>_<kind>. Enforce the shape with a regex; ignore non-matching.
[[ "$bn" =~ ^band_api([0-9]+)_(.+)_(actual|expected)$ ]] || continue
api="${BASH_REMATCH[1]}"
case="${BASH_REMATCH[2]}"
kind="${BASH_REMATCH[3]}"
key="${api}|${case}"
seen["$key"]=1
if [[ "$kind" == "actual" ]]; then
actual["$key"]="$f"
else
expected["$key"]="$f"
fi
done < <(find device-screenshots -type f -name "*.png")

# Stable order: sort the composite keys (api numeric-ish then case).
for key in $(printf '%s\n' "${!seen[@]}" | sort); do
api="${key%%|*}"
case="${key#*|}"
a="${actual[$key]:-}"
e="${expected[$key]:-}"
if [[ -n "$a" ]]; then
aurl="https://github.com/${{ github.repository }}/blob/$BRANCH_NAME/$a?raw=true"
acell="![]($aurl)"
else
acell="_(none)_"
fi
if [[ -n "$e" ]]; then
eurl="https://github.com/${{ github.repository }}/blob/$BRANCH_NAME/$e?raw=true"
ecell="![]($eurl)"
else
ecell="_(none)_"
fi
echo "| API ${api} | \`${case}\` | ${acell} | ${ecell} |" >> "$GITHUB_OUTPUT"
done
echo "${delimiter}" >> "$GITHUB_OUTPUT"

# --- Sticky comment: find the bot's previous comment (if any) by marker text ---
- name: Find existing comment
uses: peter-evans/find-comment@v3
Expand All @@ -228,9 +389,10 @@ jobs:
comment-author: 'github-actions[bot]'
body-includes: '<!-- roborazzi-screenshot-diff -->'

# --- Single sticky comment, ALWAYS posted on pull_request runs: summary + optional gallery.
# steps.gallery.outputs.body is empty on clean runs (the step was skipped), so the comment is
# just the summary; on a regression it appends the inline image gallery.
# --- Single sticky comment, ALWAYS posted on pull_request runs: summary + optional galleries.
# Order: summary -> goldens (always, the committed reference shots) -> diff gallery (regression
# only) -> device gallery (emulator captures only). Each gallery output is empty when its step
# was skipped, so the comment degrades to just the summary and whatever galleries have content.
- name: Comment screenshot report
uses: peter-evans/create-or-update-comment@v4
with:
Expand All @@ -240,7 +402,9 @@ jobs:
body: |
<!-- roborazzi-screenshot-diff -->
${{ steps.summary.outputs.body }}
${{ steps.goldens.outputs.body }}
${{ steps.gallery.outputs.body }}
${{ steps.device_gallery.outputs.body }}

# --- Housekeeping: prune companion_* branches older than 30 days so they don't pile up ---
- name: Cleanup outdated companion branches
Expand Down
18 changes: 17 additions & 1 deletion .github/workflows/screenshot-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,23 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: screenshot-summary
path: cloudy/build/test-results/roborazzi/debug/results-summary.json
# The KMP android host test writes under .../roborazzi/androidHostTest/ (not .../debug/);
# glob the variant segment so a task rename cannot silently drop the artifact again.
path: cloudy/build/test-results/roborazzi/**/results-summary.json
if-no-files-found: warn
retention-days: 7

# The committed goldens themselves (checked in under androidHostTest/assets/screenshots/). verify
# only writes PNGs to build/ on a mismatch, so a clean PR had no images to show; uploading the
# goldens gives the comment workflow an ALWAYS-ON "Screenshots" gallery -- the current state each
# spec asserts against, visible in the PR regardless of diff. Just checked-out files (~8 small
# PNGs), so this is a plain copy with no extra Gradle task.
- name: Upload screenshot goldens
if: always()
uses: actions/upload-artifact@v4
with:
name: screenshot-goldens
path: cloudy/src/androidHostTest/assets/screenshots/*.png
if-no-files-found: warn
retention-days: 7

Expand Down
Loading