-
Notifications
You must be signed in to change notification settings - Fork 0
feat: black-box E2E tests (Appium + UiAutomator2) + accessibility instrumentation #23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
63e2742
feat: black-box E2E tests (Appium + UiAutomator2) + accessibility ins…
timbortnik b6f38fa
fix: weekly chart is a 7-day forecast, not 14-day
timbortnik 37741d8
fix(ci): run e2e specs via a script file
timbortnik 5663198
fix(ci): bump setup-java/setup-node to v5 (Node 24)
timbortnik 93b7c0e
i18n: translate the 3 new a11y labels into all 35 locales
timbortnik 02fb0d5
fix(ci): wait for elements + seed a GPS fix so E2E isn't flaky on slo…
timbortnik 2e1a124
ci: capture screenshot + page source on E2E failure
timbortnik 484e02e
ci: capture a screenshot at every step and upload on every run
timbortnik 810e010
test(e2e): verify the light/dark theme switch + expose selection to a11y
timbortnik ab652c1
feat: show the weather-fetch failure reason on the error screen
timbortnik 7b3a2f0
ci(e2e): run both specs in one session (one weather fetch per job)
timbortnik 8cff42d
fix: address PR review (request-scoped fetch error, ARB order, CI scr…
timbortnik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| name: E2E (Appium) | ||
|
|
||
| # Black-box UI tests on an x86_64 emulator. Kept separate from test.yml because | ||
| # it needs KVM + an emulator (slow, ~10-15 min) and builds a different ABI. | ||
| # PR-only + manual to stay within free-tier minutes. | ||
| on: | ||
| pull_request: | ||
| branches: [main] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: e2e-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| e2e: | ||
| name: Appium UiAutomator2 (x86_64 emulator) | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 45 | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
|
|
||
| # ---- Build stage: produce the x86_64 debug APK (Flutter) ---- | ||
| # This is the only Flutter step; it just produces the artifact under test. | ||
| - name: Set up JDK 17 | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| distribution: 'corretto' | ||
| java-version: '17' | ||
| - name: Set up Flutter | ||
| uses: subosito/flutter-action@v2 | ||
| with: | ||
| channel: 'stable' | ||
| flutter-version: '3.44.1' | ||
| cache: true | ||
| - name: Install Flutter dependencies | ||
| run: flutter pub get | ||
| - name: Generate version | ||
| run: ./scripts/generate_version.sh | ||
| - name: Build x86_64 debug APK | ||
| run: flutter build apk --debug --target-platform android-x64 | ||
|
|
||
| # ---- Test stage: pure Node + Appium, consumes the prebuilt APK ---- | ||
| # No Flutter on PATH from here on; the test only needs the APK by path. | ||
| - name: Set up Node | ||
| uses: actions/setup-node@v5 | ||
| with: | ||
| node-version: '20' | ||
| cache: 'npm' | ||
| cache-dependency-path: e2e/package-lock.json | ||
| - name: Install E2E dependencies | ||
| working-directory: e2e | ||
| run: npm ci | ||
| - name: Install UiAutomator2 driver | ||
| working-directory: e2e | ||
| run: npm run driver:install # pinned uiautomator2@4.2.9 (last Appium-2.x driver) | ||
|
|
||
| - name: Enable KVM | ||
| run: | | ||
| echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \ | ||
| | sudo tee /etc/udev/rules.d/99-kvm4all.rules | ||
| sudo udevadm control --reload-rules | ||
| sudo udevadm trigger --name-match=kvm | ||
|
|
||
| - name: AVD cache | ||
| uses: actions/cache@v5 | ||
| id: avd-cache | ||
| with: | ||
| path: | | ||
| ~/.android/avd/* | ||
| ~/.android/adb* | ||
| key: avd-31-x86_64-google_apis | ||
|
|
||
| - name: Create AVD snapshot (on cache miss) | ||
| if: steps.avd-cache.outputs.cache-hit != 'true' | ||
| uses: reactivecircus/android-emulator-runner@v2 | ||
| with: | ||
| api-level: 31 | ||
| arch: x86_64 | ||
| target: google_apis | ||
| force-avd-creation: false | ||
| emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -no-audio -no-boot-anim -camera-back none | ||
| disable-animations: true | ||
| script: echo "AVD created" | ||
|
|
||
| - name: Run E2E specs | ||
| uses: reactivecircus/android-emulator-runner@v2 | ||
| with: | ||
| api-level: 31 | ||
| arch: x86_64 | ||
| target: google_apis | ||
| force-avd-creation: false | ||
| emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -no-audio -no-boot-anim -camera-back none | ||
| disable-animations: true | ||
| # Single-line invocation of a script FILE — android-emulator-runner | ||
| # mangles multi-line inline `script:` blocks. run-ci.sh cd's into e2e/. | ||
| script: bash e2e/run-ci.sh | ||
|
|
||
| - name: Upload E2E artifacts | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: e2e-artifacts | ||
| path: | | ||
| e2e/appium.log | ||
| e2e/artifacts/ | ||
| if-no-files-found: ignore | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # E2E UI tests (Appium + UiAutomator2) | ||
|
|
||
| Black-box UI tests that drive the **installed APK** on an Android emulator/device. | ||
| The harness is pure JavaScript (WebdriverIO + Appium) with **no Flutter | ||
| dependency** — it locates elements through the Android accessibility tree | ||
| (`resource-id` / `content-desc`), which the app populates via Flutter `Semantics` | ||
| (see `lib/a11y_ids.dart`). | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Node 20+ and npm | ||
| - An Android emulator or device, **API ≥ 30**, **x86_64** (matches `make debug`) | ||
| - A debug APK for that ABI — from the repo root: | ||
| ```bash | ||
| make debug # -> build/app/outputs/flutter-apk/app-debug.apk (needs JDK 17) | ||
| ``` | ||
|
|
||
| ## Run locally | ||
|
|
||
| ```bash | ||
| cd e2e | ||
| npm install | ||
| npm run driver:install # installs uiautomator2@4.2.9 into ./.appium | ||
|
|
||
| npm run appium # terminal 1: start the Appium server | ||
| npm test # terminal 2: emulator booted + apk built | ||
| ``` | ||
|
|
||
| Test a different build with `APP_PATH=/abs/path/to.apk npm test`. | ||
|
|
||
| ## Specs | ||
|
|
||
| - `specs/home_happy_path.e2e.js` — launch + core navigation (location/theme sheets). | ||
| - `specs/accessibility.e2e.js` — black-box ADA: every control exposes a | ||
| `content-desc` and is ≥ 48dp. Inline attribution links are size-exempt | ||
| (WCAG 2.5.8 inline-text exception). Contrast is **not** covered black-box (deferred). | ||
|
|
||
| ## Notes | ||
|
|
||
| - **Driver pin:** `uiautomator2@4.2.9` is the last driver compatible with Appium | ||
| 2.x (5.x+ require Appium 3). Bump both together. | ||
| - **Charts** are hybrid-composition PlatformViews with no `resource-id`; they | ||
| expose a native `content-desc` (set in `SvgChartPlatformView.kt`) and are | ||
| located by accessibility-id. | ||
| - Flutter text surfaces as `content-desc`, not the `text` attribute — locate by | ||
| `resourceId` or `description*`, never `.text()`. | ||
| - CI: `.github/workflows/e2e.yml` (PR + manual) builds the x86_64 APK then runs | ||
| these specs on a KVM-accelerated emulator. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| // Mirror of lib/a11y_ids.dart — these string values are the contract between | ||
| // the Flutter app (Semantics identifier -> Android resource-id) and these | ||
| // black-box tests. The test harness imports NO Flutter; keep this in sync by | ||
| // hand. Do not rename a value without updating lib/a11y_ids.dart. | ||
| module.exports = { | ||
| // Home screen | ||
| homeRetryButton: 'home_retry_button', | ||
| homeThemeButton: 'home_theme_button', | ||
| homeLocationSelector: 'home_location_selector', | ||
| homeOpenMeteoLink: 'home_open_meteo_link', | ||
| homeGithubLink: 'home_github_link', | ||
| // Charts are hybrid-composition PlatformViews with no resource-id; they carry | ||
| // a native content-desc instead (locate via accessibility-id if needed). | ||
|
|
||
| // Location picker sheet | ||
| locationSearchField: 'location_search_field', | ||
| locationClearSearch: 'location_clear_search', | ||
| locationGpsTile: 'location_gps_tile', | ||
| locationResultTilePrefix: 'location_result_tile', | ||
| locationRecentTilePrefix: 'location_recent_tile', | ||
|
|
||
| // Theme picker sheet | ||
| themeOptionSystem: 'theme_option_system', | ||
| themeOptionLight: 'theme_option_light', | ||
| themeOptionDark: 'theme_option_dark', | ||
| }; |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: timbortnik/widget
Length of output: 87
🏁 Script executed:
Repository: timbortnik/widget
Length of output: 4478
Pin GitHub Actions to full commit SHAs.
These
uses:entries are tag-pinned (@v*) instead of commit-pinned. That weakens supply-chain guarantees and violates strict action-pinning policy.Also applies to lines 29, 34, 49, 69, 79, 90, 104.
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 24-27: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 24-24: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Source: Linters/SAST tools