test: E2E confidence suite (iOS/Android scripted + agent evals)

.agents/skills/release/SKILL.md

@@ -24,6 +24,16 @@ Run these checks. Abort with a clear error if any fails.
    ```
    If it exists, verify it's clean and pull latest.
 7. Android bridge toolchain: `scripts/build-bridge.sh --check` succeeds.
+7.5. **E2E confidence suites are green** (see `docs/ai/xxxx-e2e-confidence-suite/`):
+   ```bash
+   make e2e            # iOS + Android scripted E2E vs Playground, in sequence
+   ```
+   (`make e2e-ios` / `make e2e-android` run one platform each.) A full green
+   iOS pass alone is ~15 min, so budget ~20+ min. It prints a full pass/fail
+   map and keeps going past failures. Any red suite blocks the release unless
+   the user explicitly waives it at Step 3.
+   Optional but recommended when skill prose changed this release (real
+   `claude -p` cost): `make eval ARGS="-y -t quick"`.
 8. Signing + notarization readiness:
    ```bash
    security find-identity -v -p codesigning | grep -F "NAVER Japan K.K. (GFPYJQXRSN)"

.gitignore

@@ -66,6 +66,15 @@ bridge/*.iml
 bridge/app/*.iml
 bridge/captures/
 
+# Android playground fixture (standalone Gradle project under Playgrounds/).
+Playgrounds/Android/.gradle/
+Playgrounds/Android/build/
+Playgrounds/Android/app/build/
+Playgrounds/Android/local.properties
+Playgrounds/Android/.idea/
+Playgrounds/Android/**/*.iml
+Playgrounds/Android/captures/
+
 # The bridge APK is a build output, not a source artifact. The Swift
 # `AndroidBackend` target loads it from `Sources/AndroidBackend/Resources/`
 # at runtime; populate that path locally via `scripts/build-bridge.sh`.
@@ -85,3 +94,10 @@ Sources/SimUse/Resources/skills/sim-use/
 Sources/SimUse/Resources/viewer/*
 !Sources/SimUse/Resources/viewer/.gitkeep
 *.profraw
+
+# Agent-eval run artifacts (reports, transcripts, screenshots).
+/e2e/agent-evals/reports/
+
+# Python bytecode from eval harness.
+__pycache__/
+*.pyc

AGENTS.md

@@ -35,6 +35,21 @@ swift test --filter TapTests     # run a single test suite
 
 When [xcsift](https://github.com/ldomaradzki/xcsift) is installed (`brew install xcsift` — optional, never required), `make build` / `make test` condense swift output into a TOON summary plus a coverage report. `SIM_USE_XCSIFT=0 make test` forces plain swift output.
 
+### End-to-end tests (live device)
+
+```bash
+make e2e            # BOTH iOS + Android in sequence (needs a booted sim AND an emulator)
+make e2e-ios        # iOS only — booted simulator + Playground fixture
+make e2e-android    # Android only — reachable device/emulator + Playground fixture
+make eval           # agent evals (real `claude -p` cost; prompts before running)
+```
+
+E2E suites compile always but skip unless `SIM_USE_E2E=1` (iOS) / `SIM_USE_E2E_ANDROID=1` (Android) is set — `make test` never touches a device, which is why CI needs no simulator. The runners set those vars for you.
+
+**Budget the time: a full green `make e2e-ios` run is ~15 minutes.** The iOS suites drive real HID gestures and wait on simulator animations/keyboard settling, so per-suite waits dominate — this is expected, not a hang. `make e2e` (both platforms) is ~20+ min. When you only touched one platform, run just that platform's target. The runners keep going past a failed suite and print a full pass/fail map at the end, so read the summary rather than assuming the first red aborted the rest.
+
+Agent-facing behaviour (the bundled skill) has its own natural-language eval layer — see `e2e/agent-evals/README.md` and `docs/ai/xxxx-e2e-confidence-suite/`.
+
 ### Verifying a change
 
 After any non-trivial change, at minimum:

CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Android Playground fixture app (`Playgrounds/Android`, `com.linecorp.simuse.playground`) and Android device E2E suites (AndroidTap/SwipeScroll/Type/KeyboardState/MultiTouch/Button/DescribeUI) gated by `SIM_USE_E2E_ANDROID=1` + `ANDROID_SERIAL`. `make e2e-android` builds the APK, installs it, runs `sim-use android init`, and executes the suites one-by-one with a pass/fail summary (`scripts/build-playground-android.sh`, `scripts/test-runner-android.sh`).
+- iOS Playground gains three screens with matching E2E suites: `paste-test` + PasteTests (Cmd+V and `--via-menu` paths, unicode, `--replace`, Allow-Paste alert handling), `orientation-test` + OrientationTests (AX-selector taps land correctly after programmatic rotation — exercises orientation self-calibration), and `permissions-test` + PermissionAlertTests (real location permission alert raised, classified via the `App: SpringBoard` header, dismissed through sim-use on both allow and deny paths).
+- Agent-eval harness (`e2e/agent-evals/`): natural-language cases executed by a headless `claude -p` agent using the bundled skill (`skills/sim-use/`) against the Playground apps, judged by deterministic device-side post-condition checks. Catches skill-prose drift and verb-steering regressions the scripted suites cannot (e.g. `paste` vs US-ASCII `type`, Android paste-denied fallback).
+- `make eval` / `pnpm eval` wrappers that check the environment, estimate the (real `claude -p`) cost, and confirm before running the agent evals.
+- E2E confidence-suite design note under `docs/ai/`; the release skill's pre-flight now requires a green `make e2e` run.
+
+### Changed
+
+- `make e2e` now runs both the iOS and Android E2E suites in sequence (continuing past a platform failure and failing if either did); the iOS-only target is `make e2e-ios`, Android-only stays `make e2e-android`.
+- `scripts/test-runner.sh` keeps running after a failed suite and prints a full pass/fail map at the end (a release gate needs the whole picture, not the first crash). The hardcoded suite list gained the missing `KeyboardStateTests` and the new suites, and the misnamed `StreamVideoDebugTests` entry — which silently matched zero tests — now points at the real `StreamVideoDebugTest` suite.
+
+### Fixed
+
+- Four E2E suites (KeyTests, KeyComboTests, KeySequenceTests, StreamVideoTests) still invoked the pre-0.5.x top-level verb forms and had failed ever since the five iOS-only verbs moved under the `ios` namespace; they now call `sim-use ios <verb>`.
+- KeyComboTests' Cmd+A test asserted a cleared text field reads nil/empty — an empty `UITextField` exposes its placeholder as the accessibility value, so the assertion could never pass. It now accepts the placeholder form.
+
 ## [0.10.0] - 2026-07-09
 
 ### Added

CONTRIBUTING.md

@@ -72,13 +72,21 @@ orientation.
 ## Testing
 
 ```bash
-make test       # unit tests via swift test
-make e2e        # end-to-end tests against a booted simulator
+make test          # unit tests via swift test
+make e2e           # both iOS + Android E2E in sequence (~20+ min; needs sim AND emulator)
+make e2e-ios       # iOS E2E against a booted simulator (~15 min for a full green run)
+make e2e-android   # Android E2E against an emulator/device
 ```
 
 For the full build-and-test harness (simulator setup, the SimUsePlayground app,
 test-plan execution), use `./test-runner.sh` (run with `--help` for options
-such as build-only / test-only modes and verbose output).
+such as build-only / test-only modes and verbose output). The Android
+counterpart is `./scripts/test-runner-android.sh`, driving the
+`Playgrounds/Android` fixture app.
+
+Agent-facing behavior (the bundled skill under `skills/sim-use/`) has its own
+natural-language eval layer under `e2e/agent-evals/` — see its README. Run it
+when you change skill prose or agent-visible output.
 
 Please add or update tests for any behavior change, and make sure the suite
 passes locally before opening a pull request.
@@ -100,7 +108,8 @@ passes locally before opening a pull request.
 
 1. Fork the repository and create a topic branch.
 2. Make your change with tests, signed off per the DCO section above.
-3. Run `make test` (and `make e2e` if your change touches device behavior).
+3. Run `make test` (and `make e2e-ios` / `make e2e-android` — or `make e2e`
+   for both — if your change touches device behavior).
 4. Open a pull request describing the motivation and the change. Link any
    related issue.
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help build test e2e clean viewer sync-skills
+.PHONY: help build test e2e e2e-ios e2e-android eval clean viewer sync-skills
 
 # pipefail below needs bash; macOS /bin/sh is bash-in-posix-mode but
 # being explicit costs nothing.
@@ -28,7 +28,10 @@ help:
 	@echo "  make build   Build sim-use"
 	@echo "  make viewer  Rebuild the Viewer SPA into Sources/SimUse/Resources/viewer/"
 	@echo "  make test    Run unit tests (no simulator needed)"
-	@echo "  make e2e     Run end-to-end tests on a booted simulator"
+	@echo "  make e2e     Run BOTH iOS + Android E2E suites in sequence (~15 min iOS alone)"
+	@echo "  make e2e-ios      Run iOS E2E tests on a booted simulator (~15 min for a full green run)"
+	@echo "  make e2e-android  Run Android E2E tests on a connected device/emulator"
+	@echo "  make eval    Run agent evals (real \`claude -p\` cost; prompts first)"
 	@echo "  make clean   Clean Swift build artifacts"
 
 # The bundled skill lives in skills/sim-use (source of truth) and is
@@ -54,8 +57,36 @@ viewer:
 test: sync-skills
 	@$(call run_swift,swift test --enable-code-coverage,--coverage)
 
+# Run both platforms in sequence (iOS then Android), continuing past a
+# platform failure so you get the full picture, and failing if either did.
+# Needs both a booted iOS simulator and a reachable Android device/emulator;
+# for one platform only, use e2e-ios / e2e-android. A full green iOS pass
+# alone is ~15 min, so budget ~20+ min for the combined run.
 e2e:
+	@fail=0; \
+	echo "== iOS E2E =="; ./scripts/test-runner.sh || fail=1; \
+	echo "== Android E2E =="; ./scripts/test-runner-android.sh || fail=1; \
+	if [ $$fail -ne 0 ]; then echo "❌ e2e: one or more platforms failed"; exit 1; fi; \
+	echo "✅ e2e: iOS + Android green"
+
+# iOS simulator E2E: builds the CLI + Playground fixture, installs it on the
+# booted simulator, and runs the iOS suites. A full green run is ~15 min
+# (each HID-driven suite waits on real simulator gestures/animations).
+e2e-ios:
 	./scripts/test-runner.sh
 
+# Android device E2E: builds the CLI + playground fixture, installs it on
+# ANDROID_SERIAL (default emulator-5554), and runs the Android suites.
+e2e-android:
+	./scripts/test-runner-android.sh
+
+# Agent evals: a headless `claude -p` drives the bundled skill against the
+# Playground apps. Each case makes real API calls, so the wrapper checks the
+# environment and prompts for confirmation before spending anything. Pass
+# options through env vars, e.g. `make eval PLATFORM=ios TAGS=release`, or
+# `make eval ARGS="-p android -- --cases oss-android-tap-three-times"`.
+eval:
+	@./scripts/eval.sh $(ARGS)
+
 clean:
 	swift package clean

Playgrounds/Android/app/build.gradle.kts

@@ -0,0 +1,56 @@
+plugins {
+    id("com.android.application")
+    id("org.jetbrains.kotlin.android")
+}
+
+// Deterministic-UI test fixture for the Android E2E suites. Mirrors
+// :app's SDK / JDK conventions (compileSdk 35, minSdk 30, JDK 17) but
+// is intentionally dependency-light: classic Views, no Compose, so
+// every screen surfaces stable `android:id` short-names that sim-use
+// exposes as `#<id>` selectors.
+android {
+    namespace = "com.linecorp.simuse.playground"
+    compileSdk = 35
+
+    defaultConfig {
+        applicationId = "com.linecorp.simuse.playground"
+        minSdk = 30
+        targetSdk = 35
+        versionCode = 1
+        versionName = "0.1.0"
+    }
+
+    buildTypes {
+        getByName("debug") {
+            isMinifyEnabled = false
+        }
+        getByName("release") {
+            isMinifyEnabled = false
+            // Debug-signed by intent — same posture as :app. This APK is
+            // a developer-only test fixture installed via `adb install`
+            // onto an operator-owned emulator/device; it is never
+            // distributed through any consumer channel.
+            signingConfig = signingConfigs.getByName("debug")
+        }
+    }
+
+    compileOptions {
+        sourceCompatibility = JavaVersion.VERSION_17
+        targetCompatibility = JavaVersion.VERSION_17
+    }
+
+    kotlinOptions {
+        jvmTarget = "17"
+    }
+}
+
+dependencies {
+    implementation("androidx.core:core-ktx:1.13.1")
+    // RecyclerView so the scroll screen exposes a Tier-1 collection
+    // container: sim-use's Android list detector only assigns `#N` list
+    // aliases to RecyclerView / ListView / GridView (or nodes carrying
+    // collectionInfo), and `--include-offscreen` keys off the
+    // recycled-but-attached off-screen cells RecyclerView leaves in the
+    // a11y tree.
+    implementation("androidx.recyclerview:recyclerview:1.3.2")
+}

Playgrounds/Android/app/src/main/AndroidManifest.xml

@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    xmlns:tools="http://schemas.android.com/tools">
+
+    <application
+        android:label="@string/app_name"
+        android:icon="@android:drawable/star_big_on"
+        android:allowBackup="false"
+        android:theme="@style/Theme.SimusePlayground"
+        tools:ignore="MissingApplicationIcon">
+
+        <!--
+          Exported + LAUNCHER so the E2E suites can drive it two ways:
+          `adb shell am start -n .../.MainActivity -e screen <name>` to
+          jump straight to a screen, and the launcher icon for manual
+          poking. singleTop so a repeat `am start` re-delivers the
+          intent to the live instance via onNewIntent instead of
+          stacking activities.
+        -->
+        <activity
+            android:name=".MainActivity"
+            android:exported="true"
+            android:launchMode="singleTop"
+            android:windowSoftInputMode="stateHidden|adjustResize">
+            <intent-filter>
+                <action android:name="android.intent.action.MAIN" />
+                <category android:name="android.intent.category.LAUNCHER" />
+            </intent-filter>
+        </activity>
+    </application>
+</manifest>