diff --git a/.agents/skills/release/SKILL.md b/.agents/skills/release/SKILL.md index 64e8831..41b5495 100644 --- a/.agents/skills/release/SKILL.md +++ b/.agents/skills/release/SKILL.md @@ -24,6 +24,16 @@ Run these checks. Abort with a clear error if any fails. ``` If it exists, verify it's clean and pull latest. 7. Android bridge toolchain: `scripts/build-bridge.sh --check` succeeds. +7.5. **E2E confidence suites are green** (see `docs/ai/xxxx-e2e-confidence-suite/`): + ```bash + make e2e # iOS + Android scripted E2E vs Playground, in sequence + ``` + (`make e2e-ios` / `make e2e-android` run one platform each.) A full green + iOS pass alone is ~15 min, so budget ~20+ min. It prints a full pass/fail + map and keeps going past failures. Any red suite blocks the release unless + the user explicitly waives it at Step 3. + Optional but recommended when skill prose changed this release (real + `claude -p` cost): `make eval ARGS="-y -t quick"`. 8. Signing + notarization readiness: ```bash security find-identity -v -p codesigning | grep -F "NAVER Japan K.K. (GFPYJQXRSN)" diff --git a/.gitignore b/.gitignore index 35c3838..1b9b66e 100644 --- a/.gitignore +++ b/.gitignore @@ -66,6 +66,15 @@ bridge/*.iml bridge/app/*.iml bridge/captures/ +# Android playground fixture (standalone Gradle project under Playgrounds/). +Playgrounds/Android/.gradle/ +Playgrounds/Android/build/ +Playgrounds/Android/app/build/ +Playgrounds/Android/local.properties +Playgrounds/Android/.idea/ +Playgrounds/Android/**/*.iml +Playgrounds/Android/captures/ + # The bridge APK is a build output, not a source artifact. The Swift # `AndroidBackend` target loads it from `Sources/AndroidBackend/Resources/` # at runtime; populate that path locally via `scripts/build-bridge.sh`. @@ -85,3 +94,10 @@ Sources/SimUse/Resources/skills/sim-use/ Sources/SimUse/Resources/viewer/* !Sources/SimUse/Resources/viewer/.gitkeep *.profraw + +# Agent-eval run artifacts (reports, transcripts, screenshots). +/e2e/agent-evals/reports/ + +# Python bytecode from eval harness. +__pycache__/ +*.pyc diff --git a/AGENTS.md b/AGENTS.md index c2759c4..563cd69 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,6 +35,21 @@ swift test --filter TapTests # run a single test suite When [xcsift](https://github.com/ldomaradzki/xcsift) is installed (`brew install xcsift` — optional, never required), `make build` / `make test` condense swift output into a TOON summary plus a coverage report. `SIM_USE_XCSIFT=0 make test` forces plain swift output. +### End-to-end tests (live device) + +```bash +make e2e # BOTH iOS + Android in sequence (needs a booted sim AND an emulator) +make e2e-ios # iOS only — booted simulator + Playground fixture +make e2e-android # Android only — reachable device/emulator + Playground fixture +make eval # agent evals (real `claude -p` cost; prompts before running) +``` + +E2E suites compile always but skip unless `SIM_USE_E2E=1` (iOS) / `SIM_USE_E2E_ANDROID=1` (Android) is set — `make test` never touches a device, which is why CI needs no simulator. The runners set those vars for you. + +**Budget the time: a full green `make e2e-ios` run is ~15 minutes.** The iOS suites drive real HID gestures and wait on simulator animations/keyboard settling, so per-suite waits dominate — this is expected, not a hang. `make e2e` (both platforms) is ~20+ min. When you only touched one platform, run just that platform's target. The runners keep going past a failed suite and print a full pass/fail map at the end, so read the summary rather than assuming the first red aborted the rest. + +Agent-facing behaviour (the bundled skill) has its own natural-language eval layer — see `e2e/agent-evals/README.md` and `docs/ai/xxxx-e2e-confidence-suite/`. + ### Verifying a change After any non-trivial change, at minimum: diff --git a/CHANGELOG.md b/CHANGELOG.md index 15dae00..06849d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Android Playground fixture app (`Playgrounds/Android`, `com.linecorp.simuse.playground`) and Android device E2E suites (AndroidTap/SwipeScroll/Type/KeyboardState/MultiTouch/Button/DescribeUI) gated by `SIM_USE_E2E_ANDROID=1` + `ANDROID_SERIAL`. `make e2e-android` builds the APK, installs it, runs `sim-use android init`, and executes the suites one-by-one with a pass/fail summary (`scripts/build-playground-android.sh`, `scripts/test-runner-android.sh`). +- iOS Playground gains three screens with matching E2E suites: `paste-test` + PasteTests (Cmd+V and `--via-menu` paths, unicode, `--replace`, Allow-Paste alert handling), `orientation-test` + OrientationTests (AX-selector taps land correctly after programmatic rotation — exercises orientation self-calibration), and `permissions-test` + PermissionAlertTests (real location permission alert raised, classified via the `App: SpringBoard` header, dismissed through sim-use on both allow and deny paths). +- Agent-eval harness (`e2e/agent-evals/`): natural-language cases executed by a headless `claude -p` agent using the bundled skill (`skills/sim-use/`) against the Playground apps, judged by deterministic device-side post-condition checks. Catches skill-prose drift and verb-steering regressions the scripted suites cannot (e.g. `paste` vs US-ASCII `type`, Android paste-denied fallback). +- `make eval` / `pnpm eval` wrappers that check the environment, estimate the (real `claude -p`) cost, and confirm before running the agent evals. +- E2E confidence-suite design note under `docs/ai/`; the release skill's pre-flight now requires a green `make e2e` run. + +### Changed + +- `make e2e` now runs both the iOS and Android E2E suites in sequence (continuing past a platform failure and failing if either did); the iOS-only target is `make e2e-ios`, Android-only stays `make e2e-android`. +- `scripts/test-runner.sh` keeps running after a failed suite and prints a full pass/fail map at the end (a release gate needs the whole picture, not the first crash). The hardcoded suite list gained the missing `KeyboardStateTests` and the new suites, and the misnamed `StreamVideoDebugTests` entry — which silently matched zero tests — now points at the real `StreamVideoDebugTest` suite. + +### Fixed + +- Four E2E suites (KeyTests, KeyComboTests, KeySequenceTests, StreamVideoTests) still invoked the pre-0.5.x top-level verb forms and had failed ever since the five iOS-only verbs moved under the `ios` namespace; they now call `sim-use ios `. +- KeyComboTests' Cmd+A test asserted a cleared text field reads nil/empty — an empty `UITextField` exposes its placeholder as the accessibility value, so the assertion could never pass. It now accepts the placeholder form. + ## [0.10.0] - 2026-07-09 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 413c4c8..2b0d8fa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -72,13 +72,21 @@ orientation. ## Testing ```bash -make test # unit tests via swift test -make e2e # end-to-end tests against a booted simulator +make test # unit tests via swift test +make e2e # both iOS + Android E2E in sequence (~20+ min; needs sim AND emulator) +make e2e-ios # iOS E2E against a booted simulator (~15 min for a full green run) +make e2e-android # Android E2E against an emulator/device ``` For the full build-and-test harness (simulator setup, the SimUsePlayground app, test-plan execution), use `./test-runner.sh` (run with `--help` for options -such as build-only / test-only modes and verbose output). +such as build-only / test-only modes and verbose output). The Android +counterpart is `./scripts/test-runner-android.sh`, driving the +`Playgrounds/Android` fixture app. + +Agent-facing behavior (the bundled skill under `skills/sim-use/`) has its own +natural-language eval layer under `e2e/agent-evals/` — see its README. Run it +when you change skill prose or agent-visible output. Please add or update tests for any behavior change, and make sure the suite passes locally before opening a pull request. @@ -100,7 +108,8 @@ passes locally before opening a pull request. 1. Fork the repository and create a topic branch. 2. Make your change with tests, signed off per the DCO section above. -3. Run `make test` (and `make e2e` if your change touches device behavior). +3. Run `make test` (and `make e2e-ios` / `make e2e-android` — or `make e2e` + for both — if your change touches device behavior). 4. Open a pull request describing the motivation and the change. Link any related issue. diff --git a/Makefile b/Makefile index fb8aeb2..a08bfdb 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help build test e2e clean viewer sync-skills +.PHONY: help build test e2e e2e-ios e2e-android eval clean viewer sync-skills # pipefail below needs bash; macOS /bin/sh is bash-in-posix-mode but # being explicit costs nothing. @@ -28,7 +28,10 @@ help: @echo " make build Build sim-use" @echo " make viewer Rebuild the Viewer SPA into Sources/SimUse/Resources/viewer/" @echo " make test Run unit tests (no simulator needed)" - @echo " make e2e Run end-to-end tests on a booted simulator" + @echo " make e2e Run BOTH iOS + Android E2E suites in sequence (~15 min iOS alone)" + @echo " make e2e-ios Run iOS E2E tests on a booted simulator (~15 min for a full green run)" + @echo " make e2e-android Run Android E2E tests on a connected device/emulator" + @echo " make eval Run agent evals (real \`claude -p\` cost; prompts first)" @echo " make clean Clean Swift build artifacts" # The bundled skill lives in skills/sim-use (source of truth) and is @@ -54,8 +57,36 @@ viewer: test: sync-skills @$(call run_swift,swift test --enable-code-coverage,--coverage) +# Run both platforms in sequence (iOS then Android), continuing past a +# platform failure so you get the full picture, and failing if either did. +# Needs both a booted iOS simulator and a reachable Android device/emulator; +# for one platform only, use e2e-ios / e2e-android. A full green iOS pass +# alone is ~15 min, so budget ~20+ min for the combined run. e2e: + @fail=0; \ + echo "== iOS E2E =="; ./scripts/test-runner.sh || fail=1; \ + echo "== Android E2E =="; ./scripts/test-runner-android.sh || fail=1; \ + if [ $$fail -ne 0 ]; then echo "❌ e2e: one or more platforms failed"; exit 1; fi; \ + echo "✅ e2e: iOS + Android green" + +# iOS simulator E2E: builds the CLI + Playground fixture, installs it on the +# booted simulator, and runs the iOS suites. A full green run is ~15 min +# (each HID-driven suite waits on real simulator gestures/animations). +e2e-ios: ./scripts/test-runner.sh +# Android device E2E: builds the CLI + playground fixture, installs it on +# ANDROID_SERIAL (default emulator-5554), and runs the Android suites. +e2e-android: + ./scripts/test-runner-android.sh + +# Agent evals: a headless `claude -p` drives the bundled skill against the +# Playground apps. Each case makes real API calls, so the wrapper checks the +# environment and prompts for confirmation before spending anything. Pass +# options through env vars, e.g. `make eval PLATFORM=ios TAGS=release`, or +# `make eval ARGS="-p android -- --cases oss-android-tap-three-times"`. +eval: + @./scripts/eval.sh $(ARGS) + clean: swift package clean diff --git a/Playgrounds/Android/app/build.gradle.kts b/Playgrounds/Android/app/build.gradle.kts new file mode 100644 index 0000000..7387b84 --- /dev/null +++ b/Playgrounds/Android/app/build.gradle.kts @@ -0,0 +1,56 @@ +plugins { + id("com.android.application") + id("org.jetbrains.kotlin.android") +} + +// Deterministic-UI test fixture for the Android E2E suites. Mirrors +// :app's SDK / JDK conventions (compileSdk 35, minSdk 30, JDK 17) but +// is intentionally dependency-light: classic Views, no Compose, so +// every screen surfaces stable `android:id` short-names that sim-use +// exposes as `#` selectors. +android { + namespace = "com.linecorp.simuse.playground" + compileSdk = 35 + + defaultConfig { + applicationId = "com.linecorp.simuse.playground" + minSdk = 30 + targetSdk = 35 + versionCode = 1 + versionName = "0.1.0" + } + + buildTypes { + getByName("debug") { + isMinifyEnabled = false + } + getByName("release") { + isMinifyEnabled = false + // Debug-signed by intent — same posture as :app. This APK is + // a developer-only test fixture installed via `adb install` + // onto an operator-owned emulator/device; it is never + // distributed through any consumer channel. + signingConfig = signingConfigs.getByName("debug") + } + } + + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } + + kotlinOptions { + jvmTarget = "17" + } +} + +dependencies { + implementation("androidx.core:core-ktx:1.13.1") + // RecyclerView so the scroll screen exposes a Tier-1 collection + // container: sim-use's Android list detector only assigns `#N` list + // aliases to RecyclerView / ListView / GridView (or nodes carrying + // collectionInfo), and `--include-offscreen` keys off the + // recycled-but-attached off-screen cells RecyclerView leaves in the + // a11y tree. + implementation("androidx.recyclerview:recyclerview:1.3.2") +} diff --git a/Playgrounds/Android/app/src/main/AndroidManifest.xml b/Playgrounds/Android/app/src/main/AndroidManifest.xml new file mode 100644 index 0000000..60e8080 --- /dev/null +++ b/Playgrounds/Android/app/src/main/AndroidManifest.xml @@ -0,0 +1,31 @@ + + + + + + + + + + + + + + diff --git a/Playgrounds/Android/app/src/main/java/com/linecorp/simuse/playground/MainActivity.kt b/Playgrounds/Android/app/src/main/java/com/linecorp/simuse/playground/MainActivity.kt new file mode 100644 index 0000000..120e52c --- /dev/null +++ b/Playgrounds/Android/app/src/main/java/com/linecorp/simuse/playground/MainActivity.kt @@ -0,0 +1,382 @@ +// SPDX-License-Identifier: Apache-2.0 +package com.linecorp.simuse.playground + +import android.annotation.SuppressLint +import android.app.Activity +import android.content.Context +import android.content.Intent +import android.os.Bundle +import android.text.Editable +import android.text.TextWatcher +import android.view.Gravity +import android.view.GestureDetector +import android.view.MotionEvent +import android.view.ScaleGestureDetector +import android.view.View +import android.view.ViewGroup +import android.view.inputmethod.InputMethodManager +import android.widget.Button +import android.widget.EditText +import android.widget.TextView +import androidx.recyclerview.widget.LinearLayoutManager +import androidx.recyclerview.widget.RecyclerView +import java.util.Locale +import kotlin.math.abs +import kotlin.math.atan2 +import kotlin.math.hypot + +/** + * Single-activity, deterministic-UI test fixture for the sim-use Android + * E2E suites. Each screen exposes stable `android:id` short-names that + * sim-use surfaces as `#` selectors, and every interaction updates a + * plain-text echo label so a describe-ui read can assert the effect of a + * command. + * + * Screens are selected by intent extra: + * adb shell am start -n com.linecorp.simuse.playground/.MainActivity \ + * -e screen tap-test + * or from the in-app menu (each item id `menu_` with `_` in place + * of the `-` that Android resource names forbid). + */ +class MainActivity : Activity() { + + private lateinit var container: ViewGroup + private var currentScreen: String = SCREEN_MENU + + // Per-screen counters. Reset whenever the owning screen is (re)shown, + // so a fresh `am start` always lands on a clean slate. + private var tapCount = 0 + private var longPressCount = 0 + private var swipeCount = 0 + private var backPressCount = 0 + + // Row ids are pre-declared in res/values/ids.xml so the 100 static + // scroll rows expose `row_1`..`row_100` short-names in describe-ui. + private val rowIds: List by lazy { + (1..ROW_COUNT).map { resources.getIdentifier("row_$it", "id", packageName) } + } + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + setContentView(R.layout.activity_main) + container = findViewById(R.id.screen_container) + render(intent) + } + + override fun onNewIntent(intent: Intent) { + super.onNewIntent(intent) + // Persist the new intent so a singleTop re-launch that targets a + // different screen is honoured on the next render(). + setIntent(intent) + render(intent) + } + + override fun onBackPressed() { + // The button-test screen counts hardware Back presses instead of + // finishing, so `sim-use button back` has an observable effect. + if (currentScreen == SCREEN_BUTTON) { + backPressCount++ + findViewById(R.id.back_press_count)?.text = + getString(R.string.fmt_back_presses, backPressCount) + return + } + @Suppress("DEPRECATION") + super.onBackPressed() + } + + private fun render(intent: Intent?) { + val requested = intent?.getStringExtra(EXTRA_SCREEN) + showScreen(normalize(requested)) + } + + /** Accept the `-` canonical names and the `_` menu-id variants alike. */ + private fun normalize(name: String?): String { + val canonical = name?.trim()?.replace('_', '-') + return when (canonical) { + SCREEN_TAP, SCREEN_SWIPE, SCREEN_SCROLL, + SCREEN_TEXT, SCREEN_MULTI, SCREEN_BUTTON -> canonical + else -> SCREEN_MENU + } + } + + private fun showScreen(name: String) { + currentScreen = name + container.removeAllViews() + val layout = when (name) { + SCREEN_TAP -> R.layout.screen_tap + SCREEN_SWIPE -> R.layout.screen_swipe + SCREEN_SCROLL -> R.layout.screen_scroll + SCREEN_TEXT -> R.layout.screen_text + SCREEN_MULTI -> R.layout.screen_multitouch + SCREEN_BUTTON -> R.layout.screen_button + else -> R.layout.screen_menu + } + val root = layoutInflater.inflate(layout, container, false) + container.addView(root) + when (name) { + SCREEN_MENU -> setupMenu(root) + SCREEN_TAP -> setupTapScreen(root) + SCREEN_SWIPE -> setupSwipeScreen(root) + SCREEN_SCROLL -> setupScrollScreen(root) + SCREEN_TEXT -> setupTextScreen(root) + SCREEN_MULTI -> setupMultiTouchScreen(root) + SCREEN_BUTTON -> setupButtonScreen(root) + } + } + + private fun setupMenu(root: View) { + val screens = mapOf( + R.id.menu_tap_test to SCREEN_TAP, + R.id.menu_swipe_test to SCREEN_SWIPE, + R.id.menu_scroll_test to SCREEN_SCROLL, + R.id.menu_text_input to SCREEN_TEXT, + R.id.menu_multi_touch to SCREEN_MULTI, + R.id.menu_button_test to SCREEN_BUTTON, + ) + for ((id, screen) in screens) { + root.findViewById