Skip to content

Commit 11e0a1f

Browse files
authored
feat: add WebView accessibility lab (#1397)
* feat: add WebView accessibility lab * refactor: tighten iOS snapshot presentation rules * fix: preserve semantic WebView containers
1 parent 5507a08 commit 11e0a1f

22 files changed

Lines changed: 735 additions & 72 deletions

File tree

CONTEXT.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@
8181
- Settled observation: opt-in (`--settle`) post-action payload on press/click/fill/longpress — the quiet-window stable loop re-captures until the UI settles, and the response carries the diff vs the pre-action tree (changed lines only, added lines with fresh refs, `refsGeneration` when the settled tree was stored). Best-effort: never fails the action; `settled: false` plus a hint on never-quiet content.
8282
- Snapshot capture plan: per-strategy ordered chain of iOS snapshot capture backends (recursive tree, query sweep, private AX) run by one plan runner under a shared wall-clock budget; recovery ordering is declared data, never a per-call-site branch.
8383
- Snapshot quality verdict: structured outcome (state, backend, reason code, effective depth, collapsed leaves) computed once by the plan runner and shipped with every planned snapshot payload; the daemon and CLI render it instead of re-deriving degradation from node shapes.
84+
- iOS WebView semantic presentation: the interactive snapshot projection that recognizes XCTest's typed `WebView` root and WebKit's `Other -> StaticText` wrapper pairs. It keeps raw diagnostics unchanged, presents ordinary wrapper text as `StaticText`, and presents wrappers carrying WebKit's numeric HTML heading level as `Heading`.
8485
- AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot failure such as `kAXErrorIllegalArgument` marks the cached `XCUIApplication` target handle suspect. The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the next command reacquire the app through normal activation.
8586
- Resolution disclosure (ADR 0012 decision 2): additive `resolution` field on press/click/fill/longpress responses discloses how the acting path resolved its target — `runtime`/`unique` or `runtime`/`disambiguated` (with `matchCount`/`winnerDiagnostic`/`tiebreak`/up-to-5 `alternatives`) on the daemon tree, `ref`/`exact` for a resolved `@ref` (runtime-ref and native-ref), `ref`/`label-fallback` when runtime-ref recovered a stale `@ref` via its recorded trailing label, or `direct-ios`/`not-observed` on the XCTest fast path; absent entirely on the coordinate path and on dispatches whose runner actually executed the maestro non-hittable coordinate fallback (permission alone keeps the direct path's `not-observed`). Pre-action diagnostics only: `winnerDiagnostic`/`alternatives` entries carry an opaque, non-`@` `diagnosticRef` that is never ref-issued, never MCP-pinned, and cannot be reused as an `@ref` target — a fresh snapshot/find is required before acting on an alternative.
8687

apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,21 +57,23 @@ extension RunnerTests {
5757
case .collectionView: return "CollectionView"
5858
case .table: return "Table"
5959
case .scrollView: return "ScrollView"
60+
case .toolbar: return "Toolbar"
6061
case .searchField: return "SearchField"
6162
case .segmentedControl: return "SegmentedControl"
6263
case .stepper: return "Stepper"
6364
case .picker: return "Picker"
65+
case .activityIndicator: return "ActivityIndicator"
66+
case .progressIndicator: return "ProgressIndicator"
6467
case .checkBox: return "CheckBox"
6568
case .menuItem: return "MenuItem"
69+
case .webView: return "WebView"
6670
case .other: return "Other"
6771
default:
6872
switch type.rawValue {
6973
case 19:
7074
return "Keyboard"
7175
case 20:
7276
return "Key"
73-
case 24:
74-
return "SearchField"
7577
default:
7678
return "Element(\(type.rawValue))"
7779
}

apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests.swift

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,8 @@ final class RunnerTests: XCTestCase {
131131
.tabBar,
132132
.textField,
133133
.secureTextField,
134-
.textView
134+
.textView,
135+
.webView
135136
]
136137
// Keep blocker actions narrow to avoid false positives from generic hittable containers.
137138
let actionableTypes: Set<XCUIElement.ElementType> = [

examples/test-app/README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,17 @@ It is intentionally small, but each surface is dense with durable accessibility
1717
- `Product detail`: back navigation, quantity stepper, multiline notes, save action
1818
- `Checkout form`: required-field validation, fill vs type, checkbox state, choice groups, keyboard dismiss, success summary
1919
- `Settings`: switch rows, accordion content, loading and error states, retry flow, destructive-confirm modal
20+
- `WebView accessibility`: a deterministic semantic fixture plus live websites with varied HTML for native accessibility snapshot verification
2021

2122
Navigation uses Expo Router native bottom tabs, so the tab bar itself is also part of the test surface.
2223

24+
The deterministic WebView fixture is the stable accessibility oracle. On iOS, an interactive
25+
snapshot should expose its root as `webview`, both titles as `heading`, paragraph and label content
26+
as `text`, and the form controls as `text-field`, `switch`, and `button`. The live-site buttons are
27+
exploratory smoke coverage for real-world WebKit trees, not stable assertion targets.
28+
Use an unscoped snapshot for this oracle: XCTest can detach a scoped WebKit document subtree from
29+
its `WebView` ancestor, leaving insufficient evidence for safe semantic projection.
30+
2331
## Coverage map
2432

2533
These are the main case families this app can support without adding more screens:

examples/test-app/app/(tabs)/settings.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ export default function SettingsRoute() {
1717
notificationsEnabled={state.notificationsEnabled}
1818
onOpenAccessorySetup={() => router.push('/accessory-setup')}
1919
onOpenInertSurface={() => router.push('/inert')}
20+
onOpenWebViewLab={() => router.push('/webview')}
2021
onConfirmReset={state.resetLabState}
2122
onLoadDiagnostics={state.loadDiagnostics}
2223
onRetryDiagnostics={state.retryDiagnostics}

examples/test-app/app/_layout.tsx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ function RootLayoutContent() {
2626
<Stack.Screen name="accessory-setup" />
2727
<Stack.Screen name="inert" options={{ presentation: 'fullScreenModal' }} />
2828
<Stack.Screen name="product/[productId]" />
29+
<Stack.Screen name="webview" />
2930
</Stack>
3031
{toastMessage ? <ToastViewport message={toastMessage} /> : null}
3132
</ThemeProvider>

examples/test-app/app/webview.tsx

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
import { useRouter } from 'expo-router';
2+
3+
import { WebViewScreen } from '../src/screens/WebViewScreen';
4+
5+
export default function WebViewRoute() {
6+
const router = useRouter();
7+
8+
return <WebViewScreen onClose={() => router.back()} />;
9+
}

examples/test-app/package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@
2626
"react-native-gesture-handler": "^2.31.2",
2727
"react-native-safe-area-context": "~5.7.0",
2828
"react-native-screens": "~4.25.2",
29-
"react-native-web": "^0.21.2"
29+
"react-native-web": "^0.21.2",
30+
"react-native-webview": "13.16.1"
3031
},
3132
"devDependencies": {
3233
"@expo/fingerprint": "^0.19.4",

0 commit comments

Comments
 (0)