perf: prune interactive AX snapshots earlier

Author: DjDeveloperr

AGENTS.md

@@ -79,7 +79,7 @@ Private simulator behavior is implemented locally in:
 
 The current repo uses the private boot path, private display bridge, and private accessibility translation bridge directly. The browser streams frames from that bridge, injects touch and keyboard events through the same native session layer, inspects accessibility through `AccessibilityPlatformTranslation`, and renders device chrome from `cli/XCWChromeRenderer.*`.
 CoreSimulator service contexts resolve the active developer directory from `DEVELOPER_DIR`, then `xcode-select -p`, then `/Applications/Xcode.app/Contents/Developer`. The display bridge prefers direct CoreSimulator screen IOSurface callbacks and activates the SimulatorKit offscreen renderable view only if direct callbacks are unavailable.
-Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. Shallow snapshots with `maxDepth <= 2` use the tokenized frontmost application translation directly when it is available, and only run the expensive recovery sweep if frontmost lookup fails, so agent-oriented describe loops avoid launchctl and hit-test recovery overhead. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot.
+Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. Shallow snapshots with `maxDepth <= 2` use the tokenized frontmost application translation directly when it is available, and only run the expensive recovery sweep if frontmost lookup fails, so agent-oriented describe loops avoid launchctl and hit-test recovery overhead. Interactive-only snapshots also prune non-actionable native AX leaves during Objective-C serialization before the Rust-side compacting pass; keep this native pruning conservative so selector taps still retain actionable rows plus their ancestors. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot.
 Physical chrome button support uses DeviceKit `chrome.json` input geometry for browser hit targets. Volume, action, mute, Apple Watch digital crown, Watch side button, and Watch left-side button dispatch through `IndigoHIDMessageForHIDArbitrary` with consumer/telephony/vendor HID usage pairs from the device chrome metadata; home, lock, and app-switcher remain on the existing SimulatorKit button paths. Apple Watch Digital Crown rotation dispatches through `IndigoHIDMessageForDigitalCrownEvent` when SimulatorKit exposes it, with `IndigoHIDMessageForScrollEvent(..., target=0x34)` as the fallback. tvOS simulators do not support direct screen touch; browser/API tap maps to Enter, swipe maps to arrow keys, and the native bridge rejects tvOS touch packets before they reach guest `SimulatorHID`. watchOS/tvOS skip dynamic pointer/mouse service warm-up because those guest runtimes abort on unsupported virtual services. Apple TV and Apple Watch simulators are fixed-orientation devices, so client and server rotation paths must not expose or dispatch device rotation for those families.
 Two-point multi-touch dispatch prefers the current SimulatorKit/Indigo packet constructor and falls back to SimDeck's manual Indigo packet adapter. On Xcode 26 SimulatorKit, the constructor expects pixel-space points and stable two-finger movement requires sending `LeftMouseDown` for both `began` and `moved`, then `LeftMouseUp` for `ended`/`cancelled`; using `LeftMouseDragged` for multi-touch moves only advances one contact in UIKit. Do not coalesce multi-touch move packets in the WebSocket or WebRTC control paths, because gesture recognizers need the intermediate two-contact samples.
 WebKit inspection uses the simulator `webinspectord` Unix socket named `com.apple.webinspectord_sim.socket` and WebKit's binary-plist Remote Inspector selectors. It lists only WebKit content that the runtime exposes as inspectable. For app-owned `WKWebView` on iOS 16.4 and newer, the app must set `isInspectable = true`.

cli/XCWAccessibilityBridge.m

@@ -681,6 +681,52 @@ static id XCWAXJSONValue(id value) {
     return role ?: @"";
 }
 
+static BOOL XCWAXRoleLooksInteractive(NSString *role) {
+    NSString *lowercase = (role ?: @"").lowercaseString;
+    for (NSString *needle in @[
+        @"button",
+        @"cell",
+        @"checkbox",
+        @"collection",
+        @"combobox",
+        @"control",
+        @"link",
+        @"menu",
+        @"picker",
+        @"radio",
+        @"scroll",
+        @"search",
+        @"segmented",
+        @"select",
+        @"slider",
+        @"stepper",
+        @"switch",
+        @"tab",
+        @"table",
+        @"textfield",
+        @"text field",
+        @"textinput",
+        @"text input",
+        @"toggle",
+        @"webview",
+    ]) {
+        if ([lowercase containsString:needle]) {
+            return YES;
+        }
+    }
+    return NO;
+}
+
+static BOOL XCWAXElementLooksInteractive(id element, NSString *role) {
+    if (XCWAXBool(element, "isAccessibilityHidden")) {
+        return NO;
+    }
+    if (XCWAXBool(element, "isAccessibilityFocused")) {
+        return YES;
+    }
+    return XCWAXRoleLooksInteractive(role) || XCWAXRoleLooksInteractive(XCWAXRoleType(role));
+}
+
 static pid_t XCWAXElementPID(id element) {
     id translation = XCWAXObject(element, "translation");
     SEL selector = sel_registerName("pid");
@@ -718,7 +764,7 @@ static pid_t XCWAXElementPID(id element) {
     return values;
 }
 
-static NSMutableDictionary *XCWAXSerializeElement(id element, NSString *token, NSHashTable *visited, NSUInteger depth, NSUInteger maxDepth) {
+static NSMutableDictionary *XCWAXSerializeElementWithOptions(id element, NSString *token, NSHashTable *visited, NSUInteger depth, NSUInteger maxDepth, BOOL interactiveOnly) {
     if (element == nil || depth > maxDepth || [visited containsObject:element]) {
         return nil;
     }
@@ -729,21 +775,33 @@ static pid_t XCWAXElementPID(id element) {
         ((void(*)(id, SEL, id))objc_msgSend)(translation, sel_registerName("setBridgeDelegateToken:"), token);
     }
 
-    NSMutableDictionary *values = [XCWAXDictionaryForElement(element) mutableCopy];
     NSMutableArray *childrenValues = [NSMutableArray array];
     id children = depth < maxDepth ? XCWAXObject(element, "accessibilityChildren") : nil;
     if ([children isKindOfClass:NSArray.class]) {
         for (id child in (NSArray *)children) {
-            NSMutableDictionary *childValues = XCWAXSerializeElement(child, token, visited, depth + 1, maxDepth);
+            NSMutableDictionary *childValues = XCWAXSerializeElementWithOptions(child, token, visited, depth + 1, maxDepth, interactiveOnly);
             if (childValues != nil) {
                 [childrenValues addObject:childValues];
             }
         }
     }
-    values[@"children"] = childrenValues;
+
+    NSString *role = XCWAXObject(element, "accessibilityRole");
+    if (interactiveOnly && childrenValues.count == 0 && !XCWAXElementLooksInteractive(element, role)) {
+        return nil;
+    }
+
+    NSMutableDictionary *values = [XCWAXDictionaryForElement(element) mutableCopy];
+    if (!interactiveOnly || childrenValues.count > 0) {
+        values[@"children"] = childrenValues;
+    }
     return values;
 }
 
+static NSMutableDictionary *XCWAXSerializeElement(id element, NSString *token, NSHashTable *visited, NSUInteger depth, NSUInteger maxDepth) {
+    return XCWAXSerializeElementWithOptions(element, token, visited, depth, maxDepth, NO);
+}
+
 static NSArray<NSValue *> *XCWAXRootRecoveryHitTestPoints(void) {
     NSMutableArray<NSValue *> *points = [NSMutableArray array];
     NSArray<NSNumber *> *xValues = @[@80, @220, @360];
@@ -1511,7 +1569,7 @@ + (nullable NSDictionary *)accessibilitySnapshotForSimulatorUDID:(NSString *)udi
             }
 
             NSHashTable *visited = [NSHashTable hashTableWithOptions:NSPointerFunctionsObjectPointerPersonality];
-            NSMutableDictionary *root = XCWAXSerializeElement(element, token, visited, 0, MIN(maxDepth, XCWAXMaxDepth));
+            NSMutableDictionary *root = XCWAXSerializeElementWithOptions(element, token, visited, 0, MIN(maxDepth, XCWAXMaxDepth), interactiveOnly);
             if (root != nil) {
                 XCWAXApplyRecoveredRootMetadata(root, rootItem);
                 [serializedRootItems addObject:@{

scripts/integration/cli.mjs

@@ -654,6 +654,10 @@ async function runRestControls() {
   await measuredStep(
     "REST accessibility-tree interactive",
     async () => {
+      const fullTree = await httpJson(
+        "GET",
+        `/api/simulators/${simulatorUDID}/accessibility-tree?source=native-ax&maxDepth=8`,
+      );
       const tree = await httpJson(
         "GET",
         `/api/simulators/${simulatorUDID}/accessibility-tree?source=native-ax&maxDepth=8&interactiveOnly=true`,
@@ -662,6 +666,13 @@ async function runRestControls() {
       if (tree.interactiveOnly !== true) {
         throw new Error("interactive tree did not report interactiveOnly=true");
       }
+      const fullCount = countAccessibilityNodes(fullTree);
+      const interactiveCount = countAccessibilityNodes(tree);
+      if (interactiveCount <= 0 || interactiveCount > fullCount) {
+        throw new Error(
+          `interactive tree node count ${interactiveCount} was not a pruned subset of full count ${fullCount}`,
+        );
+      }
     },
     { phase: phaseSetup },
   );
@@ -1911,6 +1922,24 @@ function assertRoots(payload, label) {
   }
 }
 
+function countAccessibilityNodes(snapshot) {
+  const roots = Array.isArray(snapshot?.roots) ? snapshot.roots : [];
+  let count = 0;
+  const visit = (node) => {
+    if (!node || typeof node !== "object") {
+      return;
+    }
+    count += 1;
+    for (const child of Array.isArray(node.children) ? node.children : []) {
+      visit(child);
+    }
+  };
+  for (const root of roots) {
+    visit(root);
+  }
+  return count;
+}
+
 function assertJson(payload, label) {
   if (!payload || typeof payload !== "object") {
     throw new Error(`${label} did not return a JSON object`);

server/src/main.rs

@@ -1837,7 +1837,7 @@ fn daemon_is_healthy(metadata: &DaemonMetadata) -> bool {
 }
 
 fn daemon_matches_launch_options(metadata: &DaemonMetadata, options: &DaemonLaunchOptions) -> bool {
-    metadata.port == options.port
+    daemon_port_matches_launch_options(metadata.port, options.port)
         && metadata.bind == options.bind
         && metadata.advertise_host == options.advertise_host
         && metadata.client_root == options.client_root
@@ -1852,6 +1852,11 @@ fn daemon_matches_launch_options(metadata: &DaemonMetadata, options: &DaemonLaun
         && metadata.local_stream_fps == options.local_stream_fps
 }
 
+fn daemon_port_matches_launch_options(actual: u16, preferred: u16) -> bool {
+    let start = preferred.max(1024);
+    actual >= start && actual < start.saturating_add(200)
+}
+
 fn read_daemon_metadata() -> anyhow::Result<Option<DaemonMetadata>> {
     let path = daemon_metadata_path()?;
     if !path.exists() {
@@ -5875,6 +5880,14 @@ mod tests {
         assert!(!daemon_matches_launch_options(&metadata, &options));
     }
 
+    #[test]
+    fn daemon_launch_options_accept_probed_port() {
+        let metadata = daemon_metadata_for_test(4313, "127.0.0.1", None, None);
+        let options = daemon_launch_options_for_test(4311, "127.0.0.1", None, None);
+
+        assert!(daemon_matches_launch_options(&metadata, &options));
+    }
+
     #[test]
     fn daemon_launch_options_reject_different_bind_or_client() {
         let metadata =