Remote UI Layout Analysis

Remote UI Layout Analysis

中文

The remote layout inspector converts uiautomator dump XML from the target device into semantic nodes and maps them onto the remote video overlay. It analyzes the accessibility tree, not screenshot pixels.

Authoritative code

• feature/remote/presentation/RemoteUiLayoutParser.kt: secure XML parsing, classification, labels, filtering, and deduplication.
• feature/remote/model/RemoteUiLayoutModels.kt: bounds, node, label-source, and snapshot models.
• feature/remote/ui/internal/RemoteLayoutInspectorUi.kt: coordinate mapping and overlay rendering.
• RemoteUiLayoutParserTest.kt: parser regression tests.

Data flow

1. Run uiautomator dump on the target device and retrieve its XML.
2. Parse XML with external entities and DTDs disabled.
3. Traverse nodes and capture bounds, class, resource id, text, content description, and interaction/state attributes.
4. Produce INPUT, BUTTON, TEXT, TOGGLE, IMAGE, CONTAINER, or OTHER nodes.
5. Filter invisible, zero-area, system-decoration, and meaningless nodes, then deduplicate by semantic priority.
6. Build a viewport from root bounds or node extents, map nodes onto the current remote image, and render them.

Nodes and state

RemoteUiLayoutNode stores:

• Raw identity: package, class, resource id, text, and content description.
• Geometry: RemoteUiLayoutBounds.
• Interaction attributes: clickable, focusable, scrollable, and enabled.
• State attributes: focused, checkable, checked, selected, password, and visible-to-user.
• Derived information: kind, label, label source, component key, and descendant summaries.

Effective toggle state is resolved in this order: true checked or selected means checked; otherwise explicit negative phrases such as “unchecked” win before positive phrases such as “checked”.

Classification and labels

• EditText, TextInput, and AutoComplete are primarily INPUT.
• CompoundButton, CheckBox, Switch, Toggle, and RadioButton are TOGGLE.
• Button and Chip are BUTTON; Image and Text classes map to IMAGE and TEXT.
• Small interactive Views, Images, or containers may be promoted to custom toggles using checkable state, resource ids, content descriptions, and agreement-text context.
• Labeled clickable containers with button-like geometry may be promoted to custom buttons.
• Labels prefer meaningful text or content descriptions and may humanize a resource id as fallback. Password input content is not exposed.

These heuristics serve general semantic recognition. Before adding an app-specific case, find a generalizable signal and add a test instead of accumulating package names and one-off resource ids in the main path.

Filtering, deduplication, and rendering

• Do not render visible-to-user=false, zero-area, system-bar/keyboard decoration, or pure-noise nodes.
• For overlapping or same-component candidates, keep the node with greater semantic value; inputs, toggles, and buttons outrank ordinary containers.
• A snapshot prefers root XML bounds as its viewport and infers extents from nodes only when root bounds are unavailable.
• The overlay uses colors by kind and renders labels, borders, and applicable state indicators.
• It is a semantic visualization, not a pixel-accurate reproduction of the target app.

Known boundaries

• A custom control that exposes no state may be classified correctly without a reliable checked value.
• Canvas, games, or pages without accessibility semantics expose only limited nodes.
• Resource-id and description heuristics can produce false positives or negatives; tune them using real XML samples and unit tests.
• Viewport, rotation, cropping, or aspect-ratio changes affect coordinate mapping, so parser and renderer must be checked together.

Change rules and tests

1. Reduce a sanitized XML sample to the smallest reproducible fixture.
2. Add or update RemoteUiLayoutParserTest for classification, labels, state, filtering, and deduplication.
3. Confirm existing fixtures do not regress after parser changes.
4. For coordinate or rendering changes, manually verify at least one portrait and one landscape device.
5. Never store accounts, passwords, or complete private page content in logs, fixtures, or the wiki.

Related: Remote Interaction Performance and Floating Menu and Engineering and Verification Rules.