Skip to content

observe: prefer interactive elements (TextField/SecureField/Button) when text matches multiple marks #122

Description

@SyncTekLLC

Observed during 2026-05-19 hero-demo session

During SimDrive walkthrough recording on iPhone 17 Pro / iOS 26.3 (real session, captured for launch hero video), a Claude session attempted to type into a password field via:

```
tap_first: {text: "password"}
```

The call resolved to the "Password" label (mark 6 in the observation), not the SecureField (mark 7). The agent typed into nothing, then `Sign In` triggered a downstream force-unwrap crash on an empty password. The agent recovered by retrying with explicit `mark_id: 7`, but the initial mis-resolution was a clear UX bug for any agent doing natural-language tap commands.

Why this matters

The whole pitch of SimDrive is "agents drive iOS naturally." When an agent says "tap password," they almost always mean the input, not the label. The current behavior — return the first textual match by reading order — produces broken automations whenever a UI has paired labels + fields (i.e., almost every form).

Suggested fix (heuristic, no behavior break)

Modify `tap_first(text=...)` resolution rule:

  1. Build the candidate list as today (all marks whose visible text matches).
  2. Bias toward interactive elements: TextField, SecureField, Button, Picker, Stepper, Switch, Link — basically anything in the XCUIElement `isHittable && isEnabled` set with a typeable/tappable kind.
  3. If exactly one interactive candidate, return that.
  4. If multiple interactive candidates, fall back to current first-match rule.
  5. If zero interactive candidates (text only exists on labels/static), return the label match.

This is monotonically better — never produces a worse outcome, fixes the common pattern.

Repro

Working test case lives in BugDemo at `/Users/atlas/Documents/simdrive-demo/demo-app/BugDemo/BugDemoApp.swift`:

```swift
VStack(alignment: .leading, spacing: 8) {
Text("Password")
.font(.caption).foregroundColor(.secondary)
SecureField("Password", text: $password)
.textFieldStyle(.roundedBorder)
}
```

Both the label and the SecureField have visible text "Password" — current SoM annotation marks them as separate elements (6 and 7). `tap_first({text: "password"})` should land on 7, not 6.

Related test coverage to add

  • `simdrive/tests/test_observe_resolution.py` — assert tap_first prefers SecureField over Text label when both contain matching text
  • Multi-label golden fixture using XCUIElement attributes captured from BugDemo source
  • Regression test using one of the existing dogfood recordings

Priority

Medium. Doesn't block 1.0.0b1 (the agent recovered via mark_id), but every demo / quickstart will hit this same trap. Worth fixing before meaningful adoption — the "agents drive iOS naturally" pitch undermines itself on first contact.

Discovered by

Chairman's manual hero-demo recording session, INIT-2026-549.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions