Observed during 2026-05-19 hero-demo session
During SimDrive walkthrough recording on iPhone 17 Pro / iOS 26.3 (real session, captured for launch hero video), a Claude session attempted to type into a password field via:
```
tap_first: {text: "password"}
```
The call resolved to the "Password" label (mark 6 in the observation), not the SecureField (mark 7). The agent typed into nothing, then `Sign In` triggered a downstream force-unwrap crash on an empty password. The agent recovered by retrying with explicit `mark_id: 7`, but the initial mis-resolution was a clear UX bug for any agent doing natural-language tap commands.
Why this matters
The whole pitch of SimDrive is "agents drive iOS naturally." When an agent says "tap password," they almost always mean the input, not the label. The current behavior — return the first textual match by reading order — produces broken automations whenever a UI has paired labels + fields (i.e., almost every form).
Suggested fix (heuristic, no behavior break)
Modify `tap_first(text=...)` resolution rule:
- Build the candidate list as today (all marks whose visible text matches).
- Bias toward interactive elements: TextField, SecureField, Button, Picker, Stepper, Switch, Link — basically anything in the XCUIElement `isHittable && isEnabled` set with a typeable/tappable kind.
- If exactly one interactive candidate, return that.
- If multiple interactive candidates, fall back to current first-match rule.
- If zero interactive candidates (text only exists on labels/static), return the label match.
This is monotonically better — never produces a worse outcome, fixes the common pattern.
Repro
Working test case lives in BugDemo at `/Users/atlas/Documents/simdrive-demo/demo-app/BugDemo/BugDemoApp.swift`:
```swift
VStack(alignment: .leading, spacing: 8) {
Text("Password")
.font(.caption).foregroundColor(.secondary)
SecureField("Password", text: $password)
.textFieldStyle(.roundedBorder)
}
```
Both the label and the SecureField have visible text "Password" — current SoM annotation marks them as separate elements (6 and 7). `tap_first({text: "password"})` should land on 7, not 6.
Related test coverage to add
- `simdrive/tests/test_observe_resolution.py` — assert tap_first prefers SecureField over Text label when both contain matching text
- Multi-label golden fixture using XCUIElement attributes captured from BugDemo source
- Regression test using one of the existing dogfood recordings
Priority
Medium. Doesn't block 1.0.0b1 (the agent recovered via mark_id), but every demo / quickstart will hit this same trap. Worth fixing before meaningful adoption — the "agents drive iOS naturally" pitch undermines itself on first contact.
Discovered by
Chairman's manual hero-demo recording session, INIT-2026-549.
Observed during 2026-05-19 hero-demo session
During SimDrive walkthrough recording on iPhone 17 Pro / iOS 26.3 (real session, captured for launch hero video), a Claude session attempted to type into a password field via:
```
tap_first: {text: "password"}
```
The call resolved to the "Password" label (mark 6 in the observation), not the SecureField (mark 7). The agent typed into nothing, then `Sign In` triggered a downstream force-unwrap crash on an empty password. The agent recovered by retrying with explicit `mark_id: 7`, but the initial mis-resolution was a clear UX bug for any agent doing natural-language tap commands.
Why this matters
The whole pitch of SimDrive is "agents drive iOS naturally." When an agent says "tap password," they almost always mean the input, not the label. The current behavior — return the first textual match by reading order — produces broken automations whenever a UI has paired labels + fields (i.e., almost every form).
Suggested fix (heuristic, no behavior break)
Modify `tap_first(text=...)` resolution rule:
This is monotonically better — never produces a worse outcome, fixes the common pattern.
Repro
Working test case lives in BugDemo at `/Users/atlas/Documents/simdrive-demo/demo-app/BugDemo/BugDemoApp.swift`:
```swift
VStack(alignment: .leading, spacing: 8) {
Text("Password")
.font(.caption).foregroundColor(.secondary)
SecureField("Password", text: $password)
.textFieldStyle(.roundedBorder)
}
```
Both the label and the SecureField have visible text "Password" — current SoM annotation marks them as separate elements (6 and 7). `tap_first({text: "password"})` should land on 7, not 6.
Related test coverage to add
Priority
Medium. Doesn't block 1.0.0b1 (the agent recovered via mark_id), but every demo / quickstart will hit this same trap. Worth fixing before meaningful adoption — the "agents drive iOS naturally" pitch undermines itself on first contact.
Discovered by
Chairman's manual hero-demo recording session, INIT-2026-549.