observe/tap: coordinate-driving into dense catalog UI is brittle — lean on the AX tree as primary, OCR as fallback

[mauricecarrier7]

Context

Dogfooding on the Palace triage-bot campaign, getting into the bot on an unfamiliar screen (iPhone 17, dense catalog grid) was the flakiest part of the run. A coordinate tap on a book cover opened Search + a keyboard tutorial instead of the book, needing a Continue→Cancel recovery before a mark-based tap on the book title worked reliably.

Observation

Two tap modalities, very different reliability:

• Mark/text-based taps (tap resolving a stable_id / text) — reliable; they hit the element the agent named.
• Raw coordinate taps into dense UI — a coin-flip. Overlapping hit targets, off-by-a-row grids, and cover art with no text mean the pixel an agent picks often isn't the control it wanted.

This is somewhat inherent to pixel driving, but simdrive already has the better signal available — it ships get_announcements, perform_accessibility_action, and the host-AX tree walk (the iOS-26 content-group probe from #169/1.0.0b12). The AX tree gives real element identity, role, and frame; OCR gives text-on-pixels.

Proposal

Make the accessibility tree the primary resolution path for tap/observe targeting, with OCR/pixel as the fallback when AX is unavailable (e.g. the opaque-group case) — rather than the two being co-equal. Concretely:

• When a tap target matches an AX element, prefer its AX frame/action over the OCR mark centroid.
• Surface in observe whether each mark is AX-backed vs OCR-only, so an agent (and the confidence band) can weight an AX-backed control higher than a text-on-cover-art guess.
• Relation to observe: prefer interactive elements (TextField/SecureField/Button) when text matches multiple marks #122 (prefer interactive elements when text matches multiple marks): that's the same instinct at the disambiguation layer; this is it at the resolution layer.

Impact

The single biggest reliability win for agent-driven runs on real, content-dense apps — where cover art and custom grids are exactly the surfaces OCR/coordinate driving handles worst.