Native input rewrite, multi-display/Retina fixes, companion app

.github/workflows/test.yml

@@ -0,0 +1,34 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: macos-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Install bats-core
+        run: brew install bats-core
+
+      - name: Run shell tests
+        run: bats tests/
+
+      - name: Install MCP server dependencies
+        working-directory: mcp-server
+        run: npm ci
+
+      - name: Run MCP server tests
+        working-directory: mcp-server
+        run: npm test
+
+      - name: Install shellcheck
+        run: brew install shellcheck
+
+      - name: Run shellcheck
+        run: shellcheck -x *.sh

.gitignore

@@ -3,3 +3,7 @@
 .DS_Store
 mcp-server/node_modules/
 mcp-server/dist/
+helpers/get-window-id
+helpers/ocr-image
+*.xcuserstate
+xcuserdata/

.swiftlint.yml

@@ -0,0 +1,13 @@
+# Project SwiftLint config.
+# This codebase is coordinate math: x, y, x1, y1, dx, dy, mx (margin-x) etc.
+# are the idiomatic names, so the generic short-identifier rule is relaxed.
+identifier_name:
+  min_length:
+    warning: 1
+    error: 1
+line_length:
+  warning: 300
+  error: 320
+cyclomatic_complexity:
+  warning: 15
+  error: 20

Makefile

@@ -0,0 +1,37 @@
+# iphone-control - build & test targets
+.PHONY: test test-shell test-mcp lint build build-helpers build-mcp
+
+# Run all tests
+test: test-shell test-mcp
+
+# Run bats shell tests
+test-shell:
+	@echo "=== Running shell tests (bats) ==="
+	bats tests/*.bats
+
+# Run MCP server tests
+test-mcp:
+	@echo "=== Running MCP server tests ==="
+	cd mcp-server && npx vitest run
+
+# Lint all shell scripts with shellcheck
+# helpers/common.sh is intentionally excluded as a standalone target: it is a
+# sourced-only library, so checking it in isolation flags every constant as
+# unused. -x pulls it into each script's analysis via source, where real usage
+# is visible.
+lint:
+	@echo "=== Linting shell scripts ==="
+	shellcheck -x *.sh
+
+# Build everything
+build: build-helpers build-mcp
+
+# Compile Swift helpers
+build-helpers:
+	@echo "=== Compiling Swift helpers ==="
+	swiftc -O helpers/get-window-id.swift -o helpers/get-window-id
+
+# Build MCP server TypeScript
+build-mcp:
+	@echo "=== Building MCP server ==="
+	cd mcp-server && npm run build

README.md

@@ -2,7 +2,7 @@
 
 > Control your iPhone from the Mac terminal using iPhone Mirroring. Built for AI agents.
 
-macOS Sequoia introduced [iPhone Mirroring](https://support.apple.com/en-us/105112) — your iPhone screen rendered as a native Mac window. **iphone-control** turns that window into a programmable interface: find it, screenshot it, tap it, swipe it, type into it. All from bash.
+macOS Sequoia introduced [iPhone Mirroring](https://support.apple.com/en-us/105112) - your iPhone screen rendered as a native Mac window. **iphone-control** turns that window into a programmable interface: find it, screenshot it, tap it, swipe it, type into it. All from bash.
 
 The killer use case? Hook it up to a multimodal AI (Claude, GPT-4V) and let the AI operate your phone.
 
@@ -42,23 +42,24 @@ The killer use case? Hook it up to a multimodal AI (Claude, GPT-4V) and let the
 
 - **macOS 15** (Sequoia) or later
 - **iPhone Mirroring** open and connected
-- [`cliclick`](https://github.com/BlueM/cliclick) — `brew install cliclick`
-- **Terminal permissions**: Accessibility + Screen Recording (System Settings → Privacy & Security)
+- **Terminal permissions**: Accessibility + Screen Recording (System Settings > Privacy & Security)
+- **Xcode Command Line Tools** (`xcode-select --install`): the Swift helpers auto-compile on first run
+
+No third-party binaries needed: input is delivered natively via JXA/CoreGraphics.
 
 ## Install
 
 ```bash
 git clone https://github.com/wmehanna/iphone-control.git
 cd iphone-control
-brew install cliclick
 ```
 
 ## Usage
 
 ```bash
 # Locate the iPhone Mirroring window
 ./iphone-control.sh find
-# → {"id": 1234, "x": 100, "y": 200, "width": 393, "height": 852}
+# → {"x":-1080,"y":-49,"width":410,"height":874,"content_x":-1070,"content_y":-40,"content_width":390,"content_height":844,"window_id":1234,"scale":2}
 
 # Capture a screenshot
 ./iphone-control.sh screenshot
@@ -84,11 +85,21 @@ brew install cliclick
 
 | Command | Args | Description |
 |---------|------|-------------|
-| `find` | — | Returns window position + size as JSON |
-| `screenshot` | `[output_path]` | Saves iPhone screen to PNG (default: `/tmp/iphone-screen.png`) |
+| `find` | - | Returns window + content bounds, `window_id` and `scale` as JSON |
+| `screenshot` | `[--native] [output_path]` | Saves iPhone screen to PNG in point dimensions (default: `/tmp/iphone-screen.png`); `--native` keeps pixel resolution |
 | `tap` | `<x> <y>` | Tap at coordinates |
 | `swipe` | `<x1> <y1> <x2> <y2> [ms]` | Swipe between two points |
 | `type` | `"text" [x y]` | Type text, optionally into a tapped field |
+| `home` | - | Go to home screen (View menu) |
+| `app-switcher` | - | Open app switcher (View menu) |
+| `spotlight` | - | Open Spotlight search (View menu) |
+| `open-app` | `"Name" [--spotlight]` | Open app by name (registry-first, Spotlight fallback) |
+| `status` | - | Check iPhone Mirroring connection state |
+| `map-apps` | `[max_pages]` | OCR scan of home screen pages (stdout only) |
+| `registry-scan` | `[max_pages]` | Scan home screen and save to persistent registry |
+| `registry-lookup` | `<name>` | Look up app position from registry |
+| `registry-invalidate` | - | Delete the registry file |
+| `registry-list` | - | Show cached registry contents |
 
 ## Coordinates
 
@@ -98,6 +109,20 @@ All coordinates are **relative to the iPhone screen**, not the Mac desktop.
 - Scripts automatically convert to absolute Mac screen position
 - Run `find` to see the window dimensions for your device
 
+### Multi-display and Retina
+
+The iPhone Mirroring window can sit on **any display**, including ones left of or
+above the primary (where global Mac coordinates are negative) and displays with a
+different Retina scale. Everything is handled automatically:
+
+- Window lookup, taps and swipes use global display coordinates (sign-safe)
+- The capture scale is measured from the actual window capture, so mixed 1x/2x
+  setups and windows straddling two displays resolve correctly
+- Screenshots are normalized to **point dimensions**: 1 image pixel = 1 tap
+  coordinate, regardless of which display the window is on
+- Moving the window between displays just works; the 30s window cache
+  invalidates on any move or resize
+
 ## AI Agent Integration
 
 The scripts are designed to be called by an AI agent in a loop:
@@ -113,9 +138,81 @@ The scripts are designed to be called by an AI agent in a loop:
 
 Works with any tool-using AI that supports image input. The `find` result is cached for 30 seconds, so rapid screenshot→tap→screenshot cycles are fast.
 
+## App Registry
+
+The app registry scans your home screen once and caches every app's name, page, and tap coordinates. Subsequent `open-app` calls use the cached data to navigate directly - no Spotlight delay.
+
+### How it works
+
+```
+open-app "Gmail"
+  ├─ Registry lookup → found on page 2
+  │   ├─ Dimensions match current window → navigate + tap (~0.5s)
+  │   └─ Dimensions mismatch → invalidate registry → Spotlight fallback
+  └─ Not found → Spotlight fallback (~1.5s)
+```
+
+### Quick start
+
+```bash
+# 1. Scan home screen pages (writes ~/.iphone-control/app-registry.json)
+./iphone-control.sh registry-scan
+
+# 2. Open an app (uses registry, falls back to Spotlight)
+./iphone-control.sh open-app "Gmail"
+
+# 3. Force Spotlight (bypass registry)
+./iphone-control.sh open-app "Gmail" --spotlight
+
+# 4. Look up an app's position
+./iphone-control.sh registry-lookup "Settings"
+# → {"name":"Settings","page":1,"x":100,"y":200,"content_width":402,"content_height":874}
+
+# 5. List all cached apps
+./iphone-control.sh registry-list
+
+# 6. Force rescan (e.g. after rearranging apps)
+./iphone-control.sh registry-invalidate
+./iphone-control.sh registry-scan
+```
+
+### Registry file
+
+Stored at `~/.iphone-control/app-registry.json`:
+
+```json
+{
+  "version": 2,
+  "scanned_at": "2026-02-20T16:00:00Z",
+  "total_pages": 4,
+  "content_width": 402,
+  "content_height": 874,
+  "apps": [
+    {"name": "Settings", "page": 1, "x": 100, "y": 200},
+    {"name": "Gmail", "page": 2, "x": 300, "y": 400}
+  ]
+}
+```
+
+### Invalidation
+
+| Trigger | Detection | Action |
+|---------|-----------|--------|
+| Window resized | `content_width`/`content_height` mismatch | Auto-invalidate, Spotlight fallback |
+| Apps rearranged | Manual | Run `registry-scan` to overwrite |
+| Registry missing | Lookup exits 1 | Spotlight fallback |
+| Registry from old version | `version` != 2 (pre point-space coords) | Lookup fails, Spotlight fallback; rescan to upgrade |
+
+### Performance
+
+| Method | Page 1 app | Page 3 app |
+|--------|-----------|-----------|
+| Spotlight (current) | ~1.5s | ~1.5s |
+| Registry | ~0.5s | ~1.1s |
+
 ## MCP Server
 
-An [MCP](https://modelcontextprotocol.io) server wraps the bash scripts so any MCP client (Claude Code, Claude Desktop, etc.) can call them as native tools — including returning screenshots as inline images.
+An [MCP](https://modelcontextprotocol.io) server wraps the bash scripts so any MCP client (Claude Code, Claude Desktop, etc.) can call them as native tools - including returning screenshots as inline images.
 
 ### Setup
 
@@ -143,13 +240,21 @@ Add to `.claude/settings.json`:
 
 | Tool | Params | Returns |
 |------|--------|---------|
-| `find_window` | — | JSON `{id, x, y, width, height}` |
-| `screenshot` | — | PNG image (inline, AI-visible) |
+| `find_window` | - | JSON: window + content bounds, `window_id`, `scale` (global points, negative on secondary displays) |
+| `screenshot` | - | PNG image (inline, AI-visible), point-normalized |
 | `tap` | `x, y` | Confirmation text |
 | `swipe` | `x1, y1, x2, y2, duration_ms?` | Confirmation text |
 | `type_text` | `text, x?, y?` | Confirmation text |
+| `open_app` | `name, method?` | Opens app. `method`: `"auto"` (default, registry-first) or `"spotlight"` |
+| `home` | - | Go to home screen |
+| `app_switcher` | - | Open app switcher |
+| `spotlight` | - | Open Spotlight search |
+| `status` | - | Check iPhone Mirroring connection |
+| `scan_apps` | `max_pages?` | Scans home screen, saves registry, returns full registry JSON |
+| `registry_invalidate` | - | Deletes the cached registry, forcing a rescan |
+| `list_apps` | - | Returns cached registry (no rescan) |
 
-The `screenshot` tool returns the image directly as base64-encoded PNG content, so multimodal AI models can see and analyze the iPhone screen without any file path juggling.
+The `screenshot` tool returns the image directly as base64-encoded PNG content, so multimodal AI models can see and analyze the iPhone screen without any file path juggling. The PNG is point-normalized: its pixel dimensions equal the tap/swipe coordinate space, so a model can tap exactly what it sees on any display or Retina scale.
 
 ## Permissions Setup
 
@@ -163,7 +268,6 @@ Your terminal app needs two permissions. macOS will prompt on first use, or set
 | Problem | Solution |
 |---------|----------|
 | `iPhone Mirroring window not found` | Open the iPhone Mirroring app on your Mac |
-| `cliclick not found` | `brew install cliclick` |
 | Taps land in the wrong spot | Run `find` to refresh window position (it may have moved) |
 | Screenshot is blank or fails | Grant Screen Recording permission to your terminal |
 | Clicks don't register | Grant Accessibility permission to your terminal |
@@ -173,11 +277,14 @@ Your terminal app needs two permissions. macOS will prompt on first use, or set
 
 | Script | Mechanism |
 |--------|-----------|
-| `find-window.sh` | JXA + `CGWindowListCopyWindowInfo` to find the "iPhone Mirroring" window |
-| `screenshot.sh` | `screencapture -R x,y,w,h` to capture just that region |
-| `tap.sh` | Translates relative→absolute coords, then `cliclick c:X,Y` |
-| `swipe.sh` | `cliclick dd:` → 10 interpolated `m:` moves → `du:` for smooth gesture |
-| `type-text.sh` | Optional `tap.sh` call, then `cliclick t:"text"` |
+| `find-window.sh` | Swift helper (`CGWindowListCopyWindowInfo`) + window-ID capture + alpha scan for exact content bounds; measures the Retina scale from the capture itself |
+| `screenshot.sh` | `screencapture -x -l <window_id>` (window ID, works on any display), crops to content, normalizes to point dimensions (`--native` keeps pixel resolution) |
+| `tap.sh` | Translates relative to absolute coords, then native `CGEventPost` mouse down/up |
+| `swipe.sh` | `CGEventPost` drag: mouse down, interpolated moves, mouse up |
+| `type-text.sh` | Optional `tap.sh` call, then System Events `keystroke` via JXA |
+| `open-app.sh` | Registry lookup, page navigation + tap, Spotlight fallback |
+| `registry.sh` | Wraps `map-apps.sh`, persists results to `~/.iphone-control/app-registry.json` |
+| `map-apps.sh` | Swipes through home pages, OCR via Swift helper (native res, converted to points), outputs JSON to stdout |
 
 ## License
 

SKILL.md

@@ -11,9 +11,9 @@ Control your iPhone through macOS iPhone Mirroring. The skill captures the mirro
 
 | Tool | Install | Purpose |
 |------|---------|---------|
-| `cliclick` | `brew install cliclick` | Mouse clicks, drags, typing |
-| `screencapture` | Built-in macOS | Window/region capture |
-| `osascript` | Built-in macOS | Window discovery via JXA |
+| `screencapture` | Built-in macOS | Window capture by window ID |
+| `osascript` | Built-in macOS | JXA: window discovery, CGEvent taps/swipes, typing |
+| `swiftc` | Xcode CLT | Auto-compiles the window/OCR helpers on first run |
 | iPhone Mirroring | macOS 15+ (Sequoia) | Renders iPhone on Mac |
 
 ## Permissions Required
@@ -47,11 +47,11 @@ All commands go through `iphone-control.sh`:
 
 ## Workflow for AI Agent
 
-1. `find` — locate the mirroring window (do once per session)
-2. `screenshot` — capture current state, analyze the image
+1. `find` - locate the mirroring window (do once per session)
+2. `screenshot` - capture current state, analyze the image
 3. Decide action based on what's visible
-4. `tap`/`swipe`/`type` — execute the action
-5. `screenshot` — verify the result
+4. `tap`/`swipe`/`type` - execute the action
+5. `screenshot` - verify the result
 6. Repeat 3-5
 
 ## Coordinate System
@@ -60,13 +60,15 @@ All commands go through `iphone-control.sh`:
 - Coordinates are relative to the iPhone screen, NOT the Mac screen
 - The scripts handle conversion to absolute Mac screen coordinates
 - Typical iPhone screen in mirroring: ~375x812 points (varies by model)
+- Screenshots are normalized to point dimensions: 1 image pixel = 1 tap coordinate
+- Multi-display safe: the window can be on any display, including ones with
+  negative global coordinates (left of / above primary) or a different Retina scale
 
 ## Troubleshooting
 
 | Issue | Fix |
 |-------|-----|
 | "iPhone Mirroring window not found" | Open iPhone Mirroring app on Mac |
-| "cliclick not found" | `brew install cliclick` |
-| Tap lands in wrong spot | Run `find` again — window may have moved |
+| Tap lands in wrong spot | Run `find` again - window may have moved |
 | No screen recording permission | System Settings > Privacy & Security > Screen Recording > add Terminal |
 | No accessibility permission | System Settings > Privacy & Security > Accessibility > add Terminal |