Native input engine, interaction primitives, resident daemon, MCP surface - #3
Merged
Merged
Conversation
Compiled Swift engine (helpers/input-src) replaces per-action bash+osascript spawns for tap/swipe/scroll/drag/key, cutting marginal cost per chained action from ~170ms to ~26ms. New interaction primitives: read (OCR screen dump with coordinates), wait (text/gone/still conditions instead of guessed sleeps), press, double-tap, back, scroll-to, key, clipboard, batch. Fixes bundled in: key.sh and Cmd+1/2/3 now resolve characters via the Mac's active keyboard layout (UCKeyTranslate) instead of a hardcoded US-QWERTY table; screenshot.sh no longer merges sips crop+resample into one call (silently wrong size at 2x scale); focus-restore now waits for activation to land instead of racing the process exit; numeric CLI args are range-checked before any session opens, closing a trapping-conversion crash that could leave the mouse button down and focus stolen. Live device suite (tests/live/) and 40 new bats tests cover the new surface.
`input serve` runs the engine as a unix-socket daemon, auto-started by `input send`: Vision OCR model stays warm, capture goes through ScreenCaptureKit one-shots, stillness detection diffs frames in-process, and one input session is held across commands (idle-closed after 1.5s, so a burst of actions pays the macOS focus-steal once; daemon self-exits after 5min unused). read.sh, wait.sh, screenshot.sh and tap-text.sh use the daemon when reachable and fall back to their process pipelines otherwise (ICTL_NO_DAEMON=1 forces the fallback). Window info is re-resolved per daemon command so a moved window never receives stale coordinates. Live suite 39/39 after: screenshot 267->173ms, OCR find 962->368ms, read 1329->495ms, wait-still 1097->718ms, wait-text 3786->2545ms; gesture tests dropped ~1-1.6s each because their internal waits ride the daemon.
tap-verify taps and confirms the result (screen changed by default, or --expect/--gone <text>), retrying once - its first live run already caught a genuinely missed tap and recovered on the retry. Wired into the dispatcher and the live suite (negative case: a no-op tap must be reported as unverified, catching a lying verifier). Both OCR producers (ocr-image helper and the daemon) now drop lines below 0.6 confidence: on phone-UI content Vision scores are bimodal - legitimate text reads 1.0, garbled clipped glyphs ~0.5 (measured "Rottor" misread of a clipped "Battery" at exactly 0.50) - so consumers never see junk text. Output format unchanged. Also validated the whole live suite at Retina scale (window moved to the 2x display, 13/13, coordinates and dimensions exact on both daemon and fallback paths) - previously everything had only ever run at scale=1.
Adds eleven tools the MCP server was missing relative to the CLI: tap_text, tap_verify, read_screen, wait_for, scroll, scroll_to, long_press, double_tap, press_key, go_back and drag_icon. Most notable gap closed: scroll - swipes do not scroll iOS lists through mirroring, so an agent on the old tool set could not scroll at all, read the screen, wait on conditions, or verify taps. Tool descriptions encode the platform rules discovered in testing (swipe vs scroll, Escape not closing context menus, one-slot icon moves, confidence-filtered OCR) so agents inherit them for free.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rebuilds the input path around a compiled Swift engine with a resident daemon mode, adds the interaction primitives agents were missing (read/wait/verify/long-press/double-tap/keys/clipboard/back/scroll-to), fixes non-US keyboard layouts, and exposes the full surface through the MCP server.
Changes
helpers/input-src/): single-process tap/swipe/scroll/drag/key/text with hardware-fidelity events; marginal chained-action cost ~26ms (was ~170ms). Range-checked CLI args so bad input is a clean no-op, never a mid-gesture trap; every exit path restores cursor + focus.input serve/send): warm Vision OCR, ScreenCaptureKit capture, in-process stillness detection, one input session held across command bursts (idle-closed).read/wait/screenshot/tap-textride it automatically with process-pipeline fallback (ICTL_NO_DAEMON=1opt-out).read(OCR + tappable coordinates - the only inspection channel over mirroring),wait text|gone|still,tap-verify(self-checking taps with retry),press,double-tap,key+ clipboard,back,scroll-to,batch,drag-icon --via.screenshot.shdaemon path, pipe-saferead.sh.Testing
tests/live/run-live.sh --bench 3): 42/42 across 3 iterations