feat(mirror): add Device Mirror host plugin - #149
Open
kitakkun wants to merge 17 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new host-only Device Mirror plugin module under jetwhale-plugins/mirror/host that mirrors Android devices/emulators and booted iOS simulators in the JetWhale host UI, and exposes MCP tools to let an agent capture screenshots and drive basic input.
Changes:
- Introduces a new host plugin module (
:jetwhale-plugins:mirror:host) with Compose UI for mirroring and interactive controls. - Adds MCP tools under
com.kitakkun.jetwhale.mirror.*for listing devices, capturing screenshots, and sending input (tap/swipe/button/text). - Implements device discovery + platform-specific controllers (ADB for Android,
xcrun simctl+ optionalidbfor iOS input), plus plugin packaging resources (manifest + icons).
Reviewed changes
Copilot reviewed 14 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| settings.gradle.kts | Registers the new :jetwhale-plugins:mirror:host module in the Gradle build. |
| jetwhale-plugins/mirror/host/build.gradle.kts | Defines plugin module build, packaging/publishing metadata, and dependencies. |
| jetwhale-plugins/mirror/host/src/main/resources/META-INF/jetwhale/plugin-manifest.json | Declares the plugin ID, factory, version, and icons for host discovery. |
| jetwhale-plugins/mirror/host/src/main/resources/icons/mirror_outlined.svg | Adds inactive icon asset for the plugin. |
| jetwhale-plugins/mirror/host/src/main/resources/icons/mirror_filled.svg | Adds active icon asset for the plugin. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/Shell.kt | Adds external process execution helpers + adb/idb discovery utilities. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/DeviceController.kt | Implements Android/iOS device controllers and device discovery logic. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/MirrorHostPluginFactory.kt | Provides the host plugin + polling loops and registers MCP commands. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/MirrorScreen.kt | Compose UI for selecting devices, showing frames, and sending interactions. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/MirrorMcpCommands.kt | Defines tool prefix + common helpers (OK JSON, screenshot file naming). |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/ListDevicesCommand.kt | MCP command to refresh and return the device list as JSON. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/CaptureScreenshotCommand.kt | MCP command to capture a screenshot to disk and return path + dimensions. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/TapCommand.kt | MCP command to tap on the device in screenshot pixel coordinates. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/SwipeCommand.kt | MCP command to swipe in screenshot pixel coordinates with optional duration. |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/PressButtonCommand.kt | MCP command to press hardware buttons (platform-specific support). |
| jetwhale-plugins/mirror/host/src/main/kotlin/com/kitakkun/jetwhale/plugins/mirror/host/InputTextCommand.kt | MCP command to send text input to the device. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+28
to
+36
| // Drain stderr concurrently so neither pipe can fill up and deadlock the process. | ||
| var stderr = "" | ||
| val stderrThread = Thread { | ||
| stderr = process.errorStream.bufferedReader().readText() | ||
| }.apply { start() } | ||
| val stdout = process.inputStream.readBytes() | ||
| val exitCode = process.waitFor() | ||
| stderrThread.join() | ||
| CommandResult(exitCode = exitCode, stdout = stdout, stderr = stderr) |
Comment on lines
+71
to
+73
| val escaped = text | ||
| .replace(Regex("""([\\'"`$&*()\[\]{}+|<>;?~#!])"""), """\\$1""") | ||
| .replace(" ", "%s") |
Comment on lines
+19
to
+27
| override suspend fun execute(arguments: JetWhaleMcpArguments): String { | ||
| val device = resolveDevice(arguments[deviceId]) | ||
| try { | ||
| device.controller.tap(arguments[x], arguments[y]) | ||
| } catch (e: DeviceControlException) { | ||
| throw JetWhaleMcpArgumentException(e.message ?: "tap failed") | ||
| } | ||
| return okJson() | ||
| } |
Comment on lines
+22
to
+36
| override suspend fun execute(arguments: JetWhaleMcpArguments): String { | ||
| val device = resolveDevice(arguments[deviceId]) | ||
| try { | ||
| device.controller.swipe( | ||
| fromX = arguments[fromX], | ||
| fromY = arguments[fromY], | ||
| toX = arguments[toX], | ||
| toY = arguments[toY], | ||
| durationMillis = arguments[durationMillis] ?: 300, | ||
| ) | ||
| } catch (e: DeviceControlException) { | ||
| throw JetWhaleMcpArgumentException(e.message ?: "swipe failed") | ||
| } | ||
| return okJson() | ||
| } |
Comment on lines
+146
to
+153
| override val mcpCommands: List<JetWhaleMcpCommand> = listOf( | ||
| ListDevicesCommand(refreshDevices = ::refreshDevices, selectedDeviceId = { selectedDeviceId }), | ||
| CaptureScreenshotCommand(resolveDevice = ::resolveDevice), | ||
| TapCommand(resolveDevice = ::resolveDevice), | ||
| SwipeCommand(resolveDevice = ::resolveDevice), | ||
| PressButtonCommand(resolveDevice = ::resolveDevice), | ||
| InputTextCommand(resolveDevice = ::resolveDevice), | ||
| ) |
2 tasks
kitakkun
added
feature
host
kitakkun
force-pushed
the
feature/simulator-mirror-plugin
branch
from
July 20, 2026 07:46
9b83908 to
5db8173
Compare
Mirrors Android emulator/device and booted iOS simulator screens in a host-only plugin. The mirror view supports click-to-tap and drag-to-swipe, plus buttons for screenshots, hardware keys, and text input. MCP tools (listDevices, captureScreenshot, tap, swipe, pressButton, inputText) let an AI agent drive the device in the screenshot pixel coordinate space. Android is driven via adb; iOS screenshots use `xcrun simctl`, and iOS input events require idb, failing with an actionable message when it is not installed.
A fixed 350ms sleep per frame capped mirroring at ~1-2fps on top of the capture round trip. Capture continuously with a minimal gap instead, and back off only after a failed capture.
Text could be typed but not deleted or confirmed. Android maps them to KEYCODE_DEL / KEYCODE_ENTER; iOS sends USB HID usage codes via `idb ui key`. Exposed as control-row buttons and through the existing pressButton MCP tool.
…oring Recording: Android records on-device via `adb shell screenrecord` (SIGINT to finalize the mp4, then pull), iOS via `xcrun simctl io recordVideo` (SIGINT to the local process). Exposed as a Record/Stop button and startRecording/stopRecording MCP tools; one recording at a time. Mirroring now runs one capture loop per connected device and shows all devices side by side. Taps and swipes go to the frame under the cursor (and select that device); the button/text controls target the selected, highlighted device.
Every connected device was captured and PNG-decoded at full rate, which scaled CPU linearly with device count even on static screens. Unselected devices now refresh at 500ms, and a frame whose PNG bytes match the previous capture skips decode and recomposition entirely.
Frame pixels were read during composition, so every captured frame recomposed and re-laid-out the whole screen. Composition now depends only on the frame dimensions (derivedStateOf, changes on rotation at most); pixels are read inside drawBehind, so a new frame invalidates just that pane's draw pass.
- Drain stderr with a coroutine instead of spawning a Thread per command - Escape literal % in Android inputText (input text uses % as escape prefix) - Validate tap/swipe coordinates (>= 0) and swipe duration (> 0) upfront with caller-facing errors - Add unit tests for MCP command forwarding, defaults, validation, and JSON shape
…ering The stacked toolbar rows squeezed the mirror panes into a fraction of the window. Controls now sit in one slim glyph-icon row (with hover tooltips) and the TopAppBar is gone. The Send field is replaced by a direct key-input field: committed characters are forwarded to the selected device as typed, IME composition (e.g. Japanese conversion) stays buffered until committed and is then sent as one string, and Backspace/Enter on an empty buffer are sent as device keys.
Screenshot polling topped out at a few fps. Each device's mirror loop now prefers a raw H.264 stream — `adb exec-out screenrecord` on Android (reopened at its 180s cap), `idb video-stream` on iOS — decoded with FFmpegFrameGrabber, falling back to screenshot polling when streaming is unavailable or fails before producing a frame. Unselected devices still decode but publish only every 15th frame. The recording stop now pkills by output path so it cannot kill the mirror stream's screenrecord. The ffmpeg natives are bundled for the build machine's platform only.
FFmpegFrameGrabber.start() blocks in format probing for as long as the stream process keeps stdout open without writing (e.g. idb without a working companion), leaving the pane stuck on Waiting. A watchdog kills the process if no frame arrives within 7s, unblocking the probe so the loop falls back to screenshot polling.
…blish caps The default ffmpeg input path buffers and probes for seconds, and converting every decoded frame let the pipe back up, so the mirror drifted ever further behind real time. Disable input buffering (nobuffer/low_delay, small probe window) and always drain the decoder while throttling conversion/publication by wall-clock (~30fps selected, ~2fps unselected).
Matches the decode-side publish cap; a faster stream only spends encode/decode CPU on frames that would be dropped anyway.
Sending each committed chunk immediately and clearing the field reset the macOS IME session, making Japanese input impossible. The field now purely buffers (never touching an active composition), sends the whole buffer on Enter, and forwards Enter/Backspace as device keys only when the buffer is empty and no composition is active.
The TextFieldValue-based BasicTextField does not display in-progress IME composition on desktop, so Japanese conversion was invisible while typing. Migrate the key-input field to the TextFieldState overload, which renders the composition inline; the Enter-to-send and empty-buffer device-key behavior is unchanged.
A raw key handler fires before the IME, so it stole the Enter that commits a Japanese conversion and sent the half-composed buffer. Enter now goes through KeyboardOptions(imeAction = Send) + onKeyboardAction, which only fires for an Enter the IME did not consume; only Backspace on an empty, non-composing buffer is still intercepted as a device key.
The SDK's JetWhaleMcpArguments now wraps a JsonObject instead of a Map<String, String>; migrate the command tests accordingly.
kitakkun
force-pushed
the
feature/simulator-mirror-plugin
branch
from
July 28, 2026 10:55
d746ef9 to
bf57f45
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new host-only plugin
jetwhale-plugins/mirror/host(Device Mirror,requiresAgent: false) that mirrors Android emulator/device and booted iOS simulator screens, and lets both the user and an AI agent drive the device.Mirroring
adb exec-out screencap -p(Android) /xcrun simctl io <udid> screenshot(iOS), decoded with Skia and rendered in Compose.adb devices -l+simctl list devices booted -j.Interactive controls
MCP tools (prefix
com.kitakkun.jetwhale.mirror)listDevices,captureScreenshot(returns PNG path + pixel dimensions),tap,swipe,pressButton,inputText— all coordinates share the screenshot pixel space, so an agent can capture, inspect, and act. Implemented with the delegated-property parameter DSL, one command per file.Notes
idb describedensity.adbpath discovery mirrors the host'sAdbUtil(the SDK does not expose host internals to plugins).Test plan
:jetwhale-plugins:mirror:host:buildpasses (also after rebasing onto latest main)plugin-manifest.json, icons, and the factory classrunJetWhaleLocalwith a booted emulator/simulator and exercise mirror + controls