Skip to content

Commit a8f23da

Browse files
DjDeveloperrDjDeveloperr
committed
fix: route ios multitouch through service api
1 parent 2a89995 commit a8f23da

15 files changed

Lines changed: 934 additions & 135 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ The current repo uses the private boot path, private display bridge, and private
8181
CoreSimulator service contexts resolve the active developer directory from `DEVELOPER_DIR`, then `xcode-select -p`, then `/Applications/Xcode.app/Contents/Developer`. The display bridge prefers direct CoreSimulator screen IOSurface callbacks and activates the SimulatorKit offscreen renderable view only if direct callbacks are unavailable.
8282
Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot.
8383
Physical chrome button support uses DeviceKit `chrome.json` input geometry for browser hit targets. Volume, action, mute, Apple Watch digital crown, Watch side button, and Watch left-side button dispatch through `IndigoHIDMessageForHIDArbitrary` with consumer/telephony/vendor HID usage pairs from the device chrome metadata; home, lock, and app-switcher remain on the existing SimulatorKit button paths. Apple Watch Digital Crown rotation dispatches through `IndigoHIDMessageForDigitalCrownEvent` when SimulatorKit exposes it, with `IndigoHIDMessageForScrollEvent(..., target=0x34)` as the fallback. tvOS simulators do not support direct screen touch; browser/API tap maps to Enter, swipe maps to arrow keys, and the native bridge rejects tvOS touch packets before they reach guest `SimulatorHID`. watchOS/tvOS skip dynamic pointer/mouse service warm-up because those guest runtimes abort on unsupported virtual services. Apple TV and Apple Watch simulators are fixed-orientation devices, so client and server rotation paths must not expose or dispatch device rotation for those families.
84+
Two-point multi-touch dispatch prefers the current SimulatorKit/Indigo packet constructor and falls back to SimDeck's manual Indigo packet adapter. The fallback must preserve `began`/`moved`/`ended` mouse event types instead of collapsing movement into down/up packets, because touch recognizers in guest apps otherwise see pinches as ordinary single-finger swipes.
8485
WebKit inspection uses the simulator `webinspectord` Unix socket named `com.apple.webinspectord_sim.socket` and WebKit's binary-plist Remote Inspector selectors. It lists only WebKit content that the runtime exposes as inspectable. For app-owned `WKWebView` on iOS 16.4 and newer, the app must set `isInspectable = true`.
8586

8687
## Build and Run

‎cli/DFPrivateSimulatorDisplayBridge.m‎

Lines changed: 14 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,7 @@ typedef NS_ENUM(NSInteger, DFSimulatorInputFamily) {
159159
double value;
160160
} DFUnitAngleMeasurement;
161161

162-
static DFIndigoMessage *DFCreateIndigoMultiTouchMessage(CGPoint normalizedPoint1, CGPoint normalizedPoint2, NSSize displaySize, BOOL touchDown, uint32_t target, NSError **error);
162+
static DFIndigoMessage *DFCreateIndigoMultiTouchMessage(CGPoint normalizedPoint1, CGPoint normalizedPoint2, NSSize displaySize, DFPrivateSimulatorTouchPhase phase, uint32_t target, NSError **error);
163163

164164
typedef NS_ENUM(NSInteger, DFPrivateSimulatorErrorCode) {
165165
DFPrivateSimulatorErrorCodeFrameworkLoadFailed = 1,
@@ -1435,7 +1435,7 @@ static id DFCreateLegacyHIDClientForDevice(id device, NSError **error) {
14351435
static DFIndigoMessage *DFCreateIndigoMultiTouchMessage(CGPoint normalizedPoint1,
14361436
CGPoint normalizedPoint2,
14371437
NSSize displaySize,
1438-
BOOL touchDown,
1438+
DFPrivateSimulatorTouchPhase phase,
14391439
uint32_t target,
14401440
NSError **error) {
14411441
DFIndigoHIDMessageForMouseNSEventFn mouseMessage = (DFIndigoHIDMessageForMouseNSEventFn)dlsym(RTLD_DEFAULT, "IndigoHIDMessageForMouseNSEvent");
@@ -1449,7 +1449,7 @@ static id DFCreateLegacyHIDClientForDevice(id device, NSError **error) {
14491449
return NULL;
14501450
}
14511451

1452-
NSEventType eventType = touchDown ? NSEventTypeLeftMouseDown : NSEventTypeLeftMouseUp;
1452+
NSEventType eventType = (NSEventType)DFIndigoMouseEventTypeForPhase(phase);
14531453
CGPoint ratioPoint = CGPointMake(
14541454
fmax(0.0, fmin(1.0, normalizedPoint1.x)),
14551455
fmax(0.0, fmin(1.0, normalizedPoint1.y))
@@ -4096,21 +4096,20 @@ - (BOOL)sendMultiTouchAtNormalizedX1:(double)normalizedX1
40964096
}
40974097

40984098
uint32_t target = DFIndigoDigitizerTarget;
4099-
BOOL touchDown = phase == DFPrivateSimulatorTouchPhaseBegan || phase == DFPrivateSimulatorTouchPhaseMoved;
4100-
IndigoHIDMessage *message = (IndigoHIDMessage *)DFCreateIndigoMultiTouchMessage(CGPointMake(x1, y1),
4101-
CGPointMake(x2, y2),
4102-
displaySize,
4103-
touchDown,
4104-
target,
4105-
&dispatchError);
4099+
IndigoHIDMessage *message = DFCreateIndigoMultiTouchMessageDirect(CGPointMake(x1, y1),
4100+
CGPointMake(x2, y2),
4101+
displaySize,
4102+
phase,
4103+
target);
41064104
if (message == NULL) {
41074105
const NSUInteger maxAttempts = phase == DFPrivateSimulatorTouchPhaseMoved ? 12 : 3;
41084106
for (NSUInteger attempt = 0; attempt < maxAttempts; attempt++) {
4109-
message = DFCreateIndigoMultiTouchMessageDirect(CGPointMake(x1, y1),
4110-
CGPointMake(x2, y2),
4111-
displaySize,
4112-
phase,
4113-
target);
4107+
message = (IndigoHIDMessage *)DFCreateIndigoMultiTouchMessage(CGPointMake(x1, y1),
4108+
CGPointMake(x2, y2),
4109+
displaySize,
4110+
phase,
4111+
target,
4112+
&dispatchError);
41144113
if (message != NULL) {
41154114
break;
41164115
}

‎cli/native/XCWNativeBridge.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ void * _Nullable xcw_native_input_create(const char * _Nonnull udid, char * _Nul
7575
void xcw_native_input_destroy(void * _Nullable handle);
7676
bool xcw_native_input_display_size(void * _Nonnull handle, double * _Nullable width, double * _Nullable height);
7777
bool xcw_native_input_send_touch(void * _Nonnull handle, double x, double y, const char * _Nonnull phase, char * _Nullable * _Nullable error_message);
78+
bool xcw_native_input_send_edge_touch(void * _Nonnull handle, double x, double y, const char * _Nonnull phase, uint32_t edge, char * _Nullable * _Nullable error_message);
7879
bool xcw_native_input_send_multitouch(void * _Nonnull handle, double x1, double y1, double x2, double y2, const char * _Nonnull phase, char * _Nullable * _Nullable error_message);
7980
bool xcw_native_input_send_key(void * _Nonnull handle, uint16_t key_code, uint32_t modifiers, char * _Nullable * _Nullable error_message);
8081
bool xcw_native_input_send_key_event(void * _Nonnull handle, uint16_t key_code, bool down, char * _Nullable * _Nullable error_message);

‎cli/native/XCWNativeBridge.m‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -736,6 +736,33 @@ bool xcw_native_input_send_touch(void *handle, double x, double y, const char *p
736736
}
737737
}
738738

739+
bool xcw_native_input_send_edge_touch(void *handle, double x, double y, const char *phase, uint32_t edge, char **error_message) {
740+
@autoreleasepool {
741+
if (handle == NULL) {
742+
XCWSetErrorMessage(error_message, [NSError errorWithDomain:@"SimDeck.NativeInput"
743+
code:1
744+
userInfo:@{NSLocalizedDescriptionKey: @"Native input handle is null."}]);
745+
return false;
746+
}
747+
NSError *phaseError = nil;
748+
DFPrivateSimulatorTouchPhase touchPhase = DFPrivateSimulatorTouchPhaseMoved;
749+
if (!XCWTouchPhaseFromString(XCWStringFromCString(phase), &touchPhase, &phaseError)) {
750+
XCWSetErrorMessage(error_message, phaseError);
751+
return false;
752+
}
753+
NSError *error = nil;
754+
BOOL ok = [(__bridge DFPrivateSimulatorDisplayBridge *)handle sendEdgeTouchAtNormalizedX:x
755+
normalizedY:y
756+
phase:touchPhase
757+
edge:(DFPrivateSimulatorTouchEdge)edge
758+
error:&error];
759+
if (!ok) {
760+
XCWSetErrorMessage(error_message, error);
761+
}
762+
return ok;
763+
}
764+
}
765+
739766
bool xcw_native_input_send_multitouch(void *handle, double x1, double y1, double x2, double y2, const char *phase, char **error_message) {
740767
@autoreleasepool {
741768
if (handle == NULL) {

‎docs/api/rest.md‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -171,22 +171,24 @@ Response:
171171

172172
## Input
173173

174-
| Method | Path | Body |
175-
| ------ | ----------------------------------------- | ------------------------------------------ |
176-
| `POST` | `/api/simulators/{udid}/tap` | Selector or coordinate tap |
177-
| `POST` | `/api/simulators/{udid}/touch` | `{ "x": 120, "y": 240, "phase": "began" }` |
178-
| `POST` | `/api/simulators/{udid}/touch-sequence` | Multiple touch phases |
179-
| `POST` | `/api/simulators/{udid}/key` | `{ "keyCode": 4, "modifiers": 0 }` |
180-
| `POST` | `/api/simulators/{udid}/key-sequence` | `{ "keyCodes": [11,8,15], "delayMs": 5 }` |
181-
| `POST` | `/api/simulators/{udid}/button` | `{ "button": "lock", "durationMs": 50 }` |
182-
| `POST` | `/api/simulators/{udid}/crown` | `{ "delta": 50 }` |
183-
| `POST` | `/api/simulators/{udid}/dismiss-keyboard` | Dismiss the software keyboard |
184-
| `POST` | `/api/simulators/{udid}/home` | Press Home |
185-
| `POST` | `/api/simulators/{udid}/app-switcher` | Open app switcher |
186-
| `POST` | `/api/simulators/{udid}/rotate-left` | Rotate left |
187-
| `POST` | `/api/simulators/{udid}/rotate-right` | Rotate right |
188-
189-
Touch coordinates are screen points unless the endpoint body explicitly uses normalized values.
174+
| Method | Path | Body |
175+
| ------ | ----------------------------------------- | -------------------------------------------------------------------- |
176+
| `POST` | `/api/simulators/{udid}/tap` | Selector or coordinate tap |
177+
| `POST` | `/api/simulators/{udid}/touch` | `{ "x": 0.5, "y": 0.5, "phase": "began" }` |
178+
| `POST` | `/api/simulators/{udid}/edge-touch` | `{ "x": 0.5, "y": 0.98, "phase": "began", "edge": "bottom" }` |
179+
| `POST` | `/api/simulators/{udid}/multi-touch` | `{ "x1": 0.35, "y1": 0.5, "x2": 0.65, "y2": 0.5, "phase": "began" }` |
180+
| `POST` | `/api/simulators/{udid}/touch-sequence` | Multiple touch phases |
181+
| `POST` | `/api/simulators/{udid}/key` | `{ "keyCode": 4, "modifiers": 0 }` |
182+
| `POST` | `/api/simulators/{udid}/key-sequence` | `{ "keyCodes": [11,8,15], "delayMs": 5 }` |
183+
| `POST` | `/api/simulators/{udid}/button` | `{ "button": "lock", "durationMs": 50 }` |
184+
| `POST` | `/api/simulators/{udid}/crown` | `{ "delta": 50 }` |
185+
| `POST` | `/api/simulators/{udid}/dismiss-keyboard` | Dismiss the software keyboard |
186+
| `POST` | `/api/simulators/{udid}/home` | Press Home |
187+
| `POST` | `/api/simulators/{udid}/app-switcher` | Open app switcher |
188+
| `POST` | `/api/simulators/{udid}/rotate-left` | Rotate left |
189+
| `POST` | `/api/simulators/{udid}/rotate-right` | Rotate right |
190+
191+
Touch, edge-touch, and multi-touch coordinates are normalized from `0.0` to `1.0`.
190192

191193
## UI State And Inspection
192194

0 commit comments

Comments
 (0)