Add macOS platform scaffolding

[LikeTheSalad]

This is a continuation from #64 that:

• Finishes fully removing window-specific dependencies from shared code
• Starts adding macos scaffolding

The complicated part, which I'm still not sure I fully understand, is about handling platform-agnostic key names within KeyboardLayout. So I would appreciate some guidance on that topic.

These changes are AI-assisted.

[petoncle]

Nice! Did you have any questions in mind about KeyboardLayout? Did you understand my comment in the other PR (#64 (comment)), especially the part about extending the KeyboardLayoutKey record with new fields (physical-key identifiers for macOS/Linux)?

A good start would be to look into what the equivalent of Windows's keyboard and mouse hook would be on macOS:

mousemaster/src/main/java/mousemaster/platform/windows/WindowsPlatform.java

Lines 212 to 224 in e425397

	keyboardHook = User32.INSTANCE.SetWindowsHookEx(WinUser.WH_KEYBOARD_LL,
	keyboardHookCallback, hMod, 0);
	if (keyboardHook == null)
	throw new IllegalStateException(
	"Unable to install keyboard hook " + Native.getLastError());
	logger.trace("Installed keyboard hook successfully");
	// Run mouse hook in a separate thread to avoid lags:
	// https://www.linkedin.com/pulse/windows-mouse-hook-lagging-simone-galleni
	// https://stackoverflow.com/a/52201983
	Thread mouseHookThread = new Thread(this::mouseHook);
	mouseHookThread.setName("mouse-hook");
	mouseHookThread.start();
	addJvmShutdownHook();

How can we receive keyboard and mouse events, what does that look like, what data structures do we get, what identifiers do we get to identify keys?

[LikeTheSalad]

Did you have any questions in mind about KeyboardLayout? Did you understand my comment in the other PR (#64 (comment)), especially the part about extending the KeyboardLayoutKey record with new fields (physical-key identifiers for macOS/Linux)?

I do have some questions regarding this part, it's actually the one that's a bit confusing to me.

So I'll first outline the overall key concepts to make sure I fully understand them:

• We need to capture a keyboard event as a Key object, regardless of the underlying OS.
• Each platform has its own codes for key events, and we shouldn't assume that those would overlap, so we need to be aware of each platform's key code and somehow "translate" those into a universal Key object.
• The current "translation" process for Windows happens as a combination of 2 types, WindowsVirtualKey (which contains Windows-specific codes) and KeyboardLayout (which transforms Windows-specific codes into Key objects).

If the above is correct, I see a couple of options to introduce a new set of key codes for a new OS platform:

1. Creating a new platform-specific enum (i.e. MacosVirtualKey) and then adding mappings for it within the existing KeyboardLayout type, so that it becomes aware of how to transform key codes from both platforms (it would have to define a new method to only transform MacOS-specific keys, and probably get its current Windows-key transformation method renamed to include "windows" in its name to make the intent clear).
2. Abstracting KeyboardLayout so that it can have an implementation per OS that takes care of transforming OS-specific keys into a Key object.
3. Extending the existing WindowsVirtualKey enum to include a mixture of OS key codes (from Windows and from MacOS for now), rename it to a more generic name (i.e. VirtualKey) and making KeyboardLayout smart enough to tell that multiple enum values can translate to a single Key object.

My opinions on each option:

Option 1 Separates key codes across enums, which makes for a good separation of concerns in that regard, which I like to make things clearer when it comes to changing codes in the future. However, it adds multiple OS responsibilities into the KeyboardLayout object, which will make it bigger per OS. I usually try to keep classes focused to avoid getting lost into them later when I don't remember what I did there, so it's easier for me to extend and maintain them in the long run.

Option 2 This is something I already somewhat proposed in my previous PR, I like the abstraction because each type will have a single OS responsibility to translate to Key, which makes things easier to read and therefore maintain.

Option 3 This option to me is the worst, as it overloads multiple types with multiple OS responsibilities.

That's how I see it all based on my current understanding of the project, please let me know what you think, if I misunderstood something about the architecture in general, and also please let me know about your opinions on the potential approaches to address this, or if you have a different one that I didn't mention. Thanks!

[petoncle]

Hey @LikeTheSalad, I agree with option #1: Creating a new platform-specific enum. I don't think it will be named MacosVirtualKey though, because virtual key is a Windows concept.

To be able to know what this platform-specific enum is going to be, I had to start doing some research on the macOS APIs for handling keyboard events. This should help us understand where we're going and it will drive the overall architecture. There's a lot of information below, don't be afraid if you're not familiar with this stuff. It starts making sense once you look into it and tries it yourself 😊 But yeah, we're really getting into the hard part of the macOS port here.

I initially digged into the CGEventTap API for macOS, which is the obvious equivalent to the Windows keyboard hook API. But a few hours later, I realized that it has several limitations that would prevent us from implementing some of the mousemaster features:

1. It does not cleanly map to the model that mousemaster heavily relies on, which is that "no keys are special", "it's just key press events and key release events".
2. Some keys break that model and are special in macOS's CGEventTap API: the modifier keys (leftshift, etc.) and capslock. Even though I believe there's a workaround for modifier keys like leftshift, the capslock limitation has no workaround and it wouldn't be usable in mousemaster.
3. Any apps in macOS can enable "secure event input", and when they do so, CGEventTap stops feeding keyboard events altogether. mousemaster would not receive any events, meaning that mousemaster can't be used as a mouse replacement in those apps.

See the results of my research on CGEventTap below ("Approach 2") for more details on how we would integrate it in mousemaster.

The solution to the CGEventTap limitations is to use something lower level. I've looked into how kanata does it for macOS (there's a lot of overlap between kanata and mousemaster, so it makes sense to take inspiration from it). Kanata does not use CGEventTap for the keyboard at all; on macOS it grabs the keyboard at the HID/DriverKit layer via the Karabiner-DriverKit-VirtualHIDDevice driver. With this custom driver, we can get the model that mousemaster expects: everything is just a key press event or a key release event. The cost is: it requires running as root, Input Monitoring + Accessibility permissions, and the Karabiner VirtualHIDDevice driver + daemon installed.

Now I'll share my findings about each of these 2 approaches (Karabiner-DriverKit vs CGEventTap).

1. Approach 1: Karabiner-DriverKit (ideal, but initial implementation a tiny bit complex 😊)

1.1 Wiring Karabiner-DriverKit into mousemaster with JNA

JNA can only call a flat C ABI, so we can't call the Karabiner C++ client directly. We need a native .dylib (macOS's equivalent of a Windows .dll) that exposes a few extern "C" functions, and JNA binds to that.

kanata's grab/inject code is exactly that native layer. The karabiner-driverkit crate (repo: https://github.com/Psych3r/driverkit) is a thin Rust wrapper over a single C++ file, c_src/driverkit.cpp, which already exposes a flat extern "C" ABI. We compile that one file into libdriverkit.dylib and JNA binds its symbols directly. (Alternative: depend on the crate and build a Rust cdylib; same result)

1. Build the dylib. c_src/driverkit.cpp is a single C++23 translation unit. It needs the repo's git submodules (Karabiner-DriverKit-VirtualHIDDevice headers) on the include path and links the IOKit + CoreFoundation frameworks:

git clone --recursive https://github.com/Psych3r/driverkit
cd driverkit
clang++ -std=c++23 -dynamiclib -O2 -o libdriverkit.dylib \
    c_src/driverkit.cpp \
    -I c_src/Karabiner-DriverKit-VirtualHIDDevice/include/pqrs/karabiner/driverkit \
    -I c_src/Karabiner-DriverKit-VirtualHIDDevice/vendor/vendor/include \
    -I c_src/Karabiner-DriverKit-VirtualHIDDevice/src/Daemon/vendor/include \
    -framework IOKit -framework CoreFoundation

2. The C ABI (from c_src/driverkit.hpp / .cpp): the symbols JNA binds to:

struct DKEvent { uint64_t value; uint32_t page; uint32_t code; uint64_t device_hash; };

bool driver_activated();                    // driver installed + approved?
bool register_device(const char* product);  // must register >=1 device BEFORE grab()
int  grab();                                 // 0 = success; nonzero = failure (1 = no device registered)
int  wait_key(struct DKEvent* e);            // BLOCKING: 1 = event written to *e, 0 = input released (EOF)
int  send_key(struct DKEvent* e);            // 0 = success; 1 = bad usage page, 2 = sink not ready
bool is_sink_ready();                        // virtual output device connected?
void release();                              // ungrab / cleanup

value == 1 is key down/press, value == 0 is key up/release. Required call order: driver_activated() -> register_device() -> grab() -> loop wait_key() (+ send_key() for output) -> release().

3. Bind it in Java. wait_key blocks, so the read loop runs on its own thread:

public interface Driverkit extends Library {
    Driverkit INSTANCE = Native.load("driverkit", Driverkit.class); // libdriverkit.dylib

    @Structure.FieldOrder({ "value", "page", "code", "deviceHash" })
    class DKEvent extends Structure {
        public long value;      // uint64_t  (1 = press, 0 = release)
        public int  page;       // uint32_t  HID usage page (0x07 = keyboard_or_keypad)
        public int  code;       // uint32_t  HID usage (the individual key)
        public long deviceHash; // uint64_t
    }

    boolean driver_activated();
    boolean register_device(String product);
    int     grab();              // 0 = success
    int     wait_key(DKEvent e); // JNA passes the Structure by pointer, auto-reads after return
    int     send_key(DKEvent e);
    boolean is_sink_ready();
    void    release();
}

Thread t = new Thread(() -> {
    if (!Driverkit.INSTANCE.driver_activated()) return;   // driver not installed / approved
    Driverkit.INSTANCE.register_device(...);              // register the keyboard(s) first
    if (Driverkit.INSTANCE.grab() != 0) return;           // grab failed (root? permissions? no device?)
    Driverkit.DKEvent ev = new Driverkit.DKEvent();
    while (running) {
        if (Driverkit.INSTANCE.wait_key(ev) == 0) break;  // input released (EOF)
        ev.read();
        Key key = keyFromHidUsage(ev.page, ev.code);
        KeyEvent e = ev.value == 1 ? new PressKeyEvent(now, key) : new ReleaseKeyEvent(now, key);
        // feed KeyboardManager exactly like keyboardHookCallback does today
    }
    Driverkit.INSTANCE.release();
});
t.setName("mac-hid-loop"); t.start();

1.2 Mapping HID usages to mousemaster Key

• Input events carry a raw HID usage page + usage (the official pqrs/Karabiner types are pqrs::hid::usage_page and pqrs::hid::usage; DKEvent names them page and code). usage_page 0x07 is keyboard_or_keypad; usage is the individual key. These are not kVK_* keyCodes (keyCodes are used in Approach 2 below).
• The equivalent of the WindowsVirtualKeyboard enum would be a MacosHidUsage enum:

 public enum MacosHidUsage {
      // (usagePage, usage, kVK)  -- kVK is the macOS virtual keycode
      KEYBOARD_A(0x07, 0x04, 0x00),  // ... through Z = 0x1D
      KEYBOARD_1(0x07, 0x1E, 0x12),  // ... 0 = 0x27
      KEYBOARD_RETURN(0x07, 0x28, 0x24),
      KEYBOARD_ESCAPE(0x07, 0x29, 0x35),
      KEYBOARD_DELETE_OR_BACKSPACE(0x07, 0x2A, 0x33),  // HID/Karabiner name; physical Backspace key = mousemaster backspace
      KEYBOARD_TAB(0x07, 0x2B, 0x30),
      KEYBOARD_SPACEBAR(0x07, 0x2C, 0x31),
      KEYBOARD_CAPS_LOCK(0x07, 0x39, 0x39),
      KEYBOARD_F1(0x07, 0x3A, 0x7A),  // ... F12 = 0x45
      KEYBOARD_DELETE_FORWARD(0x07, 0x4C, 0x75),  // = mousemaster del
      KEYBOARD_RIGHT_ARROW(0x07, 0x4F, 0x7C),
      KEYBOARD_LEFT_ARROW(0x07, 0x50, 0x7B),
      KEYBOARD_DOWN_ARROW(0x07, 0x51, 0x7D),
      KEYBOARD_UP_ARROW(0x07, 0x52, 0x7E),
      KEYPAD_NUM_LOCK(0x07, 0x53, 0x47),  // mac "Clear" key (kVK_ANSI_KeypadClear)
      // Modifiers — real left/right usages, no flags (unlike Approach 2) 
      KEYBOARD_LEFT_CONTROL(0x07, 0xE0, 0x3B),
      KEYBOARD_LEFT_SHIFT(0x07, 0xE1, 0x38),
      KEYBOARD_LEFT_ALT(0x07, 0xE2, 0x3A),
      KEYBOARD_LEFT_GUI(0x07, 0xE3, 0x37),  // Cmd -> mousemaster leftwin
      KEYBOARD_RIGHT_CONTROL(0x07, 0xE4, 0x3E),
      KEYBOARD_RIGHT_SHIFT(0x07, 0xE5, 0x3C),
      KEYBOARD_RIGHT_ALT(0x07, 0xE6, 0x3D),
      KEYBOARD_RIGHT_GUI(0x07, 0xE7, 0x36);

      public final int usagePage;
      public final int usage;
      public final int kVK; // macOS virtual keycode; only used for the character keys (letters/digits/punctuation) fed to UCKeyTranslate
      MacosHidUsage(int usagePage, int usage, int kVK) { this.usagePage = usagePage; this.usage = usage; this.kVK = kVK; }
  }

• Layout-dependent characters need translation: HID usage -> kVK_* (fixed positional map) -> UCKeyTranslate -> character -> Key.ofCharacter.

---

2. Approach 2: CGEventTap API in macOS (limited but probably simpler to setup)

WH_KEYBOARD_LL -> CGEventTap (Quartz Event Services, in the CoreGraphics / ApplicationServices framework).

WindowsPlatform	MacosPlatform
SetWindowsHookEx(WH_KEYBOARD_LL, ...)	CGEventTapCreate(...)
Callback function WinDef.LRESULT keyboardHookCallback(int nCode, WinDef.WPARAM wParam, WinUser.KBDLLHOOKSTRUCT info)	CGEventRef cb(proxy, CGEventType type, CGEventRef event, void* ctx)
wParam = WM_KEYDOWN / WM_KEYUP	type = kCGEventKeyDown / kCGEventKeyUp / kCGEventFlagsChanged. modifiers come as kCGEventFlagsChanged
return LRESULT(1) to eat / CallNextHookEx(...) to pass	return NULL to eat / return event to pass
KBDLLHOOKSTRUCT.scanCode	CGEventGetIntegerValueField(event, kCGKeyboardEventKeycode) → CGKeyCode (position-based, like a scan code)
KBDLLHOOKSTRUCT.vkCode	no equivalent, not needed
KBDLLHOOKSTRUCT.flags (modifier state)	CGEventGetFlags(event) → CGEventFlags bitmask
LLKHF_INJECTED / dwExtraInfo signature	CGEventGetIntegerValueField(event, kCGEventSourceUserData)
KBDLLHOOKSTRUCT.time	CGEventGetTimestamp(event)
GetAsyncKeyState (stuck-key sanity check)	CGEventSourceKeyState()
SendInput (regurgitation / injection)	CGEventPost(kCGHIDEventTap, CGEventCreateKeyboardEvent(...))
activeKeyboardLayout() via foreground window	TISCopyCurrentKeyboardLayoutInputSource() + kTISPropertyUnicodeKeyLayoutData

2.1 Mapping macOS keyCode to mousemaster Key

macOS CGKeyCode (from CGEventGetIntegerValueField(event, kCGKeyboardEventKeycode)) is position-based, like a Windows scan code: kVK_ANSI_A is always the physical "A" position regardless of the active layout. So the mapping splits into two buckets:

1. Layout-independent keys (Key.keyboardLayoutIndependentKeys): the static kVK_* -> Key table below.
2. Character-producing keys (letters, digits, punctuation): the input keycode needs to be translated UCKeyTranslate (current layout) to the base character, then Key.ofCharacter(char).

2.2 Layout-independent keys in macOS (table is shortened, this is just to give a general idea)

Keycode	hex	macOS kVK_*	mousemaster Key
48	0x30	kVK_Tab	tab
36	0x24	kVK_Return	enter
76	0x4C	kVK_ANSI_KeypadEnter	enter
53	0x35	kVK_Escape	esc
Modifiers (arrive as kCGEventFlagsChanged, not KeyDown/KeyUp — see note)
56	0x38	kVK_Shift	leftshift
60	0x3C	kVK_RightShift	rightshift
59	0x3B	kVK_Control	leftctrl
63	0x3F	kVK_Function (Fn)	no Windows equivalent. To do (?)
Function keys
122	0x7A	kVK_F1	f1
120	0x78	kVK_F2	f2
Numpad
82	0x52	kVK_ANSI_Keypad0	numpad0
83	0x53	kVK_ANSI_Keypad1	numpad1

2.3 Modifiers: macOS vs Windows

mousemaster treats a modifier as just another key (press -> decide to eat -> release).

2.3.1 Reading modifier keys

On Windows, leftshift arrives as WM_KEYDOWN/WM_KEYUP, same shape as a. On macOS, modifier keys never come as KeyDown/KeyUp — only as kCGEventFlagsChanged, which carries no press/release bit. So MacosPlatform must diff CGEventGetFlags(event) against the previous flags to decide down-vs-up, then emit the same PressKeyEvent/ReleaseKeyEvent the rest of mousemaster expects.

2.3.2 Eating modifier keys

Eating a normal key is clean on both (LRESULT(1) on Windows == return NULL on macOS). Eating a modifier key is the one hard case: eating a leftshift key press event is not enough. If you press leftshift and mousemaster eats it, then press a, macOS will still emit a a with shift flag true event. The right approach seems to be to eat the leftshift press event and generate a "leftshift is up" event.

2.3.3 Hard limitation: capslock

Capslock is keycode 57 (kVK_CapsLock), flag kCGEventFlagMaskAlphaShift (0x10000), and like the other modifiers it arrives as kCGEventFlagsChanged. The difference: the flag bit encodes the lock latch (caps on/off), not whether the key is physically held. The bit flips once per toggle and then stays.

Consequence: we can't translate that cleanly to mousemaster's "key press event" / "key release event" model. Capslock can't be used as any other key.

[petoncle]

Hi @LikeTheSalad, I've explored the ideas we discussed here and arrived at a working build of mousemaster on macOS. It needs the Karabiner virtual HID driver, and it's not signed or notarized yet (no Apple developer account).

I don't have easy access to a Mac, so testing is still difficult for me.

Nightly build: https://github.com/petoncle/mousemaster/releases/tag/nightly
Installation instructions: https://github.com/petoncle/mousemaster#macos

mousemaster-macos-demo.mp4

[LikeTheSalad]

Thank you, @petoncle ! I haven't got time to go back to it over the past weeks, so thanks for taking a look.

I tried it out, but I had to rebuild it for my Intel mac because the currently release builds are for arm ones. I created this AI-assisted PR to also created intel-supported builds: #80

It was tricky at first to enable it, but then I found out that it was Karabiner stealing the key events, so I had to go to settings -> Devices and disable all "Modify events" toggles associated to my keyboard, then it worked.

The mouse movements work well and I can also see the grid taking the whole screen, however, it seems like the "hints" don't capture the whole screen size or the like, because they seem to be narrowed on a corner and don't provide hints on top of the actual UI elements. Apart from that, it works well. I'll attach a recording of me using it. Thanks again.

Mousemaster.mp4

[petoncle]

Thanks for trying it out. I suspect the bug with the hints is related to qScreen.devicePixelRatio().

Could you share the mousemaster.log (the first lines mostly where it logs the screen list). It shows something like:

Screens [Screen[rectangle=Rectangle[x=0, y=0, width=1920, height=1080], dpi=72, scale=1.0]]

Does yours show scale=2.0?

[LikeTheSalad]

Does yours show scale=2.0?

Yes, it does. Here's the full log:

mousemaster.log

[petoncle]

Thanks for the log.

The problem was that Qt reports and draws in physical pixels on Windows, but in logical points on macOS. mousemaster assumed pixels everywhere, so on a Retina screen everything ended up at half its intended position. I've changed the macOS side to convert to pixels.
Your log should now report the screen in physical pixels, something like:

Screens [Screen[rectangle=Rectangle[x=0, y=0, width=2880, height=1800], dpi=72, scale=2.0]]

See this comment in the other PR about the Intel build:
#80 (comment)

[LikeTheSalad]

Thanks for the log.

The problem was that Qt reports and draws in physical pixels on Windows, but in logical points on macOS. mousemaster assumed pixels everywhere, so on a Retina screen everything ended up at half its intended position. I've changed the macOS side to convert to pixels. Your log should now report the screen in physical pixels, something like:

Screens [Screen[rectangle=Rectangle[x=0, y=0, width=2880, height=1800], dpi=72, scale=2.0]]

See this comment in the other PR about the Intel build: #80 (comment)

Thanks! I just tried the new nightly build and it's mostly working well now, the hints and coordinates are correct, but the initial overlay seems to be always rendered in the bottom half of the screen. I recorded it (btw the delayed animations are only in the recordings, it's all smooth when I'm using it)

Mousemaster.2.mp4

[petoncle]

Thanks for the video, very useful. I tried to fix these issues:

• There were still a couple of places where the display scaling was mishandled. The animation was affected too (wee see it in the video at 00:08: the animation shows a transition frame that is 2x too large).
• The UI hints were missing inside the page in your Brave browser.
• There was a bleed of the UI hint box color (blue) on the top-left corner of the hint.
• The transparent overlay on top of the UI hints was mispositioned.
• The text in the UI hint boxes was not perfectly vertically centered.

The nightly build just completed.

[LikeTheSalad]

Thank you @petoncle - I'll take a look over the weekend. Btw, regarding this part:

The animation was affected too (wee see it in the video at 00:08: the animation shows a transition frame that is 2x too large).

I didn't see the animation that way; I think the long/delayed effect is a side effect of the recording. Still, I'll keep an eye on it next time.

[LikeTheSalad]

Hi @petoncle

I just did a new test and this is what I found:

Good

• The hint grid is properly displayed and accurate when I select an option.
• The UI hints are properly displayed and are accurate when I select one.
• There's no animation delay (the delay observed in the recording is a recording side-effect).
• I no longer see a red frame when running mousemaster.

Bad

• When I press "G" and sometimes also when I use UI hints, the mouse gets stuck until I either run another hint or stop mousemaster (as seen in the recording)
• This happened only once and it's in the recording, but at some point I tried to show the UI hints and I got an empty overlay instead (maybe the focused app wasn't clear to mousemaster at the time) in any case, is not a big deal because I could just press ESC to remove it.

Here is the recording and the logs:

Mousemaster.15.8.mp4

mousemaster.log

Thanks again.

[petoncle]

I looked into the stuck mouse issue because that's pretty bad.
MouseManager:744 treats a smooth jump as finished only when the polled cursor position is exactly equal to the jump target. On a Retina screen that can never happen for an odd pixel: MacosScreens.pixelPoint() returns logical * scale, so at 2x every reported position is even. The jump therefore stays jumping == true forever, and every 10 ms frame re-warps the cursor back to the target.
Deployed here: https://github.com/petoncle/mousemaster/releases/tag/nightly

[LikeTheSalad]

I looked into the stuck mouse issue because that's pretty bad. MouseManager:744 treats a smooth jump as finished only when the polled cursor position is exactly equal to the jump target. On a Retina screen that can never happen for an odd pixel: MacosScreens.pixelPoint() returns logical * scale, so at 2x every reported position is even. The jump therefore stays jumping == true forever, and every 10 ms frame re-warps the cursor back to the target. Deployed here: https://github.com/petoncle/mousemaster/releases/tag/nightly

Thank you, @petoncle!

I tried the latest build and I no longer see the mouse stuck issue 👍

I think overall we can say that Mousemaster is ready to get more broadly tested in macOS, so maybe it can be more than just nightly builds, as it seems to me that the basics work well now. The only tricky bit for me was installing and configuring Karabiner to prevent it from stealing keyboard events as I mentioned earlier, and having to manually search for mousemaster in the macOS security settings to grant permissions to it, which btw, I haven't had to do so again when installing newer builds, so at least your initial warning of having to re-grant these permissions on a new install didn't apply to me, but I can't say it'd be the same for other macOS versions so I guess it's better to keep that warning in the docs.

Thank you for your patience with adding support for macOS. I believe we can close this PR now.