This document contains comprehensive debugging information for Kanata keyboard remapping integration on macOS, based on extensive real-world debugging sessions.
- Quick Reference
- The "Zombie Keyboard Capture" Issue
- Thread Safety in Swift Apps
- VirtualHID Connection Issues
- Common Problems & Solutions
- Debugging Workflows
- Architecture & Best Practices
If keyboard becomes unresponsive:
- Kanata Emergency Sequence:
Left Ctrl + Space + Esc(works even when keyboard seems dead) - Kill all processes:
sudo pkill -f kanata - Restart VirtualHID daemon: Kill and restart Karabiner daemon
# Check if Kanata is working (look for key events in output)
timeout 5s sudo kanata --cfg /path/to/config.kbd --debug
# Check daemon status
ps aux | grep -i karabiner | grep -v grep
# Monitor real-time logs
tail -f ~/Library/Logs/KeyPath/keypath-debug.log"Zombie Keyboard Capture" occurs when Kanata successfully captures keyboard input but fails to establish proper output connection, leaving the keyboard unresponsive.
Symptoms:
- Keyboard becomes completely unresponsive
- Kanata appears to be running (process exists)
- Logs show
connect_failed asio.system:61errors - Emergency sequence (Ctrl+Space+Esc) still works
Based on extensive debugging, the issue has multiple components:
- VirtualHID Connection Failures:
connect_failed asio.system:61 - Karabiner Daemon Permission Issues: Daemon runs but can't bind properly
- Multiple Process Conflicts: Concurrent Kanata starts causing exclusive access errors
The connect_failed asio.system:61 errors are NOT fatal!
Our testing revealed that Kanata continues to process keyboard events and perform remapping even with these connection errors. The key insight is that these are warnings, not blocking failures.
The solution is not eliminating the connection errors, but:
- Ensuring Karabiner daemon is running (even with permission warnings)
- Preventing multiple concurrent Kanata instances
- Implementing automatic recovery in the app
- Adding proper thread-safety to prevent crashes during recovery
Manual testing showed:
# This works despite connection errors:
sudo kanata --cfg config.kbd --debug
# Output shows:
# - connect_failed asio.system:61 β WARNING, not fatal
# - KeyEvent processing continues β ACTUAL FUNCTIONALITY WORKS
# - Key remapping operates correctly β PROVES IT'S WORKINGSwift's @Published properties are not thread-safe when modified from multiple concurrent contexts, even when using @MainActor.
Common Error:
Unlock of an os_unfair_lock not owned by current thread
Multiple updateStatus() calls can create race conditions when:
- Different tasks call
updateStatus()simultaneously - Individual
MainActor.runblocks execute concurrently @Publishedproperties are modified from different threads
Centralized State Management Pattern:
/// Main actor function to safely update all @Published properties
@MainActor
private func updatePublishedProperties(
isRunning: Bool,
lastProcessExitCode: Int32?,
lastError: String?,
shouldClearDiagnostics: Bool = false
) {
self.isRunning = isRunning
self.lastProcessExitCode = lastProcessExitCode
self.lastError = lastError
if shouldClearDiagnostics {
// Atomically clear diagnostics
let initialCount = diagnostics.count
diagnostics.removeAll { diagnostic in
diagnostic.category == .process ||
diagnostic.category == .permissions ||
(diagnostic.category == .conflict && diagnostic.title.contains("Exit"))
}
// ... logging
}
}Replace all scattered MainActor.run blocks:
// β WRONG - Can cause race conditions
await MainActor.run {
self.isRunning = false
self.lastError = error
}
await MainActor.run {
self.clearProcessDiagnostics()
}
// β
CORRECT - Atomic state update
await updatePublishedProperties(
isRunning: false,
lastProcessExitCode: exitCode,
lastError: error,
shouldClearDiagnostics: true
)Signs your fix worked:
- App runs without crashes during state transitions
- No more
os_unfair_lockerrors in crash reports - UI remains responsive during Kanata start/stop operations
Karabiner Daemon Permission Errors:
[client] [error] virtual_hid_device_service_server: bind_failed: Permission denied
- Impact: Warnings only, not fatal
- Cause: Daemon needs elevated permissions to bind properly
- Solution: Start daemon with regular user (it will still work)
Kanata Connection Errors:
connect_failed asio.system:61
- Impact: Warnings only, remapping still works
- Cause: VirtualHID connection handshake issues
- Solution: Restart daemon, but functionality continues regardless
# Start daemon (permission warnings are normal)
"/Library/Application Support/org.pqrs/Karabiner-DriverKit-VirtualHIDDevice/Applications/Karabiner-VirtualHIDDevice-Daemon.app/Contents/MacOS/Karabiner-VirtualHIDDevice-Daemon" &
# Check daemon is running
ps aux | grep -i karabiner | grep -v grep
# Expected output:
# _driverkit XXX ... DriverKit extension (system)
# malpern XXX ... VirtualHIDDevice-Daemon (user process)If you see these, the issue is elsewhere:
- Kanata debug output shows key events being processed
Attempting to write InputEventmessages appear- Emergency sequence (Ctrl+Space+Esc) works normally
Symptoms:
- No key input registers
- Lock down emitters first: Verify Kanata + simulator outputs (canonical key names, no glyphs) with unit tests before touching the overlay; prevents UI hacks chasing upstream drift.
- Instrument early, narrowly: Add small, always-on log breadcrumbs at each hop (
KeyInput β HoldActivated β ViewModel state β Overlay render) to see where labels drop. - Tunable jitter handling: Keep debounce/grace periods as named constants with comments on the trade-off (flicker vs. linger); makes rapid iteration safe.
- Separate behavior vs. presentation: Handle tap/hold decisions upstream; keep overlay changes purely visual (e.g., label size/weight), reducing regression risk.
- Cache with intent and expiry: Short TTL caches for simulator-resolved labels avoid redundant work while keeping stale values out.
- Simulator fidelity is critical: Tests should assert simulator emits Kanata names (not glyphs) and expected action strings; a small mismatch caused the star label regression.
- Have a repeatable pipeline checklist: A written βinput to overlayβ flow saves time when timing bugs appear again.
- Kanata process running
- Logs show connection errors
Debugging:
# Test if Kanata is actually processing events
timeout 5s sudo kanata --cfg /path/to/config.kbd --debug
# Look for: KeyEvent messages, InputEvent writesSolutions (in order):
- Use emergency sequence:
Left Ctrl + Space + Esc - Kill Kanata:
sudo pkill -f kanata - Restart daemon: Kill and restart Karabiner daemon
- Check for multiple instances:
ps aux | grep kanata
Error Pattern:
entering the processing loop
entering the event loop
IOHIDDeviceOpen error: (iokit/common) exclusive access and device already open
[ERROR] failed to open keyboard device(s): Couldn't register any device
Common Cause: karabiner_grabber Running
This error often occurs because karabiner_grabber is already capturing keyboard input. Karabiner uses both system-level LaunchDaemons and user-level LaunchAgents:
# Check if karabiner_grabber is running
ps aux | grep karabiner_grabber | grep -v grep
# Check system-level services
sudo launchctl list | grep karabiner_grabber
# Check user-level services
launchctl list | grep karabiner_grabber
# Comprehensive removal (requires admin privileges)
# Stop system LaunchDaemon
sudo launchctl bootout system "/Library/Application Support/org.pqrs/Karabiner-Elements/Karabiner-Elements Privileged Daemons.app/Contents/Library/LaunchDaemons/org.pqrs.service.daemon.karabiner_grabber.plist" 2>/dev/null
# Stop user LaunchAgent
launchctl bootout gui/$(id -u) "/Library/Application Support/org.pqrs/Karabiner-Elements/Karabiner-Elements Non-Privileged Agents.app/Contents/Library/LaunchAgents/org.pqrs.service.agent.karabiner_grabber.plist" 2>/dev/null
# Kill any remaining processes
sudo pkill -f karabiner_grabber
sudo pkill -9 -f karabiner_grabber # Force kill stubborn processesPrevention in Code:
// Always check before starting
if isRunning {
AppLogger.shared.log("β
[Init] Kanata already running - skipping")
return
}
// Use actor-based synchronization
await KanataManager.startupActor.synchronize {
await self.performStartKanata()
}Error:
Unlock of an os_unfair_lock not owned by current thread
Fix:
- Implement centralized state management
- Use single
@MainActorfunction for all@Publishedproperty updates - Remove scattered
MainActor.runblocks
Kanata won't start:
# Test basic permissions
sudo kanata --version
# Check sudoers configuration
sudo -l | grep kanatamacOS Specific Permission Requirements:
-
Input Monitoring: System Settings > Privacy & Security > Input Monitoring
- If using KeyPath-managed kanata, add
/Library/KeyPath/bin/kanata - If running standalone kanata outside KeyPath, add
/usr/local/bin/kanata(or/opt/homebrew/bin/kanataon ARM Macs) - Add Terminal.app or your terminal (e.g., Ghostty)
- Add KeyPath.app if using the GUI
- If using KeyPath-managed kanata, add
-
Karabiner Driver Extension: System Settings > Privacy & Security > Driver Extensions
- Enable
Karabiner-VirtualHIDDevice-Manager.app
- Enable
-
Login Items & Extensions: System Settings > General > Login Items & Extensions
- Enable "Karabiner-Elements Non-Privileged Agents"
- Enable "Karabiner-Elements Privileged Daemons"
Required sudoers entries:
# Add to /etc/sudoers via sudo visudo
username ALL=(ALL) NOPASSWD: /usr/local/bin/kanata
username ALL=(ALL) NOPASSWD: /usr/bin/pkill -f kanataAlways validate before starting:
sudo kanata --cfg /path/to/config.kbd --checkSafe config template:
;; Safe configuration template
(defcfg
process-unmapped-keys no ;; IMPORTANT: Only process mapped keys
)
(defsrc
caps
)
(deflayer base
esc
)- Check if Kanata is running:
ps aux | grep kanata - Test configuration:
sudo kanata --cfg config.kbd --check - Review recent logs:
tail -50 ~/Library/Logs/KeyPath/keypath-debug.log - Check daemon status:
ps aux | grep karabiner
# 1. Clean slate
sudo pkill -f kanata
pkill -f "Karabiner-VirtualHIDDevice-Daemon"
# 2. Start daemon
"/Library/Application Support/org.pqrs/Karabiner-DriverKit-VirtualHIDDevice/Applications/Karabiner-VirtualHIDDevice-Daemon.app/Contents/MacOS/Karabiner-VirtualHIDDevice-Daemon" > /tmp/karabiner-daemon.log 2>&1 &
# 3. Test Kanata manually with debug output
timeout 10s sudo kanata --cfg /path/to/config.kbd --debug --log-layer-changes
# 4. Look for key processing evidence
# - KeyEvent messages
# - InputEvent writes
# - Layer changes
# - Emergency sequence recognition# 1. Monitor app logs in real-time
tail -f ~/Library/Logs/KeyPath/keypath-debug.log
# 2. Start app and observe initialization
open -a KeyPath
# 3. Watch for thread safety issues
# - No os_unfair_lock crashes
# - Clean state transitions
# - Proper error handling
# 4. Test recovery systems
# - Use emergency sequence if keyboard becomes unresponsive
# - Check automatic recovery attempts in logs# Find Kanata process lifecycle
grep -E "(Starting Kanata|Successfully started|process exited)" debug.log
# Check for thread safety issues
grep -E "(MainActor|updateStatus|clearProcessDiagnostics)" debug.log
# Look for VirtualHID connection issues
grep -E "(connect_failed|asio\.system:61)" debug.log
# Track automatic recovery attempts
grep -E "(Recovery|attemptKeyboardRecovery|zombie)" debug.log// Proper sudo-based Kanata execution
let task = Process()
task.executableURL = URL(fileURLWithPath: "/usr/bin/sudo")
task.arguments = ["/usr/local/bin/kanata", "--cfg", configPath, "--debug"]
// Monitor output in real-time
let outputPipe = Pipe()
let errorPipe = Pipe()
task.standardOutput = outputPipe
task.standardError = errorPipe
// Termination works through sudo wrapper
if let process = kanataProcess, process.isRunning {
process.terminate() // Properly kills both sudo and kanata
}// Always validate before starting
func validateConfigFile() -> (isValid: Bool, errors: [String]) {
guard FileManager.default.fileExists(atPath: configPath) else {
return (false, ["Config file does not exist"])
}
// Use --check flag for validation
let task = Process()
task.executableURL = URL(fileURLWithPath: "/usr/bin/sudo")
task.arguments = ["/usr/local/bin/kanata", "--cfg", configPath, "--check"]
// ... execute and parse result
}// Automatic recovery for VirtualHID connection failures
private func diagnoseKanataFailure(_ exitCode: Int32, _ output: String) {
switch exitCode {
case 6:
if output.contains("connect_failed asio.system:61") {
// This is "zombie keyboard capture" - attempt recovery
diagnostics.append(KanataDiagnostic(
title: "VirtualHID Connection Failed",
description: "Kanata captured keyboard but failed VirtualHID connection",
canAutoFix: true
))
Task {
await attemptKeyboardRecovery()
}
}
}
}
private func attemptKeyboardRecovery() async {
AppLogger.shared.log("π¨ [Recovery] Attempting keyboard recovery")
// Step 1: Kill all Kanata processes
await killAllKanataProcesses()
// Step 2: Wait for keyboard release
try? await Task.sleep(nanoseconds: 2_000_000_000)
// Step 3: Restart VirtualHID daemon
await restartKarabinerDaemon()
// Step 4: Retry Kanata start
await startKanata()
}// Centralized state management
@MainActor
private func updatePublishedProperties(
isRunning: Bool,
lastProcessExitCode: Int32?,
lastError: String?,
shouldClearDiagnostics: Bool = false
) {
// All @Published property modifications happen atomically here
self.isRunning = isRunning
self.lastProcessExitCode = lastProcessExitCode
self.lastError = lastError
if shouldClearDiagnostics {
// Integrated diagnostics clearing prevents race conditions
let initialCount = diagnostics.count
diagnostics.removeAll { diagnostic in
diagnostic.category == .process ||
diagnostic.category == .permissions ||
(diagnostic.category == .conflict && diagnostic.title.contains("Exit"))
}
let removedCount = initialCount - diagnostics.count
if removedCount > 0 {
AppLogger.shared.log("π [Diagnostics] Cleared \(removedCount) stale diagnostics")
}
}
}
// Usage throughout the app
await updatePublishedProperties(
isRunning: false,
lastProcessExitCode: exitCode,
lastError: errorMessage,
shouldClearDiagnostics: true
)Before running Kanata manually, ensure all Karabiner-Elements components are properly configured:
- Install Karabiner-Elements (if not already installed)
- Enable Driver Extension (System Settings > Privacy & Security > Driver Extensions)
- Enable Background Services (System Settings > General > Login Items & Extensions)
- Karabiner-Elements Non-Privileged Agents β
- Karabiner-Elements Privileged Daemons β
- Grant Input Monitoring to kanata binary and terminal
- Disable karabiner_grabber to prevent conflicts
# 1. Comprehensive Karabiner grabber cleanup
# Stop system LaunchDaemon
sudo launchctl bootout system "/Library/Application Support/org.pqrs/Karabiner-Elements/Karabiner-Elements Privileged Daemons.app/Contents/Library/LaunchDaemons/org.pqrs.service.daemon.karabiner_grabber.plist" 2>/dev/null
# Stop user LaunchAgent
launchctl bootout gui/$(id -u) "/Library/Application Support/org.pqrs/Karabiner-Elements/Karabiner-Elements Non-Privileged Agents.app/Contents/Library/LaunchAgents/org.pqrs.service.agent.karabiner_grabber.plist" 2>/dev/null
# Kill any remaining processes
sudo pkill -f karabiner_grabber
sudo pkill -f kanata
# 2. Start Karabiner VirtualHID daemon (if not running)
"/Library/Application Support/org.pqrs/Karabiner-DriverKit-VirtualHIDDevice/Applications/Karabiner-VirtualHIDDevice-Daemon.app/Contents/MacOS/Karabiner-VirtualHIDDevice-Daemon" &
# 3. Start Kanata with your config
sudo kanata --cfg "/path/to/your/config.kbd"- Permission errors are normal: VirtualHID daemon shows
bind_failed: Permission deniedbut still works - Connection warnings are not fatal:
connect_failed asio.system:61messages don't prevent functionality - Keyboard may briefly freeze: Use emergency sequence (Ctrl+Space+Esc) if needed
β
Kanata binary installation - Verifies kanata executable exists
β
Karabiner driver files exist - Checks /Library/Application Support/org.pqrs/Karabiner-DriverKit-VirtualHIDDevice
β
VirtualHIDDevice-Daemon running - Verifies daemon process is active
β
Driver Extension actually enabled - Uses systemextensionsctl to verify [activated enabled] status
β
Background services enabled - Checks if Login Items & Extensions services are running
β
karabiner_grabber conflicts - Detects conflicting grabber processes
β
Input Monitoring permissions - KeyPath.app and kanata binary TCC database checks
β
Accessibility permissions - KeyPath.app system access verification
β
Service running status - Separate check for whether Kanata is actually running
The wizard can now auto-fix several issues:
- π§ Kill conflicting karabiner_grabber automatically (comprehensive system+user service removal)
- π§ Restart VirtualHID daemon if needed
- π§ Install/uninstall LaunchDaemon services
- π§ Create config directories and files
Problem: Karabiner Elements runs grabber services at both system and user levels that auto-restart when killed.
KeyPath Solution: Comprehensive service shutdown that prevents restarts:
- Stop system LaunchDaemon:
org.pqrs.service.daemon.karabiner_grabber - Stop user LaunchAgent:
org.pqrs.service.agent.karabiner_grabber - Kill remaining processes: Force-kill any stubborn grabber processes
- Verify success: Check that no grabber processes remain running
- Preserve VirtualHID: Keep the HID driver daemon that Kanata needs
Important: This only stops the keyboard grabber service, not the VirtualHID driver that both Karabiner and Kanata require for proper operation.
β Enable Driver Extension - Must be done in System Settings > Privacy & Security > Driver Extensions
β Enable Login Items & Extensions - Must add Karabiner services manually (see detailed steps below)
β Grant Input Monitoring - Must add binaries in System Settings > Privacy & Security
β Grant Accessibility - Must enable for KeyPath.app in System Settings
Problem: Karabiner background services may not appear in System Settings > General > Login Items & Extensions by default.
Solution: Manually add them as Login Items:
- Open System Settings > General > Login Items & Extensions
- Click the "Open at Login" section in the left sidebar
- Click the "+" button to add new items
- Navigate to:
/Library/Application Support/org.pqrs/Karabiner-Elements/ - Add these two applications (drag & drop or use + button):
Karabiner-Elements Non-Privileged Agents.appKarabiner-Elements Privileged Daemons.app
- Restart your Mac or log out/log in for changes to take effect
KeyPath Detection Fix (Updated 2025):
- KeyPath now has a dedicated Background Services wizard page with its own icon
- Fixed detection pattern: now correctly identifies
org.pqrs.service.agent.karabiner_*services - Background Services issues are separated from Input Monitoring permissions
KeyPath Automation Tools:
- π§ "Help" button - Shows detailed step-by-step instructions with helpful tools
- π "Open Karabiner Folder" - Opens Finder directly to the Karabiner apps location
- π "Copy File Paths" - Copies full paths to clipboard for easy navigation
- βοΈ One-click setup - Background Services cards automatically open both System Settings and Finder
Verification: After restart, check that services are running:
launchctl list | grep -i karabiner
# Expected: Multiple karabiner services with PIDs (not "-")
# Example output:
# 10916 0 org.pqrs.service.agent.Karabiner-Menu
# 10919 0 org.pqrs.service.agent.Karabiner-NotificationWindow
# 10221 0 org.pqrs.service.agent.karabiner_console_user_serverCommon Issues:
- Services don't appear in "By Category" view: They will show up in "Open at Login" after manual addition
- KeyPath shows "Background Services Disabled": This was a detection bug fixed in 2025 - the wizard now correctly detects running services
- Services show as disabled despite manual addition: Make sure you added the
.appfiles (not the.plistfiles) to Login Items
π’ All Active (Green):
- All components installed AND Kanata service running
- Shows green check icon with "Close Setup" button (no status text)
π Ready but Not Running (Orange):
- All components installed but Kanata service not running
- Shows "Service Not Running" with "Start Kanata Service" button
- Prevents misleading "all green" when service is actually stopped
π΄ Setup Issues (Red/Gray):
- Missing components or conflicts detected
- Shows specific issues and required actions
For debugging when the comprehensive wizard checks still miss issues:
# 1. Verify driver extension system status
systemextensionsctl list | grep -i karabiner
# Expected: [activated enabled]
# 2. Check system-level Karabiner services
sudo launchctl list | grep -i karabiner
# Expected: system-level daemon entries
# 3. Verify user-level background services
launchctl list | grep -i karabiner
# Expected: user-level service entries
# 4. Check for any grabber conflicts
ps aux | grep karabiner_grabber | grep -v grep
# Expected: empty (no conflicts)
# 5. Test kanata can access HID devices
sudo kanata --cfg /path/to/config.kbd --check
# Expected: no errors, clean validation
# 6. Verify TCC permissions database
sudo sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db \
"SELECT client, allowed FROM access WHERE service='kTCCServiceListenEvent';"
# Expected: KeyPath and kanata entries with allowed=1The biggest breakthrough was realizing that connect_failed asio.system:61 errors don't prevent Kanata from working. Manual testing proved that keyboard remapping continues to function despite these warnings.
Scattered MainActor.run blocks create race conditions. The solution is centralized state management with a single @MainActor function handling all @Published property updates.
The daemon will run and provide basic functionality even with permission warnings. Don't let permission errors in logs mislead you into thinking the system isn't working.
Always implement and document the Kanata emergency sequence (Left Ctrl + Space + Esc). This works even when the keyboard appears completely unresponsive.
When app integration fails, test Kanata manually to isolate whether the issue is with Kanata itself or the app's process management.
Recovery systems that attempt to restart processes must handle concurrent access properly, or they'll crash during the very scenarios they're designed to fix.
- Kanata GitHub - Main keyboard remapper project
- Karabiner-DriverKit-VirtualHIDDevice - macOS HID driver
- IOHIDFamily Documentation - Apple's HID framework
- Swift Concurrency - Thread safety with async/await
When debugging Kanata issues:
- Is Kanata actually processing key events? (Check debug output)
- Are there multiple Kanata instances? (
ps aux | grep kanata) - Is karabiner_grabber running? (
ps aux | grep karabiner_grabber) - Are both system and user grabber services stopped? (
sudo launchctl list | grep karabiner_grabberandlaunchctl list | grep karabiner_grabber) - Is Karabiner daemon running? (
ps aux | grep karabiner) - Does config validate? (
sudo kanata --cfg config.kbd --check) - Are there thread safety crashes? (Check crash reports)
- Can you use emergency sequence? (
Left Ctrl + Space + Esc) - Are VirtualHID connection errors blocking functionality? (Usually no!)
- Is the app's automatic recovery working? (Check logs)
- Driver Extension enabled? (
systemextensionsctl list | grep karabiner) - Background services enabled? (Check Login Items & Extensions - manually add if missing)
- Karabiner Login Items manually added? (
Karabiner-Elements Non-Privileged Agents.appandKarabiner-Elements Privileged Daemons.app) - Input Monitoring granted? (Check System Settings)
If KeyPath reports that Kanata is running and responding but cannot capture keyboard input, treat the state as terminal for the current repair attempt. Do not keep restarting services in the background. Collect diagnostics, then check the manual failure classes below:
- Driver approval loop: open System Settings > General > Login Items & Extensions > Driver Extensions and confirm Karabiner-VirtualHIDDevice is enabled. If macOS keeps disabling it after approval, reboot once before retrying repair.
- Endpoint-security or device-control software: temporarily disable software that can block DriverKit extensions or exclusive keyboard capture, then retry KeyPath repair.
- TCC database desync: remove and re-grant Input Monitoring and Accessibility for KeyPath and Kanata Engine from System Settings. If entries do not appear, reboot and retry the permission grant from the KeyPath wizard.
- BTM/Login Items corruption: run
sfltool resetbtm, reboot, then reopen KeyPath and retry repair. This resets macOS background-task registration state; it is intentionally user-initiated.
If Kanata exits because the generated config is invalid, repair should stop and show the parse error. Fix the rule or reset to the default config, then apply the config again. Restarting services cannot make an invalid config load.
If the bundled Kanata engine is missing from KeyPath.app, reinstall KeyPath from a signed release or run a fresh local deploy from the repository. Repair should not fabricate a replacement binary from another path because that changes the TCC identity and can invalidate Input Monitoring grants.
Remember: Many "errors" in logs are actually warnings. Focus on whether the core functionality (key remapping) is working, not whether all log messages are clean.