AirOS is a native, futuristic desktop application that turns your laptop webcam into a high-precision spatial interaction controller and air touchpad. By tracking 3D hand landmarks in real time using MediaPipe and custom gesture state machines, AirOS allows you to control the OS cursor, click, double-click, right-click, drag & drop, and scroll fluidly through the air without touching any physical hardware.
AirOS replaces standard physical mouse interactions with a comprehensive suite of spatial hand gestures. Each gesture is backed by dedicated state machines, debounce timers, dropout tolerances, and adaptive smoothing to prevent false triggers and cursor jitter.
| Gesture | Pose / Trigger | Action / Behavior |
|---|---|---|
| ☝️ Aim & Track | Index Finger Extended (Pointing Pose) |
Moves the OS cursor across the screen with adaptive One Euro smoothing and dynamic gain. |
| ✌️ Cursor Lock | Index + Middle Fingers Extended (Peace / V Pose) |
Freezes and locks the cursor exactly in place. Required before clicking to prevent cursor drift. |
| ✊ Left Click | Close Fist (Strict Fist Pose) while Locked |
Dispatches a native OS left-click at the locked coordinates after a brief confirmation hold. |
| 👍 Double Click | Thumbs Up (👍) after Selecting a Target |
Fires the second click of a double-click to open folders, files, or shortcuts without moving the cursor. |
| 🤏 Right Click | Pinch Thumb & Middle Finger (~200ms hold) |
Dispatches a native OS right-click menu at the cursor position with tracking dropout tolerance. |
| 🤌 Drag & Drop | Pinch Thumb & Index Finger | Holds left mouse button (DRAG_START), unlocks cursor for active dragging across windows, and releases on pinch open (DRAG_END). |
| 3 Fingers Up (Index, Middle, Ring extended) | Smooth, proportional vertical scrolling relative to a neutral hand anchor with automatic recentering. | |
| ⚡ Onboarding UI | Glassmorphism Interactive Guide | JARVIS-inspired tutorial window displaying all gesture cards and instructions prior to camera activation. |
| 🧭 Floating HUD | Always-on-Top Bottom-Left Guide | Collapsible, draggable quick-reference pill showing active gesture states and visual cues. |
| 📊 Vision Dashboard | Real-Time Camera Feed & Status Panel | Live telemetry displaying One Euro tracking status, finger fold detection, FPS, and debug overlay toggles. |
Traditional air-tracking systems suffer from "click drift" — the natural movement of your finger curling to click causes the cursor to shift slightly off target right before the click registers. AirOS eliminates this with a mandatory 3-stage state machine:
- Aim & Track (
CURSOR_TRACKING): Extend your index finger while curling your middle, ring, and pinky fingers. The cursor tracks landmark 8 (index finger tip) with One Euro smoothing and boundary protection. - Lock Cursor (
LOCK_MODE): When hovering over your intended target, extend your middle finger alongside your index finger into a Peace Sign / V Gesture (✌️). After a brief debounce (LOCK_DEBOUNCE_FRAMES), AirOS freezes and snapshots the exact pixel coordinates (_snap_x,_snap_y). The cursor is now locked in place, allowing you to move your hand freely without dislodging the target. - Left Click (
CLICK_TRIGGERED): While the cursor is frozen inLOCK_MODE, close your hand into a Strict Fist (✊). Holding the fist briefly (FIST_HOLD_MS) dispatches a clean OS left-click right on target. Opening your hand (WAIT_FOR_HAND_OPEN) resets the state machine for the next movement.
Opening items like desktop shortcuts, folders, or files requires two clicks at the exact same pixel. AirOS handles this seamlessly:
- Target Heuristic Detection: When
SingleTapGesturefires a left-click, AirOS checks if the element under the cursor is a double-click target (e.g., desktop icons, folder views, or file explorer entries vs. single-click UI buttons/tabs). - Cursor Anchor Freeze: If a double-click target is clicked, the cursor remains frozen at those exact coordinates (
_click_x,_click_y) and entersWAIT_FOR_THUMBS_UP. - Thumbs Up Confirmation (👍): Extend your thumb upward (
Thumbs Up Pose) for ~75ms (THUMBS_UP_HOLD_MS). AirOS instantly fires the second click of the double-click at the anchored coordinates, immediately opening your item and unfreezing the cursor. If no thumbs up is shown within the timeout window (THUMBS_UP_TIMEOUT_MS), the cursor unlocks automatically.
- Middle Pinch Activation: Bring the tips of your Thumb and Middle Finger (🤏) together (
middle_pinch_ratio_smooth < MIDDLE_PINCH_THRESHOLD). - Dropout-Tolerant Hold: Hold the pinch for ~200ms (
PINCH_HOLD_MS) to dispatch a native OS right-click context menu. AirOS includes a dropout tolerance filter (_PINCH_DROPOUT_TOLERANCE_FRAMES) that absorbs momentary tracking flickers or lighting reflections so the gesture doesn't reset when almost complete.
- Index Pinch to Drag: Pinch your Thumb and Index Finger (🤌) together while keeping your middle finger slightly separated.
- Instant Lock Moment (
LOCK_MOMENT): Upon detection, AirOS presses the left mouse button down (DRAG_START) and momentarily locks the cursor (LOCK_MOMENT_MS) to ignore hand tracking jitter during the initial pinch. - Active Drag (
DRAG_ACTIVE): The cursor automatically unlocks while keeping the mouse button pressed. Move your hand normally across any screen or application window to drag the selected item, text, or window. - Grace Period Protection: If your hand temporarily leaves the camera frame or tracking is momentarily lost while dragging, AirOS enters a safety grace period (
DRAG_GRACE_PERIOD_MS). If tracking is recovered quickly, the drag continues without dropping your item prematurely. - Release: Open your thumb-index pinch to release the left mouse button (
DRAG_END) and drop the item exactly where the cursor rests.
- Three-Finger Extended Pose: Extend three fingers upward — Index, Middle, and Ring (
↕️ ) — while keeping your Thumb and Pinky folded (SCROLL_EXTENDED_RATIO). - Neutral Anchor Initialization: Once held for ~150ms (
SCROLL_HOLD_MS), AirOS captures your initial hand/wrist vertical position as a Neutral Anchor Point (neutral_y). - Proportional Velocity Control:
- Move your hand/wrist slightly upward above the anchor point to smoothly scroll up.
- Move your hand/wrist slightly downward below the anchor point to smoothly scroll down.
- Dead Zone & Automatic Recentering: A calibrated dead zone (
SCROLL_DEAD_ZONE) surrounds the neutral anchor to eliminate idle drift. The speed scales dynamically via linear and quadratic sensitivity curves, while the anchor slowly recenters (recenter_alpha) so your hand stays comfortably centered in your webcam view during long web browsing sessions.
When launching AirOS, the system presents a futuristic, dark-themed glassmorphism tutorial dialog before initializing camera hardware. It features:
- Glowing cyan line-art hand illustrations (
HandIconWidget). - An interactive 2×3 card grid outlining every pose (
Aim,Left Click,Double Click,Right Click,Drag & Drop,Scroll). - A smooth transition into live tracking mode upon confirmation.
- Always-on-Top & Draggable: Anchors automatically to the bottom-left corner of your screen as a translucent reference overlay during live operation.
- Collapsible Pill Mode: Click the
−/+toggle button (or anywhere on the minimized pill) to collapse the full gesture list into a compact, unobtrusive⚡ AirOS Guidepill. - Window-Independent: Can be dragged anywhere on the screen without interfering with active applications or capturing keyboard focus.
- Real-Time Video Feed: Displays your webcam feed with high-visibility MediaPipe hand skeleton rendering and bounding box overlays (
CameraWidget). - Live Diagnostics: Monitors frame rate (
FPS), active tracking phase (IDLE,CALIBRATING,TRACKING,GRACE,RECOVERY), confidence scores, and individual finger fold states (I:F M:O R:O P:F). - Debug Overlay Toggle: Check the "Debug Overlay" checkbox directly in the dashboard to render real-time state machine badges, pinch ratios, and threshold indicators onto your video feed.
- Adaptive One Euro Smoothing: Dual low-pass filters (
PointSmoother) dynamically filter hand coordinates. At low speeds, heavy filtering eliminates micro-jitters; during fast hand sweeps, latency drops to zero for instantaneous responsiveness. Separate drag smoothing parameters (ONE_EURO_MIN_CUTOFF_DRAG) ensure precision drop accuracy. - Ergonomic Interaction Bounding Box (
InteractionBox): Maps a comfortable rectangular region in the center of the webcam frame directly to your full desktop resolution (1920×1080,2560×1440, etc.). You can navigate corner-to-corner across your monitor with subtle hand movements without stretching or arm fatigue. - Dynamic Gain & Edge Boost: Applies non-linear velocity acceleration (
apply_acceleration) and edge boosting (edge_boost) to make hitting screen corners effortless while keeping fine movements steady. - Grace & Recovery States: Automatically recalibrates and re-anchors tracking if your hand is temporarily obscured or moves out of bounds (
CURSOR_LOST_GRACE_MS).
- OS: Windows 10 or Windows 11
- Python: Python 3.10 or higher
- Hardware: Integrated laptop webcam or USB external webcam
-
Clone the repository and enter the directory:
cd AirOS
-
Install Python dependencies:
python -m pip install -r requirements.txt
-
Generate application icons and assets:
python scripts/generate_icon.py -
Launch AirOS:
python main.py
Or run directly using the included PowerShell / Batch shortcuts:
.\scripts\run.ps1
# OR
.\scripts\run.batTo compile AirOS into a standalone, portable Windows desktop executable without requiring Python pre-installed:
.\scripts\build_windows.ps1Output: dist/AirOS/AirOS.exe
AirOS/
├── main.py # Application entry point & initialization
├── requirements.txt # Project dependencies (OpenCV, MediaPipe, PySide6, PyAutoGUI)
├── assets/ # Icons, images, and UI themes
├── backend/
│ ├── app_controller.py # Central orchestrator connecting vision, gestures, and UI signals
│ ├── cursor_tracking/
│ │ ├── tracker.py # Core tracking phases (IDLE, CALIBRATING, TRACKING, GRACE, RECOVERY)
│ │ ├── smoothing.py # One Euro adaptive low-pass filters & PointSmoother
│ │ ├── mapper.py # Non-linear gain, dead zone, and velocity acceleration mapping
│ │ ├── interaction_box.py # Ergonomic coordinate mapping and edge boosting
│ │ └── config.py # Tracking sensitivities, cutoffs, and calibration constants
│ ├── gestures/
│ │ ├── gesture_manager.py # Multi-gesture router, priority resolution, and OS action dispatch
│ │ ├── base_gesture.py # Abstract base class for all stateful gesture detectors
│ │ ├── gesture_types.py # Shared data structures (GestureAction, DebugInfo, ActionType)
│ │ ├── single_tap/ # Aim / Peace Lock / Fist Left Click state machine & detector
│ │ ├── thumbs_open/ # Thumbs Up second-click double-click workflow & heuristics
│ │ ├── right_click/ # Thumb + Middle finger pinch detector with dropout tolerance
│ │ ├── drag_drop/ # Thumb + Index finger pinch drag-and-drop state machine
│ │ └── scroll/ # 3-Finger vertical proportional air scrolling detector
│ ├── mouse_control/
│ │ ├── click_controller.py # Native OS mouse click, double click, right click, and scroll dispatch
│ │ └── cursor_controller.py # Cursor position orchestration and drag mode toggles
│ ├── vision/
│ │ ├── camera.py # Threaded OpenCV camera capture & frame stabilization
│ │ ├── hand_tracker.py # MediaPipe Hands 21-landmark 3D tracking pipeline
│ │ └── landmarks.py # Landmark feature extractor (angles, ratios, velocities, fold states)
│ ├── ui/
│ │ ├── main_window.py # Primary PySide6 dashboard window
│ │ ├── onboarding_window.py # Futuristic glassmorphism gesture tutorial dialog
│ │ ├── gesture_guide_widget.py # Always-on-top draggable floating HUD reference pill
│ │ ├── camera_widget.py # Video rendering canvas with skeleton drawing
│ │ ├── status_panel.py # Telemetry dashboard, FPS meter, and debug switches
│ │ └── overlay.py # On-frame debug badges, state indicators, and bounding box drawing
│ └── utils/
│ ├── config_loader.py # Runtime state and application configuration settings
│ └── logger.py # Structured logging setup and formatting
└── scripts/
├── build_windows.ps1 # PyInstaller executable build script
├── generate_icon.py # App icon generator
├── run.ps1 # PowerShell launch helper
└── run.bat # Batch launch helper
| Symptom / Issue | Recommended Action |
|---|---|
| Camera Not Found / Black Screen | Close other applications currently using your webcam (Zoom, Teams, OBS). Check USB connections if using an external webcam. |
| Cursor Does Not Move in Certain Apps | Run AirOS as Administrator. Windows security policies (UIPI) block simulated mouse clicks inside elevated/admin windows unless the controller is also running as admin. |
| Jittery or Shaking Cursor | Ensure adequate room lighting so MediaPipe clearly distinguishes hand landmarks. Keep your hand inside the comfortable center interaction box rather than at the extreme screen edges. |
| False Clicks / Drifting on Click | Remember the Mandatory Lock Mode (✌️): Always show the Peace/V sign to freeze the cursor right over your target before closing into a fist (✊) to click. |
| Double Clicks Not Opening Folders | After making your initial selection click, hold your hand steady and extend your thumb into a clean Thumbs Up (👍) pose for ~75ms. Make sure your hand is well-lit. |
| Drag Dropping Prematurely | Ensure your thumb and index finger remain firmly pinched (🤌) while dragging. If your hand leaves the frame, AirOS gives you a brief grace window (~300ms) to re-enter before releasing the drop. |
- Python 3.10+ — Core programming language & async threading
- OpenCV (
cv2) — Real-time threaded video capture and frame pre-processing - MediaPipe Hands — High-speed 3D spatial 21-landmark hand skeleton tracking
- PySide6 (
Qt6) — Modern, GPU-accelerated hardware-translucent UI and custom widgets - PyAutoGUI & Win32 API — Low-level OS cursor movement and native mouse click dispatch
- NumPy & One Euro Filters — Advanced spatial signal processing, coordinate mapping, and low-pass smoothing