Hand & Face Mouse Controller

A Python application that lets you control your mouse using hand gestures and facial expressions. Point with your finger to move the cursor, make a fist to click, and stick your tongue out to toggle control on/off. Built with computer vision and real-time hand tracking.

Description

This is a hands-free mouse controller that uses your webcam to track your hand movements and facial expressions. Instead of using a physical mouse, you can control your cursor by pointing with your index finger. Make a fist to click, and stick your tongue out when you want to enable or disable mouse control. It's pretty fun and surprisingly useful!

Features

• Hand Tracking: Uses MediaPipe's hand landmark detection to track your hand in real-time
• Finger-Based Cursor Control: Point with your index finger to move the mouse cursor
• Fist Click: Close your hand into a fist to perform a click
• Tongue Toggle: Stick your tongue out to turn mouse control on/off
• Smart Manual Override: Automatically pauses hand tracking when you use your physical mouse
• Visual Feedback: Shows hand landmarks, click counter, and control status on screen
• Smooth Movement: Applies smoothing to cursor movement so it's not all jittery

Technologies Used

• Python 3: Programming language
• OpenCV: For webcam access and video processing
• MediaPipe: For hand and face landmark detection
• NumPy: For calculations and distance measurements
• PyAutoGUI: For controlling the mouse cursor and performing clicks

How It Works

1. Hand Detection: MediaPipe detects your hand and tracks 21 landmark points
2. Finger Position: The index finger tip position is used to control cursor location
3. Fist Detection: Checks if all fingers are closed by measuring distances to the palm
4. Face Detection: MediaPipe detects facial landmarks to check for mouth opening (tongue out)
5. Tongue Toggle: Opens mouth wide = tongue out = mouse control enabled
6. Mouse Mapping: Hand position in camera view is mapped to screen coordinates
7. Smoothing: Applies a smoothing algorithm to make cursor movement feel natural
8. Manual Override: If you move your physical mouse, hand tracking automatically pauses for 3 seconds

Installation

1. Clone or download this repository
2. Install required packages:

   pip install -r requirements.txt

How to Run

1. Navigate to the project directory
2. Run the main script:

   cd /tmp/eye_mouse_controller
   venv/bin/python main.py

3. Allow camera access when prompted (on macOS, you'll need to grant Accessibility permissions)
4. Your webcam will open and start tracking
5. Stick your tongue out to activate mouse control
6. Point with your index finger to move the cursor
7. Make a fist to click
8. Press ESC to quit

To Stop the Program

If you need to force quit:

pkill -f "main.py"

Usage Tips

• Make sure you have good lighting so the camera can see your hand clearly
• Keep your hand in view of the camera at all times
• The program works best when you're sitting about 2-3 feet from the webcam
• If the cursor is moving too fast or slow, you can adjust the smooth_val parameter in the code
• The fist gesture needs to have most fingers closed - doesn't have to be perfect
• For tongue detection, you just need to open your mouth wide enough

macOS Permissions

On macOS, you need to grant Accessibility permissions:

1. Open System Preferences → Security & Privacy → Privacy → Accessibility
2. Click the lock and enter your password
3. Add Terminal (or your terminal app) to the list and check the box
4. Restart the program

Configuration

You can tweak these settings in main.py:

• smooth_val: Default 5 (lower = smoother movement, higher = more responsive)
• fist_threshold: Default 2 frames (how long you need to hold a fist to click)
• manual_mouse_threshold: Default 50 pixels (sensitivity for detecting manual mouse use)

Future Ideas

• Add different hand gestures for right-click, double-click, drag operations
• Implement scrolling with hand movements
• Add a calibration mode for better accuracy
• Create keyboard controls using other gestures
• Add support for two-handed controls
• Make a simple GUI for settings
• Add voice commands as backup controls

Limitations

• Needs decent lighting to detect hand properly
• Camera FPS affects how smooth the control feels
• May struggle with very cluttered backgrounds
• Tongue detection isn't perfect - sometimes thinks you're sticking your tongue out when you're just talking
• Can be tiring to hold your hand up for extended periods

Author

Muhammad Ammar Malick

---

This is a portfolio project demonstrating computer vision and human-computer interaction concepts.