Real-Time Hand Gesture Rotary Controller & Media Interface

A touchless, vision-based Human-Computer Interaction (HCI) utility developed in Python. The system tracks 21 3D hand landmarks via MediaPipe and processes frame geometry in real time to simulate an ergonomic rotary dial knob for Windows master volume control, alongside gesture-triggered media playback commands.

---

Key Features

• Ergonomic Rotary Control (Pinch & Twist): Mimics a physical rotary knob using vector angles between the thumb and index finger.
• Relative Delta Angle Tracking: Tracks frame-by-frame angular variations rather than absolute coordinates, preserving volume state between disengagements.
• Deadzone & Jitter Mitigation: Integrates an angular threshold filter to reject micro-jitter and landmark flickering.
• Fist-Proof Pinch Detection: A closed fist can never enter grab mode, and a pinch must hold for consecutive frames before engaging.
• Hardware-Agnostic Finger Detection: Finger states are computed from PIP joint angles (and thumb distance ratios), so gestures keep working regardless of hand rotation or handedness.
• Self-Normalizing Pinch Detection: The pinch threshold scales with the detected hand size, making it robust to camera distance and resolution.
• Dynamic HUD Feedback: Renders a responsive, circular arc display and live volume percentage pinned to the hand's center point.
• Media Shortcut Gestures:
  • Fist: Play / Pause toggle
  • Peace Sign (V-Sign): Next Track
  • Three Extended Fingers: Previous Track
  • Open Palm: Mute / Unmute toggle
• Semi-Transparent Legend Overlay: Real-time on-screen guide detailing active gesture states and key mappings.

---

Technical Architecture & Mathematical Logic

1. Rotary Angle Computation

The knob mechanism relies on the vector formed by the thumb tip $(x_t, y_t)$ and index finger tip $(x_i, y_i)$:

$$\theta = \text{atan2}(y_t - y_i, x_t - x_i)$$

To eliminate coordinate flipping across the 180° boundary, angular change is normalized:

$$\Delta\theta = \theta_{\text{current}} - \theta_{\text{previous}}$$

$$\Delta\theta \leftarrow ((\Delta\theta + 180^\circ) \pmod{360^\circ}) - 180^\circ$$

2. Ergonomic Sensitivity & Deadzone

Small tremor artifacts are suppressed by requiring $\vert{}\Delta\theta\vert{} \ge \tau$ (where $\tau = 2.0^\circ$). Detected angular delta is scaled by an ergonomics factor ($S = 0.8$) to allow a full 0–100% adjustment without straining wrist joints.

3. Finger Extension Logic

Instead of comparing raw vertical coordinates, each finger's state is derived from its PIP joint angle (the angle formed at the second knuckle), which is orientation-independent:

$$\text{angle} = \arccos\left(\frac{\vec{v}_1 \cdot \vec{v}_2}{\lvert\vec{v}_1\rvert \lvert\vec{v}_2\rvert}\right)$$

Finger State = 1 (extended) when the PIP angle $\ge 140°$, otherwise 0 (folded). The thumb is classified by a distance ratio (thumb tip vs. thumb IP joint, both measured from the index MCP), which works for either hand orientation.

4. Pinch Detection

The pinch threshold is a ratio, normalized by hand size (wrist → middle MCP), so it stays valid regardless of camera distance or resolution. A pinch only engages when the ratio falls below 0.45 and the four fingers are not all folded (a fist), and only after it holds for 2 consecutive frames.

---

Dependencies & Installation

Requirements

• Python 3.10 / 3.11 (MediaPipe 0.10.x does not support 3.12+)
• Windows OS (for core audio endpoints via pycaw)
• Webcam or virtual camera stream (e.g., Camo)

Setup

# Clone the repository
git clone https://github.com/nz-tx/vision-volume-controller.git

# Install pinned dependencies
pip install -r requirements.txt

Usage

python main.py

Press Q to exit safely (releases camera and audio endpoints).

Selecting Camera Sources

If using an external device or virtual camera (e.g., Camo, DroidCam, Iriun), adjust CAMERA_INDEX in main.py:

CAMERA_INDEX = 1  # 0 = built-in webcam, 1 = virtual/external device

---

Configuration

Tune behavior at the top of main.py:

Constant	Default	Meaning
CAMERA_INDEX	0	Camera source index
TARGET_FPS	30	Max loop framerate (CPU saving)
SENSITIVITY	0.8	Degrees of rotation → % volume
DEADZONE_DEG	2.0	Minimum accepted angular delta (jitter filter)
PINCH_RATIO	0.45	Pinch threshold (normalized by hand size)
PINCH_HOLD_FRAMES	2	Consecutive frames a pinch must hold
ACTION_COOLDOWN	1.2	Seconds between gesture-triggered actions
EXTEND_ANGLE	140	PIP angle threshold for "extended" fingers
THUMB_RATIO	1.2	Thumb distance ratio for "extended" thumb
STATUS_DURATION	1.5	Seconds before the status label resets