ML-app

Real-time face + hand tracking with mouse control and gesture recognition, built as an end-to-end ML project: data collection → feature engineering → training → inference → deployment (Docker + CI).

Built on the MediaPipe Tasks API with a scikit-learn gesture classifier.

Features

• Live hand landmark tracking (21 landmarks, up to 2 hands) with skeleton overlay
• Face detection with bounding boxes and confidence scores
• Mouse control: index finger moves the cursor, bending the pinky clicks
• Exponential cursor smoothing and edge margins to reduce jitter
• Custom gesture classifier (RandomForest) trained on your own data
• Data engineering pipeline: sample collection, normalised features, train/test splits
• Fully typed package, unit-tested, linted, containerised, CI on GitHub Actions

Pipeline

camera ──► MediaPipe detection ──► feature extraction ──► gesture classifier
              │                             │
              └──► face detection ──────────┴──► mouse control + HUD

See docs/ARCHITECTURE.md for details.

Project layout

ML-app/
├── src/ml_app/              # Python package
│   ├── cli.py               # `ml-app` entry point
│   ├── config.py            # typed settings
│   ├── train.py             # gesture classifier training/eval
│   ├── train_cli.py         # `python -m ml_app.train_cli`
│   ├── collect.py           # data collection
│   ├── control/mouse.py     # cursor mapping + click detection
│   ├── data/                # feature engineering + dataset persistence
│   ├── models/              # MediaPipe asset download/lookup
│   └── tracking/            # hand/face detection wrappers + drawing
├── config/example.yaml      # example configuration
├── data/raw/gestures/       # collected training samples (.npy)
├── models/                  # model assets (downloaded, gitignored)
├── scripts/                 # helper scripts
├── tests/                   # unit tests (pytest)
├── docs/                    # architecture docs
├── Dockerfile
└── .github/workflows/ci.yml # CI pipeline

Quick start

1. Install

pip install -e ".[dev]"      # or: pip install -r requirements.txt

On Debian/Ubuntu you may also need OpenCV runtime libs:

sudo apt install libgl1 libglib2.0-0 libsm6 libxext6 libxrender1 libgomp1

2. Download model assets

ml-app download-models        # or: python -m ml_app.download_models

Model files are written to models/ (gitignored).

3. Run

ml-app run                    # camera index 0
ml-app run --camera 1

Action	Gesture
Move the mouse	Move index finger
Left click	Bend pinky finger
Emergency stop	Move mouse to top-left corner
Quit	Press ESC

Train your own gesture classifier

a) Collect data

Show a gesture in front of the camera and let the app record it:

python -m ml_app.collect --label thumbs_up --samples 120
python -m ml_app.collect --label peace      --samples 120
python -m ml_app.collect --label fist       --samples 120

Samples (normalised 42-dim feature vectors) are saved to data/raw/gestures/<label>/.

b) Train

python -m ml_app.train_cli --data-dir data/raw \
    --out models/gesture_classifier.pkl

Prints accuracy, per-class report, and confusion matrix.

c) Inference

The gesture model is loaded at runtime for live classification when models/gesture_classifier.pkl exists.

Tests & quality

pytest --cov=ml_app          # run the test suite
ruff check src tests         # lint
mypy src                     # type-check

Docker

docker build -t ml-app .
# Mount the model assets and pass through the webcam + display:
docker run --rm \
  -v "$PWD/models:/app/models" \
  --device /dev/video0 \
  -e DISPLAY="$DISPLAY" -v /tmp/.X11-unix:/tmp/.X11-unix \
  ml-app run

License

MIT — see LICENSE.