MediaPipe Hand-Gesture to Speech Pipeline

A real-time Computer Vision prototype that converts hand gestures into Morse-based text and speech output through a desktop interface and live webcam runtime.

Demo Preview

Demo

• YouTube demo: https://youtu.be/aonUtd7KMhQ?si=GLZkGS5FR3CsJ0ZU

Highlights

• Real-time hand tracking with MediaPipe Hands
• Finger-count-based gesture interaction
• Morse encoding and text decoding pipeline
• Speech output through text-to-speech
• Desktop GUI for preview and runtime launch
• Classic Computer Vision experiments for comparison and course demonstration

Why This Project

This project was developed as an academic Computer Vision prototype to explore gesture-based interaction, real-time hand tracking, and alternative communication pipelines.

Instead of aiming at full sign-language translation, it focuses on a simplified but technically explicit flow:

hand gesture -> finger count -> Morse-based encoding -> decoded text -> speech output

Pipeline

Camera -> MediaPipe Hands -> Finger Counting -> Morse Encoding -> Text Decoding -> Speech Output

Realtime Interaction Rules

The realtime runtime includes a few gesture rules in addition to basic finger counting:

• A gesture is committed after a short stability window of about 250 ms
• Gesture 5 acts as a space
• Holding gesture 0 for about 2.5 seconds clears the current sequence and decoded sentence
• Holding digits 1 to 4 repeats the same digit, making consecutive equal inputs possible without lowering the hand each time

Tech Stack

• Python
• OpenCV
• MediaPipe
• Pillow
• customtkinter
• pyttsx3
• Computer Vision

Repository Contents

• src/gesture_morse_speech/: core application code
• experiments/: classic Computer Vision demos
• docs/: architecture and technical notes
• media/: presentation slides and project media
• models/: MediaPipe model asset
• run_app.py: desktop GUI launcher
• run_realtime.py: direct realtime runtime launcher

Project Structure

gesture-morse-speech/
|-- assets/
|-- docs/
|-- experiments/
|-- media/
|   |-- slides/
|   `-- videos/
|-- models/
|-- src/
|   `-- gesture_morse_speech/
|       |-- app/
|       |-- audio/
|       |-- config/
|       |-- encoding/
|       |-- gui/
|       |-- utils/
|       `-- vision/
|-- tests/
|-- README.md
|-- requirements.txt
|-- run_app.py
`-- run_realtime.py

Environment Setup

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Run Commands

Desktop application:

python run_app.py

Realtime pipeline directly:

python run_realtime.py

Unit tests:

python -m unittest tests.test_encoding

Academic Prototype

This repository is structured as an academic prototype and portfolio-ready codebase. The focus is on clarity of pipeline, modularity, and reproducibility of the current project behavior.

Scope

This project is a gesture-based academic prototype. It does not implement full sign-language recognition or natural-language translation.

Privacy / Face Anonymization

For presentation material and visual assets, faces were anonymized using adaptive Gaussian blur proportional to face size. The configuration used was blur_ratio = 0.35, with a minimum kernel size of 45x45 and effective kernels ranging from 45x45 up to 159x159, depending on face dimensions.

Media Placement

Recommended locations for project materials:

• videos: media/videos/
• PowerPoint slides: media/slides/

Limitations

• The gesture-to-Morse mapping is simplified
• Robustness depends on lighting, camera placement, and hand visibility
• The current Morse mapping used by the project is not fully aligned with standard Morse code for several symbols. See docs/MORSE_MAPPING.md

Documentation

• docs/ARCHITECTURE.md
• docs/MORSE_MAPPING.md
• docs/PRIVACY.md
• docs/SETUP.md
• docs/EXPERIMENTS.md