GestureToText – Video Gestures to Text

This project builds an end‑to‑end gesture‑to‑text pipeline for accessibility:

• Preprocess raw videos into frames and MediaPipe hand keypoints.
• Run a gesture recognition model (TimeSformer / LSTM over keypoints).
• Convert recognized gesture tokens into natural language text using an LLM.

The implementation is still research/prototype‑stage: no API/frontend wiring yet, but you can run the full pipeline from the command line.

---

1. Setup

From the repo root:

1. (Recommended) Create and activate a virtual environment:
   • python -m venv .venv
   • source .venv/bin/activate (macOS/Linux) or .\.venv\Scripts\activate (Windows)
2. Install dependencies:
   • pip install -r requirements.txt

You also need:

• FFmpeg installed on your system (for robust frame extraction).
• A MediaPipe hand landmark model file:
  • Download hand_landmarker.task from the official MediaPipe repository.
  • Place it anywhere on disk and set:
    • export MP_HAND_LANDMARKER_PATH="/full/path/to/hand_landmarker.task"

Example (matching the current dev setup):

• Model path: models/mediapipe_models/hand_landmarker.task
• Env var:
  • export MP_HAND_LANDMARKER_PATH="$PWD/models/mediapipe_models/hand_landmarker.task"

---

2. Data Layout

The project assumes a WLASL‑style dataset structure. See data/README.md for details. At minimum:

• Raw videos: data/raw_videos/*.mp4
• Extracted frames: data/frames/<video_id>/frame_XXXX.jpg
• Keypoints: data/keypoints/<video_id>.npy

2.1 WLASL Annotations → Exact Sign Labels

We ship the official WLASL annotations file:

• data/annotations/WLASL_v0.3.json

src/utils/wlasl_annotations.py loads this into a mapping:

• video_id (e.g. "00623") → WLASL gloss (e.g. "BOOK" / "book").

The gesture recognition pipeline uses this mapping so that, for WLASL videos, the gesture tokens are the exact sign names, not generic Kinetics‑400 labels such as "sign language interpreting".

Example from the current dataset:

• In data/annotations/WLASL_v0.3.json, the first entry has:
  • gloss: "book"
  • one of its instances has "video_id": "69241".
• If you have data/raw_videos/69241.mp4, then:
  • recognize_gesture_tokens("data/raw_videos/69241.mp4") returns ["book"].
  • Those exact gloss tokens are what get passed into the seq2seq text generator.

Note: Some downloaded .mp4 files can actually be HTML error pages or truncated videos. Those will fail preprocessing; use valid, playable videos for experiments.

---

3. What Works Today

Implemented components:

• Preprocessing

  • src/preprocessing/extract_frames.py – OpenCV‑based frame extraction for all videos in data/raw_videos.
  • src/preprocessing/extract_frames_ffmpeg.py – FFmpeg‑based frame extraction (extract_frames_ffmpeg, batch_extract_ffmpeg).
  • src/preprocessing/extract_keypoints.py – MediaPipe hand keypoints (21 landmarks → 63‑dim vector per frame), compatible with both legacy mp.solutions and newer mp.tasks APIs (using hand_landmarker.task).

• Gesture Recognition

  • src/training/gesture_training_wandb.py – LSTM classifier over keypoint sequences with optional Weights & Biases logging.
  • src/gesture_recognition/timeSformer_model.py – TimeSformer wrapper and loader for video‑based gesture recognition.
  • src/gesture_recognition/infer.py – Runs TimeSformer on a single video, returns gesture token(s).

• Text Generation

  • src/text_generation/generate_text.py – LLM wrapper (Hugging Face Transformers) to turn gesture tokens (+ context) into natural language.
  • src/text_generation/finetune_lora.py – LoRA fine‑tuning utility for the LLM on (gesture_tokens, text) pairs.

• End‑to‑End Pipeline

  • src/pipeline/gesture_to_text.py – High‑level function gesture_video_to_text(...) combining gesture recognition and LLM text generation.
  • src/cli.py – Single command entrypoint: raw video → frames → keypoints → gesture tokens → text.

• Optimization Utilities

  • src/optimization/export_and_optimize.py – Export models to TorchScript / ONNX and run ONNX inference.

MCP integration, API endpoints, and frontend are not wired up yet.

---

4. How to Run the Project

4.1 One‑shot command: raw video → text

From the repo root (with env activated, dependencies installed, FFmpeg + MediaPipe model in place):

• Basic usage:
  • python -m src.cli data/raw_videos/00623.mp4
• Prefer FFmpeg for frame extraction:
  • python -m src.cli data/raw_videos/00623.mp4 --use-ffmpeg

What this does:

• Ensures data/raw_videos, data/frames, data/keypoints exist.
• Extracts frames for the given video into data/frames/00623/.
• Extracts MediaPipe hand keypoints and saves data/keypoints/00623.npy.
• Runs gesture recognition on the video and feeds gesture tokens into an LLM.
• Prints the generated text under === Generated Text ===.

Useful flags:

• --context "some context" – Additional natural‑language context for the LLM.
• --gesture-pretrained-name – Override the TimeSformer checkpoint.
• --llm-name – Choose a different Hugging Face seq2seq model.

4.2 Manual preprocessing only

If you just want frames + keypoints:

1. Extract frames for all videos (OpenCV):
   • python src/preprocessing/extract_frames.py
2. Or use FFmpeg:
   • python -c "from src.preprocessing.extract_frames_ffmpeg import batch_extract_ffmpeg; batch_extract_ffmpeg()"
3. Extract hand keypoints:
   • python src/preprocessing/extract_keypoints.py

---

5. Optional: Training and Experiments

• Train an LSTM gesture classifier on keypoints:

  • python -c "from src.training.gesture_training_wandb import train_gesture_model; train_gesture_model()"
  • Add wandb_project="your_project" argument if you want W&B logging.

• Fine‑tune the LLM with LoRA on custom (gesture_tokens, text) pairs:

  • Use src/text_generation/finetune_lora.py from a notebook or small script to construct training examples and call finetune_llm_with_lora(...).

These pieces are optional for running the basic gesture‑to‑text pipeline but support further experimentation and improvement.