After watching countless videos on Youtube on "how to build a sign language interpreter" I found a few problems. The biggest one was that almost every solution used images instead of landmarks, not accounting for changes in lighting, distance from camera and skin-tones for the end users.
This project aims to solve that problem by using Mediapipe's landmarks feature.
I'm estimating the distance of each landmark of the hand from the center of the user's wrist.
After normalizing that distance, I trained a simple model with sci-kit-learn in train.py.
(please excuse my camera death stare)
You can train a similar model by:
-
running csv-write.py (make signs for each alphabet, essentially capturing your training data)
-
running merge-data.py (
call it lack of foresight but thisconverts the csv data into our training data) -
train our model with train.py
-
run the trained model with pred.py
Future scope:
As this can work only with alphabets now, I think it can be improved by incorporating Long Short-Term Memory ([LTSM](Long short-term memory - Wikipedia)) to add gesture support so users can interpret full sentences.
Other than that, I think by using Mediapipe's holistic model, instead of Hands model, multi-hand gestures, along with facial gestures can be incorporated.