Website Sign Language Translator (BISINDO)

A real-time BISINDO (Indonesian Sign Language) translator web application featuring interactive 3D hand visualization and on-the-fly AI model training.

🌟 Features

• Real-Time Translation: Instantly translates BISINDO sign language gestures into text using a webcam.
• 3D Hand Tracking Viewer: Visualizes hand joints and bones in 3-dimensional space using Three.js and MediaPipe.
• AI Model Training Capability: Upload new datasets and retrain the sign language recognition model directly.
• WebSocket Communication: Low-latency, bidirectional streaming of hand landmark data between the frontend and backend.

🛠️ Technology Stack

• Frontend: Next.js 14, React 18, Tailwind CSS, Three.js, MediaPipe (@mediapipe/hands).
• Backend: FastAPI, TensorFlow (Keras), scikit-learn, Uvicorn, WebSockets.
• Tunneling: ngrok (for exposing the cloud-based backend).

🚀 Getting Started

The project is designed to run the frontend locally while the backend is hosted on the cloud (e.g., Google Colab) to leverage GPU acceleration for the TensorFlow model.

1. Backend Setup (Cloud / Colab)

The backend code is contained in aibackend.py, which is originally a Google Colab notebook script.

1. Open a Google Colab environment with a GPU instance.
2. Run the aibackend.py contents in the notebook.
3. Provide your NGROK_AUTH_TOKEN and NGROK_STATIC_DOMAIN when prompted in the script.
4. The server will start and expose a secure ngrok URL that the frontend will use to connect.

2. Frontend Setup (Local)

1. Clone the repository and navigate to the project directory:

   git clone <your-repo-url>
   cd WebsiteSignLanguageTranslator-Jass

2. Install frontend dependencies:

   npm install

3. Configure your environment variables: Create a .env.local file in the root directory and add the ngrok URL generated from your backend setup. For example:

   NEXT_PUBLIC_WS_URL=wss://<your-ngrok-domain>
   NEXT_PUBLIC_API_URL=https://<your-ngrok-domain>

4. Start the local development server:

   npm run dev

5. Open http://localhost:3000 in your browser.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.