Japanese Tutor

An interactive, AI-powered Japanese language learning application featuring a talking 3D avatar with real-time lip-sync. The tutor streams responses from a local LLM and speaks using a Text-to-Speech (TTS) engine, with animations synchronized dynamically using forced-alignment.

---

🏗️ Project Architecture

The project is split into two main components:

1. Backend (/backend): A FastAPI application that handles:
   • Streaming LLM responses (using LM Studio or any OpenAI-compatible API).
   • Text-to-Speech (TTS) synthesis (using Kokoro-TTS or gTTS).
   • Audio-to-phoneme/viseme alignment (using Wav2Vec2 or Rhubarb Lip Sync).
   • WebSocket connection for real-time conversation and synchronized playback.
2. Frontend (/frontend): A React + TypeScript + Vite application featuring:
   • A 3D avatar rendered with Three.js (React Three Fiber & Drei) in the browser.
   • Real-time viseme/morph target manipulation for lip-sync.
   • A modern, responsive web chat interface.

---

📋 Prerequisites

Before setting up the project, ensure you have the following installed:

• Node.js (v18 or higher)
• Python (v3.14 or higher is recommended; standard Python >= 3.10 works too)
• uv (Strongly recommended - an extremely fast Python package installer and resolver)
• Rhubarb Lip Sync - (Optional, but recommended, the results are better than Wav2Vec2, get it from https://github.com/DanielSWolf/rhubarb-lip-sync) generates lip sync information (expected to be available as a CLI tool system-wide as rhubarb, include it in the PATH of the system)
• LM Studio (Optional, to run the LLM locally)

---

🔧 Backend Setup & Configuration

1. Installation

Navigate to the backend directory:

cd backend

We recommend using uv to manage virtual environments and dependencies:

# Sync dependencies and create a virtual environment automatically
uv sync

Alternatively, if you prefer standard pip:

python -m venv .venv
# Activate the environment:
# On Windows (PowerShell):
.venv\Scripts\Activate.ps1
# On macOS/Linux:
source .venv/bin/activate

pip install -e .

2. Environment Configuration

Create or configure the .env file in the backend directory. A sample configuration:

# LLM (LM Studio / Local LLM) Settings
LLM_BASE_URL=http://127.0.0.1:1234/v1
LLM_MODEL=llm-jp-3.1-1.8b-instruct4
LLM_API_KEY=lm-studio

# TTS (Text-to-Speech) Settings
# Options: kokoro, fallback
TTS_ENGINE=kokoro

# Aligner Settings
# Options: wav2vec2, rhubarb, mock
ALIGNER_ENGINE=rhubarb
WAV2VEC2_MODEL_NAME=reazon-research/japanese-wav2vec2-base-rs35kh

Configuration Details:

• LLM Settings: Set up an OpenAI-compatible endpoint (like LM Studio or Ollama). Start your local server first before running the app.
• TTS Engine:
  • kokoro: High-quality, local bilingual TTS. Downloads models automatically on first execution.
  • fallback: Uses Google TTS (gTTS) online, and a procedural beep synthesizer if offline.
• Aligner Engine:
  • wav2vec2: Uses Japanese Wav2Vec2 CTC segmentation for phoneme-level forced alignment.
  • rhubarb: Uses the Rhubarb Lip Sync CLI tool (expects rhubarb executable in system PATH or configured via RHUBARB_PATH).
  • mock: Simple fallback alignment based on average character duration.

3. Running the Backend Server

Start the FastAPI server:

# Using uv:
uv run python main.py

# Or running uvicorn directly:
uv run uvicorn main:app --host 127.0.0.1 --port 8000 --reload

The backend server runs at http://127.0.0.1:8000.

4. Running Backend Tests

Ensure your setup is working correctly:

uv run python test_backend.py

---

💻 Frontend Setup & Execution

1. Installation

Navigate to the frontend directory:

cd frontend

Install the dependencies:

npm install

2. Running the Frontend

Start the local development server (Vite):

npm run dev

The frontend will start running at http://localhost:5173. Open this URL in your web browser.

---

🚀 Running the Whole Project Local Development Flow

To develop or run the project locally, follow these steps:

1. Start your local LLM server: If using LM Studio, download llm-jp-3.1-1.8b-instruct4 (or any other model), start the Local Server, and ensure it is listening on http://127.0.0.1:1234/v1.
2. Start the backend server: Run the FastAPI app on port 8000.
3. Start the frontend: Run the Vite dev server on port 5173 and open it in your browser.
4. Interact: Type a message in the chat box. The tutor will think, stream back the text response in real time, and start speaking with lip-sync animations!