Morphix

Real-time AI-powered motion capture for VRM avatars in the browser.

Morphix is a browser-based 3D motion capture system that maps live webcam input to fully rigged VRM avatars in real time. It combines MediaPipe Holistic tracking with Kalidokit solving and renders the result using React Three Fiber and Three.js.

Designed for:

• VTuber-style avatars
• Motion-driven 3D characters
• Web-based interactive avatars
• Experimental real-time animation pipelines

---

🏷 Badges

---

🖼️ Preview / Screenshot

---

🚀 Tech Stack

Frontend

• React 19
• TypeScript
• Vite
• TailwindCSS
• Leva (debug controls)

3D Rendering

• Three.js
• React Three Fiber
• Drei
• Postprocessing (Bloom)

Motion Capture & Solving

• MediaPipe Holistic
• Kalidokit
• Three-VRM (@pixiv/three-vrm)

State Management

• Zustand

---

🏗 Architecture Overview

Morphix runs entirely in the browser:

1. Webcam feed captured using MediaPipe Camera

2. Holistic model detects:

   • Face landmarks
   • Pose landmarks
   • Hand landmarks

3. Kalidokit converts landmarks into rig data

4. VRM humanoid bones & blendshapes updated per frame

5. Optional fallback to preloaded Mixamo animations

---

High-Level System Diagram

flowchart LR
    Webcam --> MediaPipe
    MediaPipe -->|Landmarks| Kalidokit
    Kalidokit -->|Rig Data| VRMAvatar
    VRMAvatar --> ThreeJS
    ThreeJS --> Canvas

Loading

---

📦 Module Structure

This is a single-application Vite project.

Core Modules

CameraWidget

• Initializes MediaPipe Holistic
• Captures webcam stream
• Draws pose/face/hand landmarks
• Publishes results to global store (Zustand)

VRMAvatar

• Loads VRM model

• Applies:

  • Facial blendshapes
  • Bone rotations
  • Eye tracking
  • Finger tracking

• Falls back to animation mixer when webcam is off

Experience

• Scene composition
• Lighting
• Environment preset
• Camera controls
• Avatar selector
• Postprocessing (Bloom)

video-recognition.ts

Global store containing:

• Active video element
• MediaPipe results callback

remap-mixamo-animation-to-vrm.ts

Converts Mixamo FBX animations to VRM-compatible AnimationClips using:

• Bone name mapping
• Quaternion space correction
• Hips scale normalization

---

📁 Folder Structure

.
├── public/
├── src/
│   ├── assets/
│   │   ├── animations/
│   │   ├── icons/
│   │   ├── models/
│   ├── components/
│   │   ├── camera-widget.tsx
│   │   ├── experience.tsx
│   │   ├── vrm-avatar.tsx
│   ├── hooks/
│   │   └── video-recognition.ts
│   ├── utils/
│   │   ├── mixamo-vrm-rig-map.ts
│   │   └── remap-mixamo-animation-to-vrm.ts
│   ├── App.tsx
│   ├── main.tsx
│   └── index.css
├── vite.config.ts
├── tsconfig.json
└── package.json

---

🔁 Core Application Flows

1️⃣ Webcam → Avatar Rig Flow

sequenceDiagram
    participant User
    participant CameraWidget
    participant MediaPipe
    participant Kalidokit
    participant VRMAvatar

    User->>CameraWidget: Enable Camera
    CameraWidget->>MediaPipe: Process Frame
    MediaPipe->>CameraWidget: Return Landmarks
    CameraWidget->>VRMAvatar: Send Results
    VRMAvatar->>Kalidokit: Solve Pose
    Kalidokit->>VRMAvatar: Rig Data
    VRMAvatar->>VRM Model: Apply Rotations & Expressions

Loading

---

2️⃣ Animation Fallback Flow

If webcam is disabled:

• AnimationMixer runs
• Mixamo animations are remapped to VRM
• Selected clip is played

---

🎭 Features

Face Tracking

• Eye blink detection
• Mouth vowel shapes (AA, EE, IH, OH, OU)
• Head rotation
• Eye look-at system

Pose Tracking

• Spine
• Hips
• Arms
• Forearms

Hand Tracking

• Full finger articulation:

  • Thumb
  • Index
  • Middle
  • Ring
  • Little

• All segments:

  • Proximal
  • Intermediate
  • Distal

Debug Controls (Leva)

• Expression sliders
• Animation selector
• Avatar switcher

---

🛠 Setup & Installation

Prerequisites

• Node.js 18+
• npm or pnpm
• Webcam

---

Installation

git clone https://www.github.com/grep-many/morphix.git
cd morphix
npm install

---

Development

npm run dev

Open:

http://localhost:5173

---

Production Build

npm run build
npm run preview

---

🔐 Environment Variables

No required environment variables.

MediaPipe models are loaded via CDN:

https://cdn.jsdelivr.net/npm/@mediapipe/holistic/

---

🎮 Usage

1. Start the dev server
2. Click the camera button
3. Grant webcam permission
4. Move in front of camera
5. Avatar mirrors your motion

Optional:

• Switch avatar via Leva panel
• Disable camera to test animations

---

🧠 Assumptions

• VRM models are humanoid and properly rigged
• Mixamo animations contain mixamo.com clip name
• User has webcam access
• Running in a modern browser with WebGL2

---

📈 Future Improvements

Scalability

• Offload Holistic processing to Web Worker
• GPU acceleration optimization

Performance

• Adaptive resolution based on device capability
• Frame throttling for low-end systems

Security

• Local-only processing (no server transmission)
• CSP tightening for CDN loads

Developer Experience

• Strict typing for bone mapping
• Test harness for animation remapping
• Storybook for avatar testing

Observability

• Frame time diagnostics
• Pose confidence metrics UI
• Error boundary for model loading

---

🎯 Roadmap Ideas

• Audio-driven lip sync
• Multi-user support
• WebRTC avatar streaming
• Networked VRM avatars
• Gesture recognition events
• Recording & export to GLTF animation

---

📜 License

MIT License. See .

---

🧩 Summary

Morphix demonstrates a complete real-time browser-based motion capture pipeline:

• Webcam → AI landmark detection
• Landmark solving → Rig transforms
• VRM humanoid animation
• Three.js rendering
• Interactive debugging controls

It serves as a strong foundation for web-based virtual avatars, VTuber systems, and experimental real-time animation tooling.