Teman Isyarat — Android App

Teman Isyarat (Indonesian for "Sign Friend") is an on-device, real-time Indonesian Sign Language (BISINDO) recognition app built with Flutter and Kotlin. It captures hand and pose landmarks via CameraX and MediaPipe, runs inference through a custom LiteRT model, and displays predictions — all fully offline.

---

Table of Contents

• Overview
• Features
• Architecture
• Project Structure
• Tech Stack
• Getting Started
• Building
• Project Ecosystem
• License

---

Overview

The app is the mobile endpoint of a three-part system:

1. lm/ — MediaPipe landmark extraction pipeline from raw video
2. model/ — GRU-based neural network training & LiteRT export
3. android/ (this repo) — Flutter app wrapping the exported LiteRT model for on-device inference

It recognizes 20 BISINDO vocabulary words (Central Java dialect) through live camera feed, without requiring internet connectivity. Articles are fetched from Sanity CMS via HTTP.

Category	Words
Pronouns	Aku, Kamu, Dia
Common	Salam, Terima Kasih, Maaf, Nama
Time	Hari Ini, Besok
Color	Merah, Kuning
Family	Ayah, Ibu
Count	Satu, Dua, Tiga
Other	Teman, Buku
Fruit	Apel, Pisang

Features

• Real-time camera translation — Live CameraX preview with on-screen prediction overlay
• On-device ML — All inference runs locally via LiteRT (Google AI Edge); no network required
• MediaPipe landmark tracking — Pose (9 upper-body keypoints) + hands (2 × 21 landmarks)
• Temporal smoothing — Majority-vote over a 2-frame history window for stable output
• Circular frame buffer — 110-frame native FloatArray buffer feeding the LiteRT model
• Camera switch — Toggle between front and rear cameras
• Skeleton overlay — Live canvas rendering of detected hand and pose landmarks

Architecture

┌────────────────────────────────────────────────────────────┐
│                    Flutter / Dart (UI)                     │
│  ┌─────────┐  ┌──────────────┐  ┌──────────┐  ┌─────────┐  │
│  │ Home    │  │ Translate    │  │ Artikel  │  │Settings │  │
│  │ Page    │  │ Page         │  │ Page     │  │ Page    │  │
│  └─────────┘  └──────┬───────┘  └──────────┘  └─────────┘  │
│                      │ MethodChannel                       │
├──────────────────────┼─────────────────────────────────────┤
│              Android PlatformView (Kotlin)                 │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  CameraX (PreviewView)  ◄──►  MediaPipe Landmarker   │  │
│  │                              (Pose + Hand)           │  │
│  │                                       │              │  │
│  │  ┌────────────────────────────────────┘              │  │
│  │  │  assembleFrame() → 51 landmarks × 3 = 153-dim     │  │
│  │  │                                                   │  │
│  │  │  Circular Buffer (110 frames) ──► LiteRT Interp.  │  │
│  │  │                        Softmax + Temporal Smooth. │  │
│  │  │                        ──► MethodChannel callback │  │
│  │  └───────────────────────────────────────────────────┘  │
│  └──────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────┘

Data Flow

1. CameraX frames are fed to MediaPipe Hand + Pose Landmarker
2. 51 landmarks (9 pose + 21 left hand + 21 right hand) × 3 (x, y, z) are assembled per frame
3. Landmarks are pushed into a circular 110-frame FloatArray buffer
4. When full, the buffer is sent to LiteRT Interpreter.run() with input shape [1, 110, 153]
5. Raw logits (20 classes) go through softmax → confidence threshold (0.5) → majority-vote temporal smoothing (2-frame window)
6. The predicted label is sent back to Dart via MethodChannel and rendered in the UI

Project Structure

temanisyarat/
├── lib/
│   ├── main.dart                         # App entry, MainPage, navigation, articles, settings (972 lines)
│   ├── constants.dart                    # Color palette constants (C class)
│   ├── pages/
│   │   └── translate/
│   │       ├── translate_page.dart        # Camera + sign translation UI (299 lines)
│   │       ├── translate_controller.dart  # Business logic, channel bridge, state (121 lines)
│   │       └── widgets/
│   │           └── scanning_dots.dart     # Animated scanning indicator (55 lines)
│   └── services/
│       └── sanity_service.dart            # Sanity CMS article fetching via HTTP (257 lines)
├── android/app/src/main/kotlin/com/hibah/temanisyarat/
│   ├── handlandmarker/
│   │   ├── HandLandmarkerPlugin.kt        # Flutter plugin registration (PlatformViewFactory)
│   │   ├── HandLandmarkerView.kt          # PlatformView: CameraX + landmarks + LiteRT (498 lines)
│   │   ├── HandLandmarkerHelper.kt        # MediaPipe Hand/Pose Landmarker wrapper
│   │   └── HandLandmarkerOverlay.kt       # Canvas skeleton overlay
│   └── MainActivity.kt                   # FlutterActivity entry
├── android/app/src/main/assets/models/
│   └── model_raw.tflite                   # Trained LiteRT classification model
├── android/                              # Android native project root
├── assets/                               # SVG illustrations, icons, launcher icon
├── pubspec.yaml                          # Flutter dependencies & config
└── .tool-versions                        # Tool version pinning

Tech Stack

UI Layer (Dart / Flutter)

Component	Technology
Framework	Flutter 3.44 / Dart
State Management	StatefulWidget + setState
Navigation	Navigator.push / Bottom Navigation
HTTP Client	http (Sanity CMS article fetching)
Persistence	path_provider (file I/O)
Permissions	permission_handler (camera)
Assets	flutter_svg (SVG rendering)

Native Layer (Kotlin / Android)

Component	Technology
Camera	CameraX 1.6.1 (core, camera2, lifecycle, view)
Landmark Detection	MediaPipe Tasks Vision 0.10.35 (Pose + Hand)
ML Runtime	LiteRT 1.4.1 + TensorFlow Select TF Ops 2.16.1
Platform Bridge	Flutter MethodChannel + PlatformView
Min SDK	24
Kotlin	2.3.21
Gradle	9.5.1

ML Model

Property	Value
Input shape	[1, 110, 153]
Output shape	[1, 20] (logits)
Classes	20 BISINDO words
Architecture	GRU + 1D Conv + TempAttention
Model size	~2.6 MB (LiteRT FP16)
Temporal smoothing	2-frame majority vote
Confidence threshold	0.5

Getting Started

Prerequisites

• Flutter 3.44+ (see .tool-versions)
• JDK 17+ (OpenJDK 26 recommended)
• Android SDK (compileSdk from Flutter Gradle plugin)
• Android device or emulator (API 24+)

Setup

# Clone the repository
git clone https://github.com/temanisyarat/android.git
cd android

# Install Flutter dependencies
flutter pub get

# Run on connected device
flutter run

The LiteRT model is bundled in android/app/src/main/assets/models/model_raw.tflite and is copied to the device cache directory on first launch.

Lint & Test

flutter analyze        # Static analysis
flutter test           # Run widget tests

Building

# Debug APK
flutter build apk --debug

# Release APK
flutter build apk --release

# App Bundle
flutter build appbundle --release

# iOS (macOS only)
flutter build ios --release

Note: Release signing currently uses the debug configuration. Configure a proper release keystore before publishing.

Project Ecosystem

This app is part of the Teman Isyarat monorepo, which includes:

Repository	Purpose
lm/	MediaPipe landmark extraction pipeline — converts raw BISINDO videos to .npz landmark arrays
model/	GRU-based neural network training, evaluation, and LiteRT export
android/ (you are here)	Flutter + Kotlin Android app for on-device real-time recognition
manager/	Obsidian vault with ADRs, specs, sprint tracking, and team documentation

Data Flow Across Repositories

Video (raw MP4)  ──►  lm/  ──►  .npz landmarks  ──►  model/  ──►  .tflite  ──►  android/  ──►  Live predictions
                         (MediaPipe extract)           (Train GRU)           (LiteRT on-device inference)

License

This project is developed for academic purposes under the Hibah Jarprak program at Universitas Sebelas Maret (UNS), in partnership with GERKATIN Solo.

---

Built with Flutter, MediaPipe, and LiteRT for Indonesian Sign Language (BISINDO) recognition.