LocalLens

Offline Android assistive vision with phone camera, Ray-Ban Meta glasses, and Gemma 4 on-device.

Download LocalLens 1.0 APK | Release notes

What It Does

LocalLens is a proof-of-concept Android app for blind and visually impaired users. It captures the current scene from either the phone camera or Ray-Ban Meta glasses, records a spoken request, runs Gemma 4 locally through LiteRT-LM, and speaks the answer with Android system Text-to-Speech.

The public app is intentionally lean: no cloud calls during assist turns, no separate STT model, no lab pages, and no experimental Sherpa or Kokoro paths. After setup, the main assist loop runs on the phone.

Important Safety Disclaimer

LocalLens is provided as a proof of concept only. It should not be considered safe or reliable for full disability use cases, and it is not a replacement for certified assistive, navigation, medical, or safety-critical tools.

It can be wrong, slow, unavailable, or incomplete. App performance depends heavily on the phone GPU, CPU, memory, thermals, storage, Android version, installed TTS voices, model files, and connected camera or glasses state.

Download And Quick Start

1. Download the latest APK from GitHub Releases.
2. Sideload the APK on an Android phone. LocalLens is not distributed through the Play Store.
3. Launch the app and accept the proof-of-concept disclaimer.
4. Use Wi-Fi or mobile data to download the offline Gemma 4 model when prompted. At least 4 GB free storage is recommended.
5. Grant camera and microphone permissions.
6. Choose Phone or Glasses, then use Talk, Chat, or the wake phrase: Hey, LocalLens.

After the APK and model are installed, assist turns run locally on the phone. Airplane mode can be enabled for offline testing, although glasses mode uses Bluetooth for the camera connection, so Bluetooth may need to stay enabled. Android system services and TTS behavior may also vary by device.

If a future release switches to a production signing key, Android may require uninstalling the proof-of-concept APK before installing the newer build.

Features

• Offline assist turns with Gemma 4 E2B through LiteRT-LM.
• Phone camera mode using CameraX.
• Ray-Ban Meta glasses camera mode using Meta Wearables DAT.
• Multimodal turns that combine the current image with spoken audio.
• Audio-only chat turns for follow-up questions or general conversation.
• Wake activation with Hey, LocalLens.
• Android system Text-to-Speech output with device-installed voices.
• First-run model setup with download resume, storage checks, and update-required states.
• Open-source notices available inside the app.

Language Support

Gemma 4 advertises out-of-the-box support for 35+ languages in text, with pretraining across many more languages. LocalLens asks Gemma to answer in the user's Android language and then uses Android system Text-to-Speech for spoken output.

Audio-input coverage is less clear. Gemma 4 E2B supports audio input and multilingual speech tasks, but Google does not publish a precise end-to-end audio language list for this LiteRT app path. Your mileage may vary, especially with accents, noise, glasses microphones, mixed-language documents, and missing offline Android TTS voices.

English is the primary tested path for this proof-of-concept release. Other languages should be treated as experimental until validated on real device audio. See Gemma 4 Audio Language Validation Matrix.

Supported Devices And System Requirements

Minimum:

• ARM64 Android phone running Android 10 or newer.
• Usable GPU acceleration for LiteRT-LM/TensorFlow Lite GPU inference.
• 4 GB free storage for the downloaded Gemma model and app data.
• Enough available RAM to load Gemma 4 E2B LiteRT-LM; no hard RAM minimum has been validated across devices.
• Camera and microphone permissions for visual assist turns.
• Android system Text-to-Speech engine for spoken output.

Recommended:

• Recent higher-end Android phone with a strong mobile GPU and good thermal headroom.
• 8 GB or more system RAM.
• 6 GB or more free storage before first setup.
• Android 13 or newer.
• An Android TTS voice available for the user's preferred language.
• Wi-Fi or mobile data for the first Gemma model download and future model updates.
• Optional: Ray-Ban Meta glasses paired through the Meta AI app for glasses camera mode.

LocalLens is not expected to work well on emulators, x86 devices, very low-end phones, devices without usable GPU acceleration, or devices without enough free storage and memory for the model.

For developers:

• Android Studio or Android SDK command-line tools.
• Android SDK platform 36.1 or newer.
• Android NDK.
• JDK 21.
• A connected Android device with adb.
• Optional: Meta Wearables DAT application id and client token for glasses testing.

Developer Setup

Meta DAT dependencies are resolved from GitHub Packages. Local builds need one of github.token, github_token, GH_PACKAGES_TOKEN, or GITHUB_TOKEN set to a GitHub token with read:packages access. GitHub Actions CI uses a repository secret named GH_PACKAGES_TOKEN.

Configure Meta DAT credentials through Gradle properties or ignored local.properties:

mwdat.applicationId=YOUR_META_DAT_APPLICATION_ID
mwdat.clientToken=YOUR_META_DAT_CLIENT_TOKEN

Build a debug APK:

.\gradlew --no-daemon --no-configuration-cache :app:assembleDebug

Install on a connected device:

adb install -r app\build\outputs\apk\debug\app-debug.apk

Run the main CI checks locally:

.\gradlew --no-daemon --no-configuration-cache test lint :app:assembleDebug

Gemma 4 Model And MTP

The Gemma model is not bundled in the APK. On first launch, LocalLens downloads the Gemma 4 E2B LiteRT-LM bundle with Android DownloadManager and stores it under app-specific external storage:

Android/data/com.example.locallens/files/vlm/gemma_4_e2b/

The installer resumes existing downloads, validates file size, checks Gemma 4 MTP/speculative decoding support, and asks for a model update if an older pre-MTP bundle is already installed.

Gemma 4 MTP requires LiteRT-LM 0.11.0 and the May 5, 2026 or newer litert-community/gemma-4-E2B-it-litert-lm bundle. Existing manually staged model files may need to be replaced to enable speculative decoding.

Developer model staging:

.\scripts\install_gemma4_e2b_litertlm.ps1

Force a fresh developer download if your local copy predates the MTP-capable bundle:

.\scripts\install_gemma4_e2b_litertlm.ps1 -ForceDownload

More details: Gemma 4 LiteRT-LM integration.

Glasses Support

Phone camera mode is the default and uses CameraX. Glasses mode uses Meta Wearables Device Access Toolkit (DAT).

Known Meta DAT-side limitations:

• LocalLens targets Meta DAT 0.6.0 for Gen 1 Ray-Ban Meta Wayfarer compatibility.
• In device testing, Meta DAT 0.7.0 reported the Gen 1 Wayfarer firmware as DEVICE_UPDATE_REQUIRED, even though no firmware update was available in the Meta AI app.
• Meta DAT exposes still photo capture through a camera stream/session API rather than a standalone one-shot photo API. LocalLens uses a short, low-frame-rate stream for still captures when preview is not already running, but this still costs more glasses battery than a true photo-only capture path would.

Repository Branches

• main: polished public app branch for users and app-focused contributors.
• research-release-1.0: dissertation and reproducibility branch with older lab pages, benchmark scripts, Sherpa/Kokoro experiments, llama.cpp harnesses, and model comparison tooling.

This public branch intentionally does not include large or device-specific artifacts such as model weights, Qualcomm SDKs, QNN/QAIRT packages, Hexagon SDKs, generated native libraries, or local Android configuration.

Safety

LocalLens is a research-derived assistive prototype, not a safety-certified navigation, medical, or emergency tool. Treat every answer as uncertain, especially around hazards, medication, money, travel, documents, people, and physical navigation.

Project Status

LocalLens was developed for completion of an MSc in Artificial Intelligence. Continued maintenance may be limited, so please fork the code if you would like to adapt, extend, or continue development.

License And Notices

LocalLens is licensed under the MIT License.

The app includes an open-source notices screen for third-party software and SDK components. Gemma model downloads are governed by the applicable Gemma and Hugging Face model terms shown by the model provider.