A real-time AI object detection system that speaks to visually impaired users, guiding them safely through their environment.
Virtual Eye is a mobile application built for visually impaired individuals that uses the smartphone camera to detect surrounding objects in real-time and delivers instant voice descriptions. The system sends captured frames to a YOLOv8-powered FastAPI backend, generates natural language descriptions of detected objects, and speaks them aloud using Text-to-Speech — without cutting off mid-sentence. Haptic vibration alerts are triggered when dangerous objects are detected nearby.
The app is built with React Native (Expo) and packaged as a standalone Android APK, requiring no specialized hardware — just an Android smartphone and a WiFi connection to the backend server.
Note: This project was developed as part of an academic exhibition to explore the practical application of AI in accessibility technology. While currently a functional prototype, the system is designed with scalability in mind and can be expanded into a full production-grade assistive tool with proper funding, infrastructure, and investment.
- 🔍 Real-Time Object Detection — Captures frames and detects objects using YOLOv8m
- 🔊 Full Voice Guidance — Speaks complete sentences, never cuts off mid-speech
⚠️ Danger Warnings — Highlights dangerous objects with orange UI and faster speech- 📳 Haptic Alerts — Vibrates the phone when a hazardous object is detected
- 🔦 Manual Flashlight Control — Toggle torch ON/OFF with voice confirmation
- 💤 Smart Detection Loop — Waits for speech to finish before scanning again
- 🟢 Live Indicator — Green dot shows when detection is actively running
- 📱 Standalone Android APK — Installable directly, no Expo Go required
- 🎙️ Voice Assistant Launch — Open app via "Hey Google, open Virtual Eye"
Virtual-Eye-App/
├── backend/ # Original backend
├── backend-exhibition/ # Exhibition-optimized backend
│ ├── main.py # FastAPI server entry point
│ ├── detector.py # YOLOv8 detection logic
│ └── requirements.txt # Python dependencies
├── VirtualEye/ # Original React Native app
└── VirtualEye-Exhibition/ # Exhibition-optimized app
├── app/
│ └── (tabs)/
│ └── index.tsx # Main screen — camera, detection, UI
├── app.json # Expo config + intent filters
├── eas.json # EAS Build config (APK)
└── package.json
📷 Camera Captures Frame
↓
📡 Frame Sent to FastAPI Backend (HTTP POST)
↓
🧠 YOLOv8m Detects Objects + Confidence Scores
↓
💬 Natural Language Description Generated
↓
🔊 Text-to-Speech Speaks Full Sentence
↓
📳 Haptic Vibration if Danger Detected
↓
⏳ Wait for Speech to Finish
↓
🔁 Loop Repeats Automatically
| Technology | Purpose |
|---|---|
| React Native + Expo | Cross-platform mobile framework |
| TypeScript | Type-safe development |
| Expo Camera | Real-time camera frame capture |
| Expo Speech | Text-to-Speech voice output |
| Expo Haptics | Vibration feedback for warnings |
| EAS Build | Compiles standalone Android APK |
| Technology | Purpose |
|---|---|
| Python 3.10+ | Backend language |
| FastAPI | High-performance REST API server |
| Ultralytics YOLOv8m | Object detection AI model |
| OpenCV | Image preprocessing |
| Uvicorn | ASGI server for FastAPI |
- Python 3.10+
- Node.js 18+
- Android smartphone (for app testing)
- Both phone and laptop on the same WiFi network
git clone https://github.com/haroon1nonly/Virtual-Eye-App.git
cd Virtual-Eye-Appcd backend-exhibition
pip install -r requirements.txtUpdate the backend IP in the app if needed (see Step 4), then start the server:
python main.pyThe server runs on http://0.0.0.0:8000. Check your local IP using:
# Windows
ipconfig
# Linux/Mac
ifconfigcd VirtualEye-Exhibition
npm installOpen app/(tabs)/index.tsx and update the BACKEND constant:
const BACKEND = 'http://YOUR_LAPTOP_IP:8000';Replace YOUR_LAPTOP_IP with the IP address from Step 2 (e.g., 192.168.1.5).
npx expo startScan the QR code with Expo Go on your Android phone.
npm install -g eas-cli
eas login
eas build --platform android --profile previewOnce built, download the .apk link and install directly on your Android phone.
- Install the APK on your Android phone
- Start the backend on your laptop (
python main.py) - Connect both devices to the same WiFi network
- Open Virtual Eye on your phone (or say "Hey Google, open Virtual Eye")
- Tap anywhere on the screen to start detection
- Listen — the app will speak what it sees continuously
- Tap again to stop detection
- Torch button (top right) — manually toggle flashlight ON/OFF
MODEL = "yolov8m.pt" # Model size: n / s / m / l / x
CONFIDENCE = 0.25 # Minimum confidence threshold (0.0 - 1.0)
IMG_SIZE = 960 # Input image resolutionrate: 0.55, // Speech speed (0.1 slow — 1.0 fast)
pitch: 1.0, // Voice pitch
volume: 1.0, // Volume levelawait sleep(1500); // Pause between detections (ms)
await sleep(4000); // Pause after server error (ms)Virtual Eye is built as a scalable foundation. The following advancements are planned or possible with further development and investment:
The system can be embedded into smart glasses hardware, eliminating the need to hold a phone. The camera feed would stream directly from the glasses to the detection backend, providing a completely hands-free and natural experience for visually impaired users — making it feel invisible and effortless.
Users can feed the model photos of their home rooms and architectural floor plans to create a personalized indoor map. The AI would then recognize familiar rooms, furniture positions, and specific objects unique to that household — providing highly accurate, room-aware guidance like:
"The sofa is 2 steps to your left. The kitchen door is straight ahead."
By converting the YOLOv8 model to TFLite with INT8 quantization, the entire detection pipeline can run directly on the phone's Neural Processing Unit (NPU) — no WiFi, no server, no latency. Complete independence anywhere in the world.
Integration with GPS and Google Maps API to provide outdoor turn-by-turn navigation combined with real-time object detection — warning about traffic, pedestrians, stairs, and road crossings simultaneously.
Allow users or caregivers to fine-tune the detection model with custom object categories specific to their needs — recognizing personal belongings, medicines, specific people's faces, or workplace equipment.
Integrate face recognition so the app can identify and announce familiar people by name — helping users know who is approaching them in a room or public space.
Expand Text-to-Speech support beyond English to Urdu, Arabic, Hindi, and other languages — making the app accessible to a global audience including users in Pakistan and the wider developing world.
Connect with smartwatches for silent haptic navigation cues, or integrate with IoT smart home systems to announce when doors open, appliances are running, or visitors arrive.
💡 Interested in contributing or investing? This project has real-world impact potential for millions of visually impaired individuals worldwide. Reach out to collaborate or support its development.
Muhammad Haroon Abbas
This project is licensed under the MIT License. See the LICENSE file for details.
- Ultralytics YOLOv8 — Object detection model
- Expo — React Native development platform
- FastAPI — Backend framework