🧠 Deep Learning Perception System — Real-Time Computer Vision on Edge Hardware

MSc Mechatronic Systems Dissertation - End-to-end deep learning perception pipeline with multi-sensor fusion, deployed on Quanser QCar hardware.

🎯 Project Overview

End-to-end autonomous vehicle perception system developed for MSc Mechatronic Systems dissertation at Kingston University, London. The system progresses from simulation (QLabs) to real hardware deployment (Quanser QCar), demonstrating sim-to-real transfer of deep learning models.

🏆 Key Achievements

Metric	Simulation	Real Hardware
YOLOv8 Traffic Detection (mAP@0.5)	99.50%	92.87%
ENet Lane Segmentation (mIoU)	-	93.73%
End-to-End Performance	25 FPS	18-22 FPS
TCP/IP Latency	—	<50ms
Sensor Fusion False Positives	0	0

Sim-to-Real Gap: Quantified at 6.57% performance degradation.

🎬 Demo Videos

🎥 Watch the System in Action

Development journey: Simulation → Real Hardware → Multi-Sensor Fusion

🎮 Simulation (QLabs)

🎬 Lane Following with YOLO Detection QLabs simulation with stop sign detection

🎬 ⭐ Complete QLabs Track Navigation Top-down aerial view of full test environment

🚗 Real Hardware Deployment (Quanser QCar)

🎬 Autonomous Lane Following - Lab Demo Real QCar driving in lab environment

🎬 ⭐ ENet Lane Segmentation (QCar View) First-person view with 93.73% mIoU

🎯 Multi-Sensor Fusion (Production System)

🎬 ⭐ Obstacle Detection QCar with LiDAR + RealSense fusion

🎬 ⭐⭐ Multi-Sensor Fusion Demo LiDAR + Depth Camera + ENet (STRICT fusion)

🎬 Click any thumbnail to watch the full video on YouTube

🚗 Real Hardware Deployment

QCar in Lab Environment	System Running Live

🏗️ Hardware Setup

Quanser QCar Hardware	Test Environment

🎯 Multi-Sensor Fusion

Sensor Fusion View 1	Sensor Fusion View 2

📡 Individual Sensor Outputs

LIDAR 360° View	RealSense Depth Camera

🛣️ Lane Segmentation - ENet (Chosen for Deployment)

ENet Trajectory (93.73% mIoU)	ENet Performance Analysis

🔍 Lane Segmentation - U-Net (Baseline Comparison)

U-Net Trajectory (93.56% mIoU)	U-Net Performance Analysis

Result: ENet chosen for deployment — 87× parameter reduction (0.36M vs 31M) while matching U-Net accuracy (+0.17% mIoU).

🚦 YOLO Object Detection

No Right Turn Detection	Traffic Light Detection
Stop Sign Detection	Occlusion Robustness Test

🎮 Simulation Results (QLabs)

Simulation Full View	QLabs Environment
YOLO: Red Light	YOLO: Green Light
YOLO: Stop Sign	Lane Following
Waypoint Navigation	Curve Traversal

🏗️ System Architecture

The system uses a distributed architecture between the Quanser QCar (running on Jetson TX2) and a laptop with NVIDIA RTX 3060 GPU.

Sensors on QCar:

• CSI Camera (RGB) - Lane detection input
• Intel RealSense D435 - Depth perception
• RPLidar A2 - 360° obstacle detection

Communication: TCP/IP socket connection with <50ms latency

✨ Features

🔍 Object Detection (YOLOv8)

• 5-class custom-trained model
• Classes: Red light, Green light, Yellow light, Stop sign, No-right-turn
• 3,802 annotated training instances
• Real-time inference at 18-25 FPS

🛣️ Lane Segmentation (ENet vs U-Net)

• ENet: 93.73% mIoU, 0.36M params (chosen for deployment)
• U-Net: 93.56% mIoU, 31M params (baseline)
• 87× parameter reduction while exceeding U-Net accuracy by 0.17 percentage points

🎯 Multi-Sensor Fusion

• Intel RealSense D435 depth camera
• RPLidar A2 360° LiDAR
• AND-logic fusion for zero false positives
• Kalman filtering for state estimation

🏎️ Autonomous Control

• Pure Pursuit trajectory tracking
• Adaptive speed control (slow zones, post-curve)
• Automatic braking on obstacle detection
• Waypoint-based navigation for complex maneuvers

🌐 Distributed Architecture

• TCP/IP communication between QCar (Jetson) and laptop GPU
• <50ms end-to-end latency
• Production-ready scalable design

📁 Repository Structure

├── docs/                          # Documentation
│   └── dissertation.pdf
│
├── simulation/                    # QLabs Simulation Code
│   ├── main/
│   │   ├── final_method.py        ⭐ Main file
│   │   ├── environment.py
│   │   └── qlabs_setup.py
│   ├── yolo_training/
│   ├── dataset_tools/
│   ├── utilities/
│   ├── waypoints/
│   └── models/
│
├── real_qcar/                     # Real Hardware Code
│   ├── stream/
│   │   └── stream_server_enet_lidar+realsense.py ⭐ Production
│   ├── perception/
│   └── utilities/
│
├── demo/                          # Videos and images
└── results/                       # Performance metrics

🚀 Quick Start

Simulation (QLabs)

cd simulation/main
python final_method.py

Real Hardware (QCar)

On QCar (Jetson TX2):

cd real_qcar/stream
python stream_client_lidar.py

On Laptop (GPU inference):

cd real_qcar/stream
python stream_server_enet_lidar+realsense.py

🛠️ Tech Stack

Languages: Python 3.10
Deep Learning: PyTorch, YOLOv8 (Ultralytics), TensorFlow/Keras Computer Vision: OpenCV, semantic segmentation (ENet, U-Net)
ML Engineering: Model optimization, edge deployment, multi-sensor fusion Hardware / Deployment: NVIDIA Jetson TX2, Intel RealSense D435, RPLidar A2 Platform: Quanser QCar

📊 Development Journey

1. Simulation First (QLabs) - Validated algorithms in safe environment
2. U-Net Lane Detection - Baseline implementation (31M parameters)
3. ENet Optimization - 87x parameter reduction for embedded deployment
4. LiDAR Integration - 360° obstacle detection
5. RealSense Integration - Depth-based perception
6. Multi-Sensor Fusion - AND-logic for zero false positives
7. Real Hardware Deployment - QCar with distributed architecture

📖 Dissertation

📄 Full Dissertation (120 pages)

🎓 About

Author: Sarath Kumar Komathukattil
Degree: MSc Mechatronic Systems
University: Kingston University, London (2025–2026)

🔮 Future Work

• TensorRT inference optimization
• Quantization for faster edge inference
• Expand detection classes / retrain on larger dataset

📫 Contact

• 🇨🇦 Location: Canada

📜 License

MIT License - See LICENSE for details

---

⭐ If you find this project interesting, please consider starring the repository!