🤖 GCBOT — Gamified Cleaning Bot

An AI-powered autonomous trash-collecting robot with real-time web control, live video streaming, YOLOv10 waste detection, and a combined AI + weight-based scoring system.

📖 Table of Contents

Overview
Screenshots
Architecture
Features
Hardware
Software Stack
AI Model — YOLOv10 Trash Detection
Project Structure
Setup & Installation
How to Run
Control Protocol
Scoring System
Web Control Interface
Contributing

🌟 Overview

GCBOT is a remotely controlled garbage-collecting robot built for competitions and real-world waste management challenges. It combines:

🏎️ A 4-wheel drive chassis controlled via a motor driver
🦾 A servo-powered arm & gripper for picking up trash
📷 A live camera stream from an onboard Raspberry Pi
🧠 YOLOv10 AI running real-time trash detection on the laptop
🏆 Combined AI + Weight scoring — points are based on both what the AI detected AND how heavy the object is
⚖️ HX711 load cell measures object weight for score multiplier
🌐 A browser-based control panel accessible from any device on the network
🔐 Login & player system — sign in or create an account before controlling the robot
📲 QR onboarding page — scan to join Wi-Fi, scan again to open the game instantly

The entire system communicates over TCP sockets — the laptop runs the Flask web server, AI inference, and score computation; the Raspberry Pi acts as a bridge between the laptop and the Arduino; and the Arduino directly controls all motors, servos, and the load cell.

📸 Screenshots

🔐 Login Page

Players sign in or create an account before accessing the robot controls. The glassmorphism login card sits on the animated dark background.

📲 QR Onboarding Page (`/qr`)

Displayed on a screen at the competition venue. Players scan Step 1 to join the Wi-Fi hotspot, then scan Step 2 to instantly open the control page — no typing required.

📱 Mobile Views

The interface is fully responsive. On portrait mobile it prompts rotation; on landscape it shows the full control panel.

Portrait (Login)	Landscape

🆕 Create Account Page

New players fill in their Full Name, Nickname (shown on leaderboard), choose an avatar, and set a password. The profile is saved and they are logged in immediately.

🏆 Leaderboard Page

Shows all players ranked by total score. Each entry displays the player's avatar, nickname, full name, and points earned in the session.

🖥️ Web Control Interface (Desktop)

The full-screen landscape control panel with live AI Detection feed, movement D-pad, gripper & arm controls, and the real-time score badge.

🏗️ System Architecture

┌──────────────────────────────────────────────────────────────────┐
│                   GCBOT — Gamified Cleaning Bot                  │
│                                                                  │
│  ┌────────────┐   WiFi/TCP   ┌──────────────┐   Serial/USB       │
│  │   LAPTOP   │◄────────────►│ Raspberry Pi │◄─────────────────► │
│  │            │              │              │                    │
│  │ Flask App  │              │ pi_control   │   ┌─────────────┐  │
│  │ (Port 5001)│              │ (Port 5000)  │   │   Arduino   │  │
│  │            │              │              │   │             │  │
│  │ YOLOv10    │◄── Video ───►│ Camera       │   │ Motor Driver│  │
│  │ Inference  │   (Port 9999)│ Stream Tx    │   │ Servos (x4) │  │
│  │            │              │              │   │ Load Cell   │  │
│  │ Web UI     │◄── Cmds ────►│ Serial Relay │   │ HX711       │  │
│  └────────────┘              └──────────────┘   └─────────────┘  │
│        ▲                                                         │
│        │  HTTP (Browser)                                         │
│  ┌─────┴──────┐                                                  │
│  │ Any Device │ (Phone / Tablet / PC on same network)            │
│  └────────────┘                                                  │
└──────────────────────────────────────────────────────────────────┘

Three-Tier Architecture Diagram

graph TD
    %% Define Node Styles
    classDef component fill:#ffffff,stroke:#333333,stroke-width:1px;

    subgraph Tier3 ["Tier 3: Host / Fog Layer"]
        Browser[("Web Browser (Mobile/PC)\nControl UI & Video")]:::component
        Flask["Flask Web Server"]:::component
        YOLO["YOLOv10 Inference Engine"]:::component

        Browser <==>|HTTP / WebSocket| Flask
        Flask <==>|Frames / Classes| YOLO
    end
    style Tier3 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px;

    subgraph Tier2 ["Tier 2: Edge Layer"]
        Pi["Raspberry Pi 4"]:::component
        Cam[("USB Web Camera")]:::component

        Cam -->|Raw Video| Pi
    end
    style Tier2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;

    subgraph Tier1 ["Tier 1: Hardware Layer"]
        Arduino["Arduino Uno"]:::component
        L298N["L298N Motor Driver"]:::component
        Motors(("4x DC Motors")):::component
        Servos(("4x Servo Motors\n(Arm & Gripper)")):::component
        HX711["HX711 Amplifier"]:::component
        LoadCell[("1Kg Load Cell")]:::component

        Arduino -->|PWM Signals| L298N
        L298N -->|Power| Motors
        Arduino -->|PWM Signals| Servos
        HX711 -->|Digital Weight Data| Arduino
        LoadCell -->|Analog Signal| HX711
    end
    style Tier1 fill:#fff3e0,stroke:#e65100,stroke-width:2px;

    %% Cross-layer connections
    Flask ==>|TCP Control Commands| Pi
    Pi ==>|MJPEG Video Stream| Flask
    Pi <==>|Serial USB| Arduino

Communication Flow

sequenceDiagram
    participant Browser as 🌐 Browser
    participant Laptop as 💻 Laptop (Flask)
    participant Pi as 🍓 Raspberry Pi
    participant Arduino as ⚡ Arduino

    Browser->>Laptop: HTTP POST /cmd {"cmd": "F"}
    Laptop->>Pi: TCP "F\n" (Port 5000)
    Pi->>Arduino: Serial "F\n"
    Arduino-->>Arduino: forward()

    Browser->>Laptop: HTTP GET /video_feed
    Pi-->>Laptop: TCP Video Frame (Port 9999)
    Laptop-->>Laptop: YOLOv10 Inference → tracks last_detected_class
    Laptop-->>Browser: MJPEG Stream

    Note over Browser,Arduino: DROP OBJECT Flow (AI + Weight Scoring)
    Browser->>Laptop: POST /cmd {"cmd": "WEIGHTNOW"}
    Laptop->>Laptop: weighnow_pending = True
    Laptop->>Pi: TCP "WEIGHTNOW\n"
    Pi->>Arduino: Serial "WEIGHTNOW\n"
    Arduino-->>Arduino: HX711 read → diff = current - lastWeight
    Arduino-->>Pi: Serial "W:23.45\n"
    Pi-->>Laptop: TCP "W:23.45\n" (broadcast)
    Laptop-->>Laptop: compute_combined_score(23.45g)
    Note over Laptop: score = round(base × (1 + 23.45/100))
    Laptop-->>Browser: JSON {score: 12, event: {cls: "Bottle", pts: 12, weight: 23.5}}

✨ Features

Feature	Description
🎮 WASD / D-pad Control	Move forward, backward, left, right with keyboard or touch
🦾 Arm Lift Control	Dual servo arm with animated angle ring indicator
🤏 Gripper Control	Dual servo gripper (open/close) with visual feedback
🗑️ Drop Object	One-tap: releases gripper → waits 2s → weighs → combines AI class + weight → scores
📷 Live Camera Feed	MJPEG stream from Pi camera at ~30 FPS
🤖 AI Detection Feed	YOLOv10 trash overlay feed (host only) with toggle
🏆 AI + Weight Scoring	Score = `base_pts × (1 + weight_g / 100)` — heavier + rarer = more points
⚖️ Load Cell Integration	HX711 reads weight diff on command, laptop combines with AI class
📱 Mobile Responsive	Landscape-optimized with portrait rotation hint
⚡ Low Latency	TCP_NODELAY + no blocking delays = near-zero command lag
🔄 Auto-Reconnect	All TCP connections auto-reconnect on drop

🔧 Hardware

Components List

Component	Quantity	Purpose
Arduino Uno	1	Motor & servo control, load cell reading
Raspberry Pi 4	1	Video streaming, TCP bridge to Arduino
L298N Motor Driver	1	4WD motor control
DC Gear Motors (TT Motor)	4	Drive wheels
SG90 / MG90S Servo	4	2× Arm lift + 2× Gripper
HX711 Load Cell Amplifier	1	Weight sensing for scoring
Load Cell (1kg)	1	Detects collected trash weight
Pi Camera Module	1	Live video stream
3S LiPo / 18650 Pack	1	Power supply
Robot Chassis (4WD)	1	Physical body

Wiring — Arduino Pin Map

Arduino Uno
├── Pin  2  → L298N IN1  (Left Motor A)
├── Pin  3  → L298N IN2  (Left Motor B)
├── Pin  4  → L298N IN3  (Right Motor A)
├── Pin  7  → L298N IN4  (Right Motor B)
├── Pin  8  → L298N ENA  (Left Motor PWM)
├── Pin 12  → L298N ENB  (Right Motor PWM)
├── Pin  5  → Gripper Servo A (grabA)
├── Pin  6  → Gripper Servo B (grabB) [mirrored]
├── Pin  9  → Lift Servo A (liftA)
├── Pin 10  → Lift Servo B (liftB) [mirrored]
├── Pin A0  → HX711 SCK
└── Pin A1  → HX711 DT

💻 Software Stack

Layer	Technology
Web Framework	Flask (Python)
AI Inference	Ultralytics YOLOv10 (`best.pt`)
Computer Vision	OpenCV (`cv2`)
Video Protocol	MJPEG over HTTP (multipart)
Control Protocol	Raw TCP sockets with TCP_NODELAY
Serial	PySerial (Pi ↔ Arduino)
Frontend	Vanilla HTML/CSS/JS with glassmorphism UI
Fonts	Google Inter
Microcontroller	Arduino C++ with Servo.h + HX711.h
Pi Script	Python threading + socket server

🧠 AI Model — YOLOv10 Trash Detection

The onboard AI model is a custom-trained YOLOv10 fine-tuned on a multi-class garbage detection dataset. It runs locally on the laptop CPU in a dedicated inference thread at 416×416 resolution.

Detected Waste Classes (18 classes)

Aluminium foil · Bottle · Bottle cap · Broken glass · Can · Carton · Cigarette · Cup · Lid · Other litter · Other plastic · Paper · Plastic bag/wrapper · Plastic container · Pop tab · Straw · Styrofoam piece · Unlabeled litter

Training Performance Graphs

Precision-Recall Curve (Train)	Confusion Matrix (Train)

F1-Confidence Curve	Precision Curve

Validation Performance Graphs

Precision-Recall Curve (Val)	Confusion Matrix (Val)

Inference Optimizations

✅ Frame resized to 416×416 before inference (3× speedup vs full-res)
✅ Duplicate frame skipping — hashes frame identity, skips if unchanged
✅ PyTorch thread capping — prevents CPU over-subscription
✅ Confidence threshold 0.25 — filters weak detections
✅ Annotated frame scaled back to original resolution after inference

📁 Project Structure

GCBOT/
│
├── app.py                              # 💻 Laptop: Flask server, YOLOv10 inference, AI+weight scoring
├── pi_control.py                       # 🍓 Raspberry Pi: TCP server + Serial relay to Arduino
│
├── gcbot_arduino/
│   └── gcbot_arduino.ino              # ⚡ Arduino: motors, servos, load cell (weight only)
│
├── Trash_detection_Yolov10_StreamLit/
│   ├── best.pt                        # 🧠 Trained YOLOv10 model weights
│   ├── train_graphs/                  # 📊 Training performance graphs
│   │   ├── PR_curve.png
│   │   ├── F1_curve.png
│   │   ├── P_curve.png
│   │   ├── R_curve.png
│   │   └── confusion_matrix_normalized.png
│   ├── val_graphs/                    # 📊 Validation performance graphs
│   │   ├── PR_curve_val.png
│   │   ├── F1_curve_val.png
│   │   ├── P_curve_val.png
│   │   ├── R_curve_val.png
│   │   └── confusion_matrix_normalized_val.png
│   ├── train_yolov10_garbage_detection.ipynb  # 📓 Training notebook
│   └── requirements.txt
│
├── .gitignore
├── Screenshot 2026-03-26 213021.png   # 🖼️ Desktop UI screenshot
├── WhatsApp Image 2026-03-17 *.jpeg   # 🖼️ Hardware & mobile UI photos
└── README.md

⚙️ Setup & Installation

Prerequisites

Python 3.10+ on the Laptop
Python 3.x on the Raspberry Pi
Arduino IDE for flashing the .ino sketch

1. Laptop Setup

# Clone the repository
git clone https://github.com/your-username/gcbot.git
cd gcbot

# Create virtual environment
python -m venv .venv
.venv\Scripts\activate      # Windows
# source .venv/bin/activate # Linux/Mac

# Install dependencies
pip install flask ultralytics opencv-python numpy torch

2. Raspberry Pi Setup

# Copy pi_control.py to the Raspberry Pi
scp pi_control.py pi@kavin.local:~/

# SSH into Pi and install dependencies
ssh pi@kavin.local
pip install pyserial

3. Arduino Setup

Open gcbot_arduino/gcbot_arduino.ino in Arduino IDE
Install required libraries:
- Servo (built-in)
- HX711 by bogde
Flash to your Arduino Uno

⚠️ Calibrate your load cell! Replace 2280.0 in the sketch with your actual calibration factor. Run a calibration sketch with a known weight to find the correct value.

4. Configuration

In app.py, update the Pi's hostname or IP:

PI_IP        = "kavin.local"   # or use the Pi's IP e.g. "192.168.1.42"
CONTROL_PORT = 5000
VIDEO_PORT   = 9999

In pi_control.py, update the serial port if needed:

ARDUINO_PORT = "/dev/ttyUSB0"   # use /dev/ttyACM0 for some Arduinos
ARDUINO_BAUD = 9600

🚀 How to Run

Step 1 — Flash the Arduino

Connect the Arduino to the Raspberry Pi via USB, then flash gcbot_arduino.ino from your development machine.

Step 2 — Start the Pi Bridge

# On Raspberry Pi
python3 pi_control.py

The Pi will:

Open the Arduino serial port
Start listening for laptop connections on port 5000
Start a video stream sender on port 9999 (handled by the Pi camera script — ensure your Pi camera streaming script is also running)

Step 3 — Start the Laptop Server

# On the Laptop (must be on same WiFi as the Pi)
python app.py

Step 4 — Open the Web UI

Navigate to:

http://localhost:5001          # on the laptop (AI Detection feed)
http://<laptop-ip>:5001        # on any phone/tablet on the same WiFi

📡 Control Protocol

Commands are sent as newline-terminated ASCII strings over TCP:

Command	Description
`F`	Move forward
`B`	Move backward
`L`	Turn left
`R`	Turn right
`S`	Stop all motors
`U<angle>`	Set arm lift angle (0–180°), e.g. `U90`
`G<angle>`	Set gripper angle (0–180°), e.g. `G120`
`WEIGHTNOW`	Trigger a weight check on the load cell

HTTP API (Flask)

Endpoint	Method	Description
`/`	GET	Serve the control UI
`/cmd`	POST	Send a command to the robot
`/video_feed`	GET	Raw MJPEG camera stream
`/video_feed_detected`	GET	YOLOv10 annotated MJPEG stream
`/score`	GET	Current score + last scoring event breakdown
`/scorechart`	GET	Full score classification chart as JSON

Example cURL:

# Send a movement command
curl -X POST http://localhost:5001/cmd \
     -H "Content-Type: application/json" \
     -d '{"cmd": "F"}'

# Get the score chart
curl http://localhost:5001/scorechart

🏆 Scoring System

Scoring uses a combined AI detection + weight measurement system. The laptop computes the final score using both what the AI sees and how heavy the object is.

Formula

final_score = round(base_score × (1 + weight_g / 100))

base_score — determined by the detected trash class (see chart below)
weight_g — weight difference measured by the HX711 load cell (in grams)
No detection fallback — if no trash was detected by AI: base = max(1, weight_g ÷ 10)

Score Classification Chart

Object Class	Base Points	@ 50g	@ 100g	@ 200g	Reason
Broken glass	15	22	30	45	Dangerous — bonus reward
Bottle	10	15	20	30	Recyclable
Can	10	15	20	30	Recyclable
Plastic bag / wrapper	10	15	20	30	High environmental impact
Carton	8	12	16	24	Recyclable
Cup	8	12	16	24	Disposable
Plastic container	8	12	16	24	Recyclable
Styrofoam piece	8	12	16	24	Hard to recycle
Other plastic	7	10	14	21	—
Aluminium foil	5	7	10	15	Recyclable
Paper	5	7	10	15	Recyclable
Straw	5	7	10	15	Environmental hazard
Other litter	5	7	10	15	—
Cigarette	3	4	6	9	Small but toxic
Lid	3	4	6	9	Small
Bottle cap	3	4	6	9	Small
Pop tab	3	4	6	9	Small
Unlabeled litter	2	3	4	6	Unknown fallback
(No detection)	weight÷10	5	10	20	Weight-only fallback

Scoring Flow

1. User presses 🗑️ DROP OBJECT button in the browser
2. Gripper opens to 50° (releases trash)
3. After 2s delay, WEIGHTNOW command sent
4. Arduino reads HX711 load cell (avg 3 readings)
5. Arduino sends weight diff: "W:23.45\n"
6. Pi broadcasts W:23.45 to all connected laptops
7. Laptop's app.py combines:
   • last_detected_class (from YOLO inference) → base = 10 (Bottle)
   • weight_g = 23.45g → multiplier = 1.23
   • final_score = round(10 × 1.23) = 12 pts
8. Browser shows: "+12 Bottle (23.5g)" and updates the score badge

💡 Tip: The weight multiplier means heavier objects of the same class score higher. A full 200g bottle scores 30 pts vs. a small 50g bottle at 15 pts.

⚖️ No detection fallback: If the AI didn't detect the object before drop, scoring falls back to weight-only: 1 point per 10 grams.

🌐 Web Control Interface

The control UI is a single-page, glassmorphism-styled dashboard embedded in Flask (app.py). No external CSS framework is used.

UI Features

🎮 D-pad for movement (touch + WASD keyboard)
🔵 SVG ring indicator for real-time arm/gripper angle display
🟢 Score badge (top-right, always visible) with scale-bounce animation on score update
📡 Feed toggle — switch between Raw and AI Detection video feed
🟢 Status dot with pulse animation indicating live connection
↻ Portrait warning — prompts rotation on mobile portrait mode
✚ Crosshair overlay on the video feed for targeting

🤝 Contributing

Contributions, issues, and feature requests are welcome!

Fork the repository
Create your feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

📄 License

This project is open source and available under the MIT License.

MIT License

Copyright (c) 2026 Kavin and Team

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Built with ❤️ for making the world cleaner, one piece of trash at a time 🌍♻️

GCBOT — Gamified Cleaning Bot

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
Trash_detection_Yolov10_StreamLit		Trash_detection_Yolov10_StreamLit
gcbot_arduino		gcbot_arduino
screenshots		screenshots
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
Screenshot 2026-03-26 213021.png		Screenshot 2026-03-26 213021.png
WhatsApp Image 2026-03-17 at 8.08.44 PM.jpeg		WhatsApp Image 2026-03-17 at 8.08.44 PM.jpeg
WhatsApp Image 2026-03-17 at 8.20.31 PM.jpeg		WhatsApp Image 2026-03-17 at 8.20.31 PM.jpeg
WhatsApp Image 2026-03-26 at 6.47.48 PM.jpeg		WhatsApp Image 2026-03-26 at 6.47.48 PM.jpeg
app.py		app.py
pi_control.py		pi_control.py

Folders and files

Latest commit

History

Repository files navigation

🤖 GCBOT — Gamified Cleaning Bot

📖 Table of Contents

🌟 Overview

📸 Screenshots

🔐 Login Page

📲 QR Onboarding Page (/qr)

📱 Mobile Views

🆕 Create Account Page

🏆 Leaderboard Page

🖥️ Web Control Interface (Desktop)

🏗️ System Architecture

Three-Tier Architecture Diagram

Communication Flow

✨ Features

🔧 Hardware

Components List

Wiring — Arduino Pin Map

💻 Software Stack

🧠 AI Model — YOLOv10 Trash Detection

Detected Waste Classes (18 classes)

Training Performance Graphs

Validation Performance Graphs

Inference Optimizations

📁 Project Structure

⚙️ Setup & Installation

Prerequisites

1. Laptop Setup

2. Raspberry Pi Setup

3. Arduino Setup

4. Configuration

🚀 How to Run

Step 1 — Flash the Arduino

Step 2 — Start the Pi Bridge

Step 3 — Start the Laptop Server

Step 4 — Open the Web UI

📡 Control Protocol

HTTP API (Flask)

🏆 Scoring System

Formula

Score Classification Chart

Scoring Flow

🌐 Web Control Interface

UI Features

🤝 Contributing

📄 License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

📲 QR Onboarding Page (`/qr`)

Packages