TAPO C230 Edge AI Tracker & Analyzer

A high-performance, Low-latency computer vision and PTZ tracking system for TP-Link Tapo C230 cameras, powered by Nvidia Jetson.

📌 Overview

This project is an advanced Edge AI platform designed for seamless integration with TP-Link Tapo security cameras over a local network. By combining FastAPI, asynchronous processing, and ONVIF-based camera control, the system enables low-latency, real-time camera interaction and intelligent motorized tracking of moving subjects.

The platform extends beyond traditional video surveillance by combining real-time computer vision, facial recognition, behavior analysis, and automated event detection directly at the edge. Running on NVIDIA Jetson hardware, the solution uses GPU-accelerated AI inference to process video streams efficiently without relying on cloud services, ensuring enhanced privacy, reduced latency, and reliable operation even in offline environments.

⚠️ Note: This project uses ONVIF and unofficial RTSP methods. It is not affiliated with TP-Link. You must enable "Camera Account" (ONVIF/RTSP credentials) inside the official Tapo app for these scripts to function.

🧠 Key Features

• ⚡ Low-Latency PTZ Tracking: Uses a decoupled asynchronous architecture and KCF tracking to physically pan and tilt the camera motors without network buffering lag.
• 🕵️ AI Behavior & Emotion Analysis: Integrates YOLOv8 and DeepFace on background threads to detect complex actions (eating, drinking, working) and facial expressions seamlessly.
• 🌐 FastAPI Web Interface: A lightweight MJPEG web streaming dashboard to register faces, start/stop tracking, and monitor AI status natively in any browser.
• 🌍 Mathematical 360° Panorama: Bypasses standard 2D feature-matching to stitch twelve 30° incremental images into a flawless 360° spherical projection using absolute 3D rotation matrices.
• 🔍 QR & Frame Utilities: Dedicated scripts for reading QR codes and capturing high-res frames from the RTSP stream.

🗂️ Files & Scripts

Script	Purpose
fast2.py	Main Application. The high-performance FastAPI web server for zero-latency streaming, UI control, face registration, and active tracking.
multipleaction.py	Headless tracking script featuring advanced asynchronous YOLO/DeepFace behavior analysis.
panaroma.py	Custom 3D spherical rotation matrix stitcher for creating 360° environments.
newregister.py	Standalone utility for capturing and storing new face encodings via OpenCV.
persondetect.py	Clean, optimized YOLOv8 person detection overlay (no motor control).
send_instruction.py	Script for triggering the Tapo camera's built-in siren and alarms.
image_capture.py	Capture still frames from the high-res camera stream.
image_capture+qr.py	Capture frames and decode embedded QR codes.

Additional required directories include /faces (for storing pickled facial encodings) and local .pt weights for YOLOv8.

🚀 Getting Started

🔁 Prerequisites

• Compute: Nvidia Jetson (Nano, TX2, Xavier, or Orin) running JetPack.
• Camera: TP-Link Tapo C230 connected to the same local network.
• Tapo App Setup: Open the Tapo app -> Camera Settings -> Advanced Settings -> Camera Account -> Create credentials.

🧰 Install Dependencies

Install the required system-level packages for audio and deep learning operations:

sudo apt update
sudo apt install ffmpeg mpg123

Install the required Python libraries:

pip install -r requirements.txt

CRITICAL FOR JETSON: Jetson's native OpenCV compilation requires NumPy 1.x. If installing deep learning libraries auto-upgrades NumPy, you must downgrade it or the tracker will crash: pip install "numpy<2"

📍 Configuration

Before running the main scripts, open them and update the configuration variables at the top of the file with your specific camera's local IP address and the Camera Account credentials you created in the Tapo app:

RTSP_URL = "rtsp://YOUR_USERNAME:YOUR_PASSWORD@192.168.X.X:554/stream1"
ONVIF_IP = "192.168.X.X"
USERNAME = "YOUR_USERNAME"
PASSWORD = "YOUR_PASSWORD"

▶️ Usage Examples

Launch the FastAPI Tracking Dashboard (Recommended)

python fast2.py

Once running, open a web browser on your network and navigate to http://<JETSON_IP>:8000 to access the UI.

Generate a Mathematical Panorama (Ensure you have 12 sequential images captured exactly 30° apart in the working directory).

python panaroma.py

Trigger the Camera Siren

python send_instruction.py

❓ Troubleshooting

• Tracker Initialization Error: If cv2.TrackerKCF_create() fails, the code is designed to auto-fallback to cv2.legacy.TrackerKCF_create(). If both fail, ensure you have installed opencv-contrib-python.
• DeepFace Download Hang: The very first time a script runs emotion detection, it will download the facial_expression_model_weights.h5 file from GitHub. Ensure your Jetson has internet access for this initial run.
• PTZ Motors Not Moving: Verify that your ONVIF_PORT is set correctly (default is usually 2020 for Tapo) and that the Camera Account is properly authenticated.

📜 License

This project is MIT Licensed.

🤝 Contributing

Contributions are welcome! Whether it’s:

Bug fixes

Feature additions

Packaging & examples

Better documentation

Submit a pull request or open an issue