Computer Vision Target Tracking for Civilian Drones

A video-based object detection, appearance matching, and target lock-on pipeline built with YOLOv8 and OpenCLIP.

The system detects objects in recorded video or image sequences, compares candidate crops against a multi-reference embedding bank, and maintains a target lock using a state machine with region-of-interest recovery.

The repository is intended for civilian computer-vision research, inspection, and tracking experiments using prerecorded media.

Engineering highlights

• YOLOv8 detection and tracking
• OpenCLIP appearance embeddings for ReID-style similarity
• Multi-reference target bank for changes in angle, scale, lighting, and partial occlusion
• Search/locked state machine with ROI-based compute reduction
• Configurable target acquisition, retention, switching, and recovery thresholds
• Full-frame recovery when the target leaves the active ROI
• JSON, annotated-image, and video outputs
• CPU, desktop CUDA, and Jetson-oriented configuration guidance

Technology

Python · YOLOv8 · OpenCLIP · PyTorch · OpenCV · NumPy · Video Processing · Embedding Similarity

Pipeline

Video or frame sequence
        │
        ▼
YOLO detection / tracking
        │
        ▼
Candidate crops
        │
        ▼
OpenCLIP embeddings ── compare with reference bank
        │
        ▼
Target score and selection
        │
        ▼
SEARCH / LOCKED state machine
        │
        ▼
Annotated output, logs, and metrics

Lock-on behavior

SEARCH

• Run detection on the full frame
• Compute appearance similarity for eligible candidates
• Select the best candidate
• Enter LOCKED when the acquisition threshold is reached

LOCKED

• Build an ROI around the previous target location
• Detect and score candidates inside the ROI
• Expand the ROI when the target is temporarily missing
• Periodically perform global recovery
• Return to SEARCH after the configured lost-target TTL

Repository structure

src/
├── build_ref_bank.py   # Build the multi-image embedding bank
├── clip_embedder.py    # OpenCLIP encoding and normalization
├── detect_frame.py     # Single-frame detection and structured output
├── live_runner.py      # Full-frame detection/tracking with similarity
└── lockon_runner.py    # Search/lock state machine and ROI recovery

.env.example            # Runtime configuration reference
requirements.txt        # Python dependencies

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Prepare several clean reference crops:

data/ref_bank/
├── front.jpg
├── side.jpg
└── rear.jpg

Build the embedding bank:

python src/build_ref_bank.py \
  --ref_dir data/ref_bank \
  --out_npy outputs/ref_bank.npy \
  --out_json outputs/ref_bank.json

Run the lock-on pipeline:

python src/lockon_runner.py --env_file .env

Runtime controls:

• q: quit
• p: pause or resume
• n: process one frame while paused

Reference-bank strategy

Each candidate crop is converted to a normalized embedding and compared against all reference embeddings. The repository supports several aggregation strategies:

• max: strongest individual match
• mean: average similarity across references
• topk_mean: average of the strongest matches; useful when the bank covers several viewpoints

Reference quality matters more than quantity. Tight crops with varied angles and lighting generally produce better matching than large images dominated by background.

Performance controls

The configuration exposes controls for:

• detector model and confidence
• target classes
• acquisition and retention thresholds
• reference aggregation and top-k size
• ROI scale and recovery intervals
• candidate gates and score weighting
• input/output paths and application limits

For embedded hardware, use a smaller detector, reduce frame resolution, limit candidate classes, precompute the reference bank, and tune how often full-frame recovery runs.

Outputs

Depending on the selected command, the pipeline can produce:

• structured detection JSON
• reference-bank metadata
• annotated frames or video
• per-frame similarity and target-state logs
• timing information for performance tuning

Project status

This repository is a practical computer-vision prototype that demonstrates object detection, embedding-based matching, stateful target tracking, ROI optimization, and deployment-aware configuration.