Synthetic Drone Detection Dataset Generator

中文文档 (Chinese)

Generate photorealistic synthetic datasets for training drone detection models (YOLOv8, etc.) using Blender Cycles path-tracing with HDRI Image-Based Lighting.

The key idea: HDRI panoramic skies provide physically accurate lighting that simultaneously serves as the background and light source, ensuring the 3D drone model's shading, shadows, and reflections are naturally consistent with the environment — no manual light setup needed.

Features

• Photorealistic rendering — Blender Cycles path tracing with OIDN denoising
• Automatic YOLO annotations — Bounding boxes computed via 3D-to-2D projection, 100% accurate
• Domain randomization — Sky background, sun direction, light intensity, viewing angle, distance, FOV, and flight attitude are all randomized
• Physically constrained poses — Drone roll/pitch follow Gaussian distributions with flight-envelope limits
• Multiple engagement profiles — Weighted sampling from tail-chase, side-approach, head-on, above, and below viewpoints
• Dataset splitting — Automatic train/val/test split with YOLOv8-ready dataset.yaml
• Real-time mode — EEVEE/Cycles-low renderer with ZMQ frame publishing for HITL simulation
• CPU compatible — Works without GPU (slower but same quality), ideal for CI/CD pipelines

Quick Start

1. Install Blender (portable, no sudo)

cd ~
wget https://mirror.clarkson.edu/blender/release/Blender4.2/blender-4.2.0-linux-x64.tar.xz
tar xf blender-4.2.0-linux-x64.tar.xz
export BLENDER=~/blender-4.2.0-linux-x64/blender

2. Download assets

# Download HDRI sky maps and a sample drone model
bash download_assets.sh

Or manually:

• HDRI skies: Download 4K .hdr files from Polyhaven Skies into hdri/
• 3D drone model: Download a .glb model from Sketchfab into models/

3. Generate dataset

$BLENDER --background --python render_cycles.py -- \
    --model models/dji_mavic_3.glb \
    --hdri-dir hdri \
    --output output/drone_dataset \
    --num 1500 \
    --width 1920 --height 1080 \
    --samples 32

4. Train YOLOv8

pip install ultralytics
yolo detect train data=output/drone_dataset/dataset.yaml model=yolov8n.pt epochs=100 imgsz=1080

Output Structure

output/drone_dataset/
├── images/             # All rendered images
├── labels/             # YOLO format annotations
├── train/
│   ├── images/         # 80% training split
│   └── labels/
├── val/
│   ├── images/         # 15% validation split
│   └── labels/
├── test/
│   ├── images/         # 5% test split
│   └── labels/
├── annotations.json    # Full metadata (distance, FOV, attitude, profile, etc.)
└── dataset.yaml        # YOLOv8 config file

Real-time Rendering

For simulation-in-the-loop testing, use the real-time renderer:

$BLENDER --background --python render_eevee_realtime.py -- \
    --model models/dji_mavic_3.glb \
    --hdri-dir hdri \
    --mode demo \
    --duration 15 \
    --save-frames output/realtime_demo \
    --fallback cycles-low

Frames are published via ZMQ PUB socket (default port 5555). Receive them in another process:

import zmq, numpy as np, cv2

ctx = zmq.Context()
sub = ctx.socket(zmq.SUB)
sub.connect('tcp://127.0.0.1:5555')
sub.setsockopt(zmq.SUBSCRIBE, b'')

while True:
    header = sub.recv_json()
    buf = sub.recv()
    img = np.frombuffer(buf, dtype=np.uint8).reshape(header['h'], header['w'], 3)
    cv2.imshow('drone_cam', img[..., ::-1])
    cv2.waitKey(1)

CLI Reference

render_cycles.py (Offline Dataset Generation)

Argument	Default	Description
--model	(required)	Path to .glb/.obj/.fbx drone model
--hdri-dir	hdri	Directory containing .hdr/.exr files
--output	output/dataset	Output directory
--num	300	Number of images to generate
--width	1920	Image width
--height	1080	Image height
--samples	32	Cycles samples per pixel
--min-pixels	100	Minimum target pixel area to keep
--train-ratio	0.8	Training set ratio
--val-ratio	0.15	Validation set ratio
--class-name	drone	Object class name in annotations
--target-size	3.5	Normalize model to this size (meters)
--seed	42	Random seed for reproducibility

render_eevee_realtime.py (Real-time Simulation)

Argument	Default	Description
--model	(required)	Path to .glb drone model
--hdri-dir	hdri	Directory with .hdr files
--hdri	(auto)	Specific HDRI file to use
--mode	demo	demo or server
--duration	15.0	Demo trajectory duration (seconds)
--fps-target	30.0	Target frame rate
--fallback	eevee	Engine: eevee, workbench, or cycles-low
--save-frames	(none)	Directory to save rendered frames
--zmq-pub-port	5555	ZMQ PUB port for frame output

Performance

Configuration	Per-frame	1500 images
RTX 4090 (native Linux)	~0.15s	~4 min
i7-12700 CPU	~2.0s	~50 min
i7-12700 CPU (WSL2)	~2.5s	~63 min

How It Works

1. HDRI as unified environment — A single HDR panorama provides background, lighting, and reflections simultaneously via Blender's IBL (Image-Based Lighting)
2. Domain randomization — Each frame randomizes: HDRI selection, sky rotation, light intensity, camera-target geometry (5 weighted profiles), drone attitude (physically constrained), and FOV
3. 3D bounding box projection — Uses obj.bound_box corners projected through world_to_camera_view() for pixel-accurate YOLO annotations without rendering a segmentation mask
4. Quality filtering — Images where the target is too small (< min_pixels) or off-screen are automatically discarded

Engagement Profiles

The generator includes 5 weighted engagement profiles simulating typical interception viewpoints:

Profile	Weight	Azimuth	Elevation	Distance (m)	FOV (°)	Description
tail_chase	35%	±30°	-9°~14°	30–200	25–50	Pursuit from behind — most common
side_approach	25%	57°~123°	-6°~17°	40–300	25–50	Lateral crossing approach
head_on	15%	160°~200°	-6°~9°	50–400	20–45	Head-on with high closing speed
above_rear	15%	±23°	17°~40°	20–150	25–55	Above and behind the target
below_close	10%	±34°	-23°~-6°	20–80	30–55	Close range from below

Drone attitude is also randomized per frame: roll follows N(0, 3°), pitch N(-2°, 2°), yaw is uniform random — all clamped to realistic flight-envelope limits.

Tuning Guide

Recommended Configurations

Quick prototyping (~5 min):

$BLENDER --background --python render_cycles.py -- \
    --model models/dji_mavic_3.glb --hdri-dir hdri \
    --output output/quick_test --num 50 --samples 16 --width 640 --height 480

Full training set (~50 min CPU / ~4 min GPU):

$BLENDER --background --python render_cycles.py -- \
    --model models/dji_mavic_3.glb --hdri-dir hdri \
    --output output/drone_dataset --num 1500 --samples 32 --width 1920 --height 1080

High-quality paper figures (~30 min / 100 images):

$BLENDER --background --python render_cycles.py -- \
    --model models/dji_mavic_3.glb --hdri-dir hdri \
    --output output/paper_figures --num 100 --samples 256 --width 1920 --height 1080

Key Parameter Effects

Parameter	Increase	Decrease	Recommended
--samples	Less noise, slower	More noise, faster	32 (sufficient with OIDN denoising)
--target-size	Larger drone in scene	Smaller drone	3.5 (normalizes ~0.35m wingspan)
--min-pixels	Filters more small targets	Keeps more distant targets	100 (training) / 50 (with hard negatives)
--width/height	Higher resolution	Faster rendering	1920×1080 (match deployment)

Technical Details

Why Blender Cycles + HDRI?

After evaluating multiple rendering approaches (Gazebo OGRE2, 3D Gaussian Splatting compositing, Blender EEVEE, etc.), Cycles + HDRI was validated as the approach with the smallest sim-to-real visual domain gap:

Approach	Lighting Physics	Speed	Sim-to-Real Gap
Gazebo OGRE2	No IBL, crude lighting	Real-time	Large
3DGS + HDRI compositing	No relighting, baked colors	Real-time (~50 FPS)	Medium-Large
Blender EEVEE	Approximate IBL, some reflections	Near real-time	Medium
Blender Cycles + HDRI	Path-traced IBL, physically accurate	Offline (~2s/frame)	Small

Core advantages of Cycles path tracing:

1. Correct light interaction — HDRI light rays properly reflect, refract, and scatter on the drone surface
2. Natural shadows — Self-shadows and ambient occlusion computed automatically
3. PBR materials — Metallic, roughness, normal maps fully preserved; appearance changes naturally under different lighting
4. Unified environment — A single HDRI provides background, lighting, and reflections simultaneously, ensuring perfect consistency in color temperature, brightness, and directionality

Rendering Pipeline

                    ┌─────────────┐
                    │  HDRI Sky   │ (Polyhaven 4K .hdr)
                    └──────┬──────┘
                           │ IBL (Image-Based Lighting)
                           ▼
┌──────────┐      ┌────────────────┐      ┌──────────────┐
│ GLB Model│ ──→  │ Blender Cycles │ ──→  │  JPEG Image  │
│(PBR mat.)│      │  Path Tracing  │      │ + YOLO Label │
└──────────┘      └────────┬───────┘      └──────────────┘
                           │
              Domain randomization: HDRI rotation,
              light intensity, camera distance/angle/FOV,
              drone attitude

Annotation Method

YOLO annotations are computed via 3D-to-2D projection (not pixel-level segmentation):

1. Get all mesh bound_box corners (8 vertices per object)
2. Project via bpy_extras.object_utils.world_to_camera_view()
3. Compute the minimum bounding rectangle in image coordinates
4. Convert to YOLO normalized format (class, cx, cy, w, h)

This method is 100% accurate and requires no extra render passes (like segmentation masks), orders of magnitude faster than manual annotation.

WSL2 Notes

This project has been tested on WSL2 (Ubuntu). Key notes:

1. Blender requires Xvfb on headless WSL2:

   sudo apt install xvfb
   xvfb-run -a $BLENDER --background --python render_cycles.py -- ...

2. GPU rendering: WSL2 supports CUDA, but ensure NVIDIA drivers and CUDA toolkit are installed. The renderer falls back to CPU automatically if GPU is unavailable.

3. EEVEE limitations: WSL2 has limited OpenGL support. render_eevee_realtime.py provides --fallback cycles-low as an alternative.

4. Disk I/O: Keep the project on the Linux filesystem (/home/) rather than Windows mount paths (/mnt/c/) for better performance.

Full YOLOv8 Workflow

# 1. Download assets
bash download_assets.sh

# 2. Generate training data (~50 min CPU)
export BLENDER=~/blender-4.2.0-linux-x64/blender
xvfb-run -a $BLENDER --background --python render_cycles.py -- \
    --model models/dji_mavic_3.glb \
    --hdri-dir hdri \
    --output output/drone_dataset \
    --num 1500 --samples 32

# 3. Verify annotations (optional)
python3 -c "
import json
ann = json.load(open('output/drone_dataset/annotations.json'))
print(f'Images: {len(ann)}')
profiles = {}
for a in ann:
    p = a['profile']
    profiles[p] = profiles.get(p, 0) + 1
for k, v in sorted(profiles.items(), key=lambda x: -x[1]):
    print(f'  {k}: {v} ({v/len(ann)*100:.0f}%)')
"

# 4. Train YOLOv8
pip install ultralytics
yolo detect train \
    data=output/drone_dataset/dataset.yaml \
    model=yolov8n.pt \
    epochs=100 \
    imgsz=1080 \
    batch=16

# 5. Evaluate
yolo detect val \
    data=output/drone_dataset/dataset.yaml \
    model=runs/detect/train/weights/best.pt

Requirements

• Blender 4.0+ (tested with 4.2.0)
• Python 3.10+ (bundled with Blender)
• Optional: xvfb (WSL2 / headless environments)
• Optional: pyzmq for real-time frame publishing
• Optional: ultralytics for YOLOv8 training

License

MIT License. See LICENSE for details.

HDRI assets from Polyhaven are licensed under CC0.