Real-time multi-person tracking and soft biometric attribute classification using YOLOv8, BotSort, and a custom MNAT deep learning model
This project is part of the Artificial Vision Contest 2024/2025 at the University of Salerno. The system performs real-time multi-person tracking and soft biometric attribute classification from surveillance video footage, leveraging deep learning and computer vision techniques. Given a video stream and a camera configuration file, the pipeline detects and tracks individuals across frames, classifies each person's soft attributes (gender, hat, bag), and logs their trajectories relative to configurable virtual line sensors projected from 3D world coordinates onto the image plane.
- ✅ Multi-Person Detection & Tracking: Uses YOLOv8 with BotSort tracker for robust real-time person tracking
- ✅ Soft Attribute Classification: Custom MNAT model predicts gender, hat presence, and bag presence per tracked individual
- ✅ Virtual Line Sensors: 3D-to-2D projection of configurable floor lines to detect and count person crossings
├── 📁 code/
│ ├── 📁 data/
│ │ ├── 📁 configs/
│ │ │ ├── botsort.yaml
│ │ │ └── config.json
│ │ ├── 📁 videos/
│ │ │ └── Contest.mp4
│ │ └── 📁 results/
│ │ └── output.txt
│ ├── 📁 models/
│ │ ├── yolov8m.pt
│ │ └── classifiers.pth
│ ├── 📁 modules/
│ │ ├── 📁 classifier/
│ │ │ ├── CBAM.py
│ │ │ ├── MLP.py
│ │ │ └── MNAT.py
│ │ ├── classification.py
│ │ └── virtual_sensors.py
│ ├── 📁 utils/
│ │ ├── bbox.py
│ │ └── video_utils.py
│ └── main.py
├── requirements.txt
├── LICENSE
└── README.md
The pipeline is designed as a sequential processing loop operating frame-by-frame over the input video. At each step, the YOLOv8 tracker produces bounding boxes and persistent track IDs for all detected persons. These detections are then passed through two parallel processing branches: the attribute classifier and the virtual sensor engine.
The classification branch leverages the custom MNAT (Multi-task Normalized Attention Transformer) model, which takes a cropped and resized person image (64×192 pixels) as input and outputs sigmoid-activated predictions for three binary soft attributes — gender, hat, and bag. To improve temporal consistency, classification is not run on every frame; instead, per-ID confidence scores are averaged over multiple evaluations using a rolling window strategy, reducing noise from partial occlusions or motion blur.
The virtual sensor branch uses the LineSensor class to project 3D floor lines — defined in world coordinates within config.json — onto the 2D image plane using a full perspective camera model (rotation, translation, focal length, and sensor size). For each tracked person, the system checks whether the movement vector between consecutive frame positions intersects any projected virtual line, applying geometric tolerances to handle oblique crossings. When a valid crossing is detected, the line ID is recorded in that person's trajectory log and a passage counter is incremented.
At the end of the run, all results are serialized to a structured JSON file containing, for each tracked person: their ID, gender, hat and bag attributes, and the ordered list of virtual line IDs they crossed.
- Python 3.8+
- CUDA-compatible GPU (recommended for real-time inference)
- PyTorch with CUDA support (
torch==2.3.1+cu118) - Ultralytics YOLOv8
# Clone the repository
git clone https://github.com/Crostino14/Artificial-Vision-Project.git
cd Artificial-Vision-Project
# Install dependencies
pip install -r requirements.txtIf you're interested to use this project ask for the pre-trained weights of the classifier!
Place the following pre-trained model files inside code/models/ before running:
yolov8m.pt— YOLOv8 medium detector (downloadable viaultralytics)classifiers.pth— Pre-trained MNAT attribute classifier checkpoint
cd code
# Run with default configuration
python main.pyThe default paths are configured directly in main.py:
video_path = "data/videos/Contest.mp4"
config_path = "data/configs/config.json"
tracker_name = "botsort"
output_path = "data/results/output.txt"
frame_skip = 2The camera and virtual line configuration is loaded from data/configs/config.json. This file must define the camera extrinsic/intrinsic parameters and the list of floor lines in world coordinates:
{
"xc": 0.0, "yc": 0.0, "zc": 5.0,
"thpitch": 0.0, "throll": 0.0, "thyaw": 0.0,
"sw": 36.0, "sh": 24.0,
"f": 35.0,
"U": 1920, "V": 1080,
"lines": [
{ "id": 1, "x1": -3.0, "y1": 2.0, "x2": 3.0, "y2": 2.0 },
{ "id": 2, "x1": -3.0, "y1": -1.0, "x2": 3.0, "y2": -1.0 }
]
}Results are saved as a JSON file with the following structure:
{
"people": [
{
"id": 1,
"gender": "m",
"hat": false,
"bag": true,
"trajectory":
}
]
}Each entry in trajectory corresponds to the ID of a virtual line the person crossed, in chronological order.
A custom multi-task neural network for soft attribute recognition. The architecture integrates a CBAM (Convolutional Block Attention Module) for spatial and channel-wise attention, combined with an MLP head per attribute. The model accepts person crop images resized to 64×192 and outputs three independent binary predictions via sigmoid activation.
Implements the LineSensor class which handles the full 3D-to-2D projection pipeline. Using camera intrinsics and extrinsics from the config file, it computes rotation and translation matrices to project floor-level world coordinates onto image pixels. Crossing detection uses segment intersection geometry with configurable perpendicular tolerance (40.0 px) and directional threshold (-0.4) to filter spurious detections.
The VideoLoader class wraps OpenCV video I/O and provides frame-level iteration with real-time overlay rendering — drawing bounding boxes, track IDs, attribute labels, virtual lines, and passage counters directly onto each frame for visual inspection.
The BoundingBox utility class encapsulates bounding box geometry, providing methods to extract person crops from raw frames (extract_person_images) and compute foot-point centers for line-crossing estimation.