This is the repository for a computer vision research project (CIS*4900) I worked on with Dr. Stefan Kremer. It analyzes soccer footage with machine learning and computer vision techniques, and extracts soccer statistics such as possession from the data. For detailed information about the objects, implementation and results of the project, please see my report.
Soccer teams have begun to use statistics and artificial intelligence to enhance tactical decision making, scouting, among many other uses. Some researchers and soccer teams have used computer vision techniques to analyze soccer footage, as a means to provide enhanced analysis. This research aims to analyze and improve existing methodologies of soccer footage through computer vision, to identify optimal solutions and existing challenges within this domain. Specifically, it seeks to detect soccer players, referees and the ball, perform perspective transformation of these detected objects into a 2D view, and then perform analysis of the footage by computing relevant statistics. YOLOv8 (You Only Look Once) models were used for object and keypoint detection, with the keypoints being used for perspective transformation of the detected objects. Text embeddings were used to improve team classification. The team classification and object detection methodology proved to be robust, but accurate keypoint detection was an issue with perspective transformation. These findings suggest that alternative methods should be studied in the future to improve the capabilities of perspective transformation with sub-optimal footage.
Footage Annotated with Detections
Same Footage, but Transformed into 2D View
The key features of this project are:
- MLS Next PRO Dataset
- Object Detection of Ball, Referee and Players
- Perspective Transformation from 3D to 2D
- Team Classification
- Possession Calculuation
I have described these in simplified terms below, but they're described in more detail in my report.
I recorded about 30 videos and created an annotated dataset of images, in the images-object and images-pitch directory. The videos were not included due to GitHub memory limits.
To identify the players in the footage, I trained a YOLOv8 object detection model, with a player, goalkeeper, ball and referee class. The model differentiates between these 4 classes, but does not differentiate between goalkeepers and players on different teams. These detections are encapsulated in a bounding box that is annotated on the screen, and the model returns cartesian coordinates that correspond to the centre of this bounding box for each detection.
To conduct perspective transformation from 3D to 2D, the following steps were performed for each frame:
- Identify relevant objects with object detection (see above section).
- Identify pre-selected pitch keypoints in the image.
- Compute a homography matrix with the pitch keypoints' coordinates.
- Transform the 3D coordinates provided by the object detection into 2D using the homography matrix mapping.
- Plot them onto the 2D representation of the pitch.
To identify pitch keypoints, I identified 36 pitch keypoints, and annotated them across 1,000+ images. I trained a YOLOv8 pose detection model to identify these pitch keypoints. When I run the model, it identifies any keypoints that are shown in the image, and returns their corresponding cartesian coordinates.
The homography matrix is computed using the pitch keypoints' coordinates, as well as their corresponding mapping in the 2D representation of the pitch.
After calculating the matrix, the matrix transforms any coordinate in 3D space into a new coordinate that corresponds to the 2D representation of the pitch. This coordinate is plotted onto that pitch, representing the 2D representation of the original object detection of a player, ball or referee.
Given that 'player' is a single class in the object detection model, it does not differentiate between players on different teams. Thus, these need to be separated, and they are best classified by a separate model. The obvious method to differentiate between teams is by jersey colour. However, given lighting differences and the fact that a significant amount of every image contains the green colour of the grass, this is not robust enough. Thus, for each bounding box image of a player detection, image embeddings were extracted using SIGLip, dimensionality reduction was performed using UMAP, and the classifier used was K-Means, with 2 clusters (corresponding to 2 teams) were classified.
These steps were used to compute possession statistics:
- Create counters for the number of frames where team 1 was assigned possession, and where team 2 was assigned possession.
- For each frame's corresponding 2D coordinates:
- Compute the Euclidean distance for each player with respect to the ball.
- Determine which player has the shortest Euclidean distance to the ball.
- Increment the corresponding possession team counter of that corresponding player.
- Compute the possession for each team by using the ratio of times the team was in possession in all of the frames relative to the number of total frames.
You must set one of your environment variables with the API key from Roboflow to run this model.
You must have Python version 3.11 or below. You must also set-up CUDA 11.8.
python -m venv soccer-env
In cmd.exe
soccer-env\Scripts\activate.bat
In PowerShell
soccer-env\Scripts\Activate.ps1