YOLO-Style Sprite Object Detection

Course: CSCI 495/595, Deep Learning, Spring 2025

Assignment: Research Assignment (10 points)

This repository was split out of a working directory that also contained a separate synthetic-dataset-generation toolkit; see sprite-dataset-generator for the code that produced the training data used here.

Assignment Intent

The assignment asked students to research and implement a convolutional neural network for object detection, segmentation, or classification: pick a domain, pick a research focus among the three, either build a training dataset from scratch or use an existing one, research and implement a CNN training technique on that data, then test the trained model against new unseen data and report the results. The report was expected to cover the domain chosen, the research focus and reasoning, a description of the dataset (size, dimensions, point count), the training code/output, and a discussion of the model's performance on unseen test data.

Confirmed implemented, per notebooks/assignment_5.ipynb (see Overview below): every step is present and verifiable directly in the notebook's own cells and outputs.

• Domain and research focus: object detection on a custom, synthetic sprite domain (game-style target/distractor sprites composited onto background tiles), rather than an existing public dataset.
• Dataset built from scratch: 2,500 synthetic 256x256 samples (1,750 train / 375 val / 375 test), each with exactly 2 "target" and 2 "distractor" objects and YOLO-format bounding-box labels, generated by the sibling sprite-dataset-generator repository. The notebook verifies the dataset's integrity before use (equal image/label counts, expected per-class object counts) rather than assuming it's correct.
• CNN training technique: a custom YOLO-style single-shot detector (build_model(): a 4-block Conv2D/MaxPooling backbone feeding a 16x16 grid detection head) trained with a custom combined loss (focal loss for classification + a grid-based localization/objectness/classification loss), Adam optimization, and early stopping on validation loss.
• Testing on unseen data: the tuned model (NMS + confidence threshold selected via a precision/recall/F1 sweep on the validation set) is evaluated against the held-out test split -- data the model never saw during training or threshold tuning -- reporting precision, recall, and mean IoU.

Overview

notebooks/assignment_5.ipynb (101 cells) walks through the full pipeline:

1. Initialization -- establishes file paths, loads class names (target, distractor) from the dataset's metadata.
2. Exploring Data -- verifies the dataset (equal image/label counts, non-empty labels, expected object counts per class: exactly 2,500 x 2 targets and 2,500 x 2 distractors expected and confirmed), and previews sample images with their bounding boxes drawn and labeled.
3. Preprocessing -- normalizes images to tensors, converts the raw YOLO-format label files into a 16x16 grid target format (format_yolo_labels) matching the model's detection head, and builds tf.data pipelines for train/validation/test.
4. Model -- defines the YOLO-style backbone + detection head, focal loss, and grid loss (localization + objectness + classification combined), and compiles with Adam.
5. Training -- trains for up to 20 epochs with early stopping (patience on val_loss); loss converged from 5.54 -> 0.0036 (training) / 0.39 -> 0.017 (validation) over the logged run, with no sign of divergence. Loss curves are plotted to visually check for overfitting.
6. Fine Tuning -- implements IoU-matrix computation, non-maximum suppression (NMS), and Average Precision (AP) evaluation from scratch; sweeps confidence thresholds against the validation set to find the best NMS IoU / confidence threshold combination via an F1-score curve. Validation-set results at the tuned settings: AP50 0.634, Precision 1.0000, Recall 1.0000, Mean IoU 0.9778.
7. Testing -- applies the tuned settings to the held-out test set: 750 true positives, 0 false positives, 0 false negatives, Precision 1.0000, Recall 1.0000, Mean IoU 0.9780. Sample predictions are visualized against ground truth.
8. Presentation -- saves the trained model (notebooks/dataset_2500/model/yolo_mini_model.keras, committed in this repo).

The perfect precision/recall on the test set should be read in context: the dataset is a controlled synthetic domain (fixed 256x256 canvas, exactly 4 objects per image, 32x32 object size, non-overlapping placement) rather than noisy real-world imagery, so a well-trained detector reaching perfect scores here is plausible and consistent with the domain's difficulty, not evidence of a training bug.

Dataset

The full 2,500-image training set (~272MB) is committed to this repository in full at notebooks/dataset_2500/images/. This is premade, specialized training data -- not something this repository's own code generates -- so it's tracked directly rather than gitignored. notebooks/dataset_2500/images_sample/ additionally keeps 8 representative images for quick visual reference without pulling the full set. Also committed in full:

• notebooks/dataset_2500/labels/ -- all 2,500 YOLO-format label files (~3.7MB)
• notebooks/dataset_2500/datasplits/ -- the exact train/val/test file lists used
• notebooks/dataset_2500/metadata/ -- classes.txt, dataset_2500.yaml
• notebooks/dataset_2500/model/yolo_mini_model.keras -- the trained model (~4.6MB)

If you'd rather generate your own dataset instead of using the committed one (e.g. with different parameters), run generate_dataset.py from the sprite-dataset-generator repository (2,500 samples by default, matching this dataset's parameters exactly -- see that repo's README for how this was confirmed).

Dependencies

A requirements.txt is included (pip install -r requirements.txt): tensorflow, numpy, pandas, matplotlib, scikit-learn, Pillow, plus jupyter/nbconvert to run the notebook itself.

Environment Setup

python -m venv venv
# Windows
venv\Scripts\activate
# macOS/Linux
source venv/bin/activate

pip install -r requirements.txt

The notebook expects the full notebooks/dataset_2500/images/ directory, which is committed in this repository along with labels/, datasplits/, and metadata/ (see Dataset above) -- no separate regeneration step is needed to run the notebook top to bottom.

jupyter notebook notebooks/assignment_5.ipynb

Continuous Integration

A GitHub Actions workflow (.github/workflows/ci.yml) runs on every push and can also be triggered manually via workflow_dispatch. It installs requirements.txt and syntax-checks the notebook (converts it to a plain Python script via nbconvert and compiles it with py_compile) -- this confirms the notebook is free of syntax errors and that every cell's code parses correctly, but it is not a full re-execution.

The full dataset is committed (see Dataset above), so CI has everything it needs to train against, but this workflow deliberately doesn't run a full re-execution: retraining a YOLO-style model end to end is a non-trivial time/resource cost to pay on every push. Syntax-checking catches broken cells cheaply; a full re-execution job (mirroring the approach used for other notebook-based repos in this portfolio) is a reasonable future addition if this repository's CI is revisited.

Known Issues

Security

Code-level findings: none. The notebook performs local file I/O against paths derived from a fixed project structure (no user/network input), no eval/exec, no subprocess/shell execution, no credentials, and no deserialization of untrusted data -- model.save()/loading a .keras file only ever round-trips a model this project trained itself. As with the portfolio's other Keras-based projects: if this saved model is ever loaded by code outside this repository, treat it as untrusted input requiring the same caution as any other .keras/.h5 file (see the Keras model-file findings in this portfolio's other neural-network repositories for the detailed reasoning).

PII exposure -- Fixed, History Squashed. A PII exposure was found and fixed in the working tree, but because the exposed version had already been committed and pushed -- and was, at the time, the current state of this repository's default branch on GitHub -- the entire git history was squashed to a single commit before this push specifically to remove that exposure from history, not just the current working tree.

Status

Complete and verified working. The full pipeline -- dataset verification, preprocessing, custom YOLO-style model, training, NMS/AP-based tuning, and test-set evaluation -- runs end to end per the notebook's own saved outputs, achieving perfect precision/recall on the held-out test set at the tuned confidence/NMS settings. CI syntax-checks the notebook on every push; see Continuous Integration above for why full re-execution isn't run there.