Warsaw University of Technology - Erasmus Exchange Semester
This project implements and compares three different convolutional neural network (CNN) architectures for predicting 15 facial keypoints from 96×96 grayscale images, sourced from a Kaggle competition involving 7,049 images. This research demonstrates the effectiveness of deeper architectures and data augmentation techniques for facial landmark localization tasks.
- Best Model: ResNet18 with Data Augmentation achieving 2.1766 pixels validation RMSE
- Performance Improvement: 44.4% reduction in error compared to baseline Simple CNN (3.9119 pixels)
- Architecture Impact: ResNet18 alone provided 43.6% improvement over Simple CNN
- Data Augmentation Benefits: Additional 1.26% improvement with augmentation techniques
The model detects 15 facial keypoints (30 coordinates total - x,y pairs):
- Eyes: Left/Right eye centers, inner corners, outer corners (6 keypoints)
- Eyebrows: Left/Right eyebrow inner and outer ends (4 keypoints)
- Nose: Nose tip (1 keypoint)
- Mouth: Left/Right mouth corners, mouth center top and bottom lip (4 keypoints)
- Source: Kaggle Facial Keypoint Detection Competition (https://www.kaggle.com/c/facial-keypoints-detection)
- Images: 7,049 grayscale images (96×96 pixels)
- Challenge: ~105,412 missing keypoint annotations out of total possible values
- Split: 5,639 training images, 1,410 validation images
- Data Completeness: 50.4% in validation set
Final-Project-IntroToAI/
├── simple_cnn.py # Basic CNN implementation (TensorFlow/Keras)
├── resnet_model.py # ResNet18-based model (PyTorch)
├── resnet_with_data_augmentation.py # Enhanced ResNet with data augmentation
├── Facial Keypoint Detection using Convolutional - Final Paper.pdf
└── README.md
- Framework: TensorFlow/Keras
- Architecture: Basic convolutional neural network with 16,952,106 parameters
- Results: Validation RMSE: 3.9119 pixels
- Features:
- 2 Conv2D layers (32 and 64 filters) with ReLU activation
- MaxPooling and Dropout layers (0.1, 0.2 rates) for regularization
- Dense layer with 500 units
- Custom masked MSE loss function to handle missing keypoints
- Early stopping (patience: 5 epochs)
- Framework: PyTorch
- Architecture: Modified ResNet18 with ImageNet pretrained weights (~11.45M parameters)
- Results: Validation RMSE: 2.2044 pixels (43.6% improvement over Simple CNN)
- Features:
- Transfer learning from ImageNet with grayscale adaptation
- Modified conv1 layer for single-channel input (224×224)
- Custom final layers: Dropout(0.5) → Linear(512) → Dropout(0.3) → Linear(30)
- Learning rate: 1×10⁻⁴ with ReduceLROnPlateau scheduler
- Early stopping (patience: 10 epochs)
- Framework: PyTorch
- Architecture: Identical to ResNet18 model above
- Results: Validation RMSE: 2.1766 pixels (Best performance, 1.26% improvement over non-augmented)
- Training Time: ~2.2 hours (CPU)
- Image Processing: Pixel values normalized to [0,1] range, resized to 224×224 for ResNet models
- Missing Data Handling: Preserved NaN values, handled via masked loss functions
- Normalization: ImageNet statistics for ResNet models ([0.485] mean, [0.229] std for grayscale)
- Reproducibility: Fixed random seed (8269) across all experiments
The paper details a comprehensive keypoint-aware augmentation pipeline:
- Horizontal Flips (p=0.5): Automatic left/right keypoint coordinate swapping
- Random Rotations (p=0.4): ±10° with geometric keypoint transformation
- Random Scaling (p=0.4): 0.9-1.1× isotropic scaling with crop/pad
- Random Translations (p=0.4): ±8% horizontal/vertical with coordinate adjustment
- Color Jitter (p=0.3): ±0.25 brightness/contrast variation
- Gaussian Noise (p=0.25): σ=0.025 for robustness
- Masked MSE Loss: Custom implementation ignoring NaN values in ground truth
- Masked RMSE: Primary evaluation metric considering only valid keypoints
- Per-keypoint Analysis: Individual RMSE calculation for detailed performance insights
| Model | Validation RMSE | Improvement | Training Time |
|---|---|---|---|
| Simple CNN | 3.9119 pixels | Baseline | ~25 minutes |
| ResNet18 | 2.2044 pixels | 43.6% | ~2.5 hours |
| ResNet18 + Augmentation | 2.1766 pixels | 44.4% | ~2.2 hours |
Keypoint-Level Performance:
- Best Performing:
left_eye_inner_corner_y(1.324 pixels RMSE) - Most Challenging:
mouth_center_bottom_lip_y(3.835 pixels RMSE) - Coordinate Accuracy: Y-coordinates (2.163 pixels) vs X-coordinates (2.256 pixels)
Feature Group Performance:
- Right Eye: 2.027 pixels RMSE (best group)
- Left Eye: Similar performance to right eye
- Mouth Center: 3.149 pixels RMSE (most challenging)
- Nose: 3.101 pixels RMSE
- Performance Ratio: 1.55× (worst/best group)
Key Insights:
- Eye-related keypoints consistently achieve lower errors
- Mouth and nose regions more challenging due to expression variability
- Data availability correlates with performance (Right Eye most annotated, Left Eyebrow least)
# Core frameworks
tensorflow>=2.x # For Simple CNN
torch>=1.x # For ResNet models
torchvision # For pretrained models
# Data processing & analysis
numpy
pandas
matplotlib
scikit-learn
Pillow # PIL for image processing
# Optional for enhanced functionality
jupyter # For notebook analysis
seaborn # For advanced plottingtraining/
└── training.csv # Contains image pixel data and keypoint coordinates
CSV Format:
- Image column: Space-separated pixel values (96×96 = 9,216 values)
- Keypoint columns: 30 float coordinates (x,y pairs for 15 keypoints)
- Missing annotations represented as NaN values
git clone https://github.com/MimisGkolias/Final-Project-IntroToAI.git
cd Final-Project-IntroToAI
pip install -r requirements.txt # Create this file with above dependencies# Run baseline model (fastest)
python simple_cnn.py
# Run ResNet18 model (better performance)
python resnet_model.py
# Run best model with data augmentation
python resnet_with_data_augmentation.py- Simple CNN: ~25 minutes
- ResNet18: ~2.5 hours
- ResNet18 + Augmentation: ~2.2 hours