Empirical Optimization and Architectural Forensics of ResNet-18

This project provides a comprehensive implementation, optimization, and forensic analysis of a custom ResNet-18 architecture trained from scratch on the ImageNet-100 dataset. Developed as part of the COL775: Deep Learning course at the Indian Institute of Technology (IIT) Delhi, this study moves beyond baseline configurations to investigate the mathematical dynamics of normalization schemes and regularization strategies.

Overview

The primary objective of this repository is to deconstruct the generalization gaps inherent in deep convolutional networks and engineer a robust optimization pipeline. By utilizing a rigorous 5-Round Progressive Ablation Protocol, we identified and mitigated critical failure modes such as Shortcut Collapse, Feature Washout, and Spatial Dimensionality Collapse.

Key Achievements

• Architecture: A full ResNet-18 implementation built entirely from first principles using vanilla PyTorch.
• Normalization Research: Custom implementations and comparative benchmarks for Batch, Instance, Batch-Instance, Layer, and Group Normalization.
• Peak Performance: Achieved a top-1 validation accuracy of 82.06% on ImageNet-100, outperforming native baseline configurations.
• Interpretability: Integration of Grad-CAM to visualize spatial attention and diagnose failure modes in incorrect classifications.

Technical Methodology

5-Round Progressive Ablation Protocol

To isolate variables and optimize the compute budget, the model development followed a gated ablation strategy:

1. Micro-Ablation (5 Epochs): Diagnosed early failures such as Zero-Initialization causing shortcut collapse.
2. Combinatorial Synergy (15 Epochs): Validated the interaction between Nesterov SGD and Strong AutoAugment.
3. Generalization Compression (30 Epochs): Tuned dropout layers to bridge the training-validation gap (compressed to 0.6%).
4. Master Baseline Validation (100 Epochs): Established the optimal set of hyperparameters for full-scale training.
5. Normalization Showdown: A mathematical "level playing field" comparison of six different normalization topologies.

Optimized Hyperparameter Configuration

Parameter	Value
Optimizer	SGD with Nesterov Momentum (mu = 0.9)
Learning Rate	0.05 with Cosine Annealing
Warmup	5 Epochs Linear
Weight Decay	1e-3 (Selective to 2D/4D weights)
Normalization	Custom Batch Normalization (gamma_init = 1.0)
Regularization	Dropout (p = 0.35), Label Smoothing (0.1)
Augmentation	Strong AutoAugment (phflip = 0.2)

Repository Structure

.
├── code/
│   ├── src/
│   │   ├── resnet.py            # From-scratch ResNet-18 architecture
│   │   ├── normalization.py     # Custom Normalization layer implementations
│   │   ├── normalization_cuda.py# Optimized CUDA-specific normalization
│   │   └── loss.py              # Custom loss functions (Label Smoothing)
│   ├── training.py              # Main training orchestration script
│   ├── engine.py                # Training and evaluation loops
│   ├── dataset.py               # Data loading and augmentation pipeline
│   ├── gradcam.py               # Grad-CAM visualization engine
│   ├── utils.py                 # Logging, telemetry, and checkpointing
│   ├── prediction.py            # Model inference and analysis
│   └── report.pdf               # Comprehensive technical report
└── COL775_A1_Part1.pdf         # Assignment specifications

Implementation Highlights

Custom Normalization

The project includes "from-scratch" implementations of the following modules, inheriting from torch.nn.Module:

• Batch Normalization (BN): Includes vectorized running statistics and Bessel's correction.
• Layer Normalization (LN): Analysis revealed "Feature Washout" where localized spatial features were squashed by global variance.
• Group Normalization (GN): Diagnosed a "Dead ReLU" crisis at micro-batch sizes due to spatial dimensionality collapse (1x1 dimensions).
• Batch-Instance Normalization (BIN): A hybrid approach for adaptively style-invariant features.

Interpretability via Grad-CAM

The gradcam.py module computes gradients of input feature maps with respect to target classes, providing heatmaps that explain model decisions.

• Correct Classifications: Show localized attention on primary structural features.
• Incorrect Classifications: Reveal distractions by spurious background correlations or occlusions.

Setup and Usage

Requirements

• Python 3.10+
• PyTorch 2.0+
• torchvision
• matplotlib, numpy, pandas

Training

To execute the training pipeline with the optimized configuration:

python code/training.py --batch_size 128 --epochs 100 --norm_type batch_norm

Evaluation and Grad-CAM

To generate class activation maps for the best performing model:

python code/gradcam.py --checkpoint path/to/model.pth --num_images 5

Credits

Author: Rishit Jakharia Course: COL775: Deep Learning, IIT Delhi Date: March 2026