A reproducible transfer learning benchmark comparing ResNet-50, EfficientNet-B3, ViT-B/16, and MobileNet-V3 on the RDD2022 Japan pavement distress dataset. Evaluates five dimensions: accuracy, macro F1, calibration error (ECE), inference latency, and parameter count.
| Model | Test Acc ↑ | F1-Macro ↑ | ECE ↓ | Latency (ms) ↓ | Params |
|---|---|---|---|---|---|
| ResNet-50 | 0.6481 | 0.6276 | 0.1467 | 7.8 | 23.5M |
| EfficientNet-B3 | 0.6190 | 0.6087 | 0.0981 | 8.7 | 10.7M |
| ViT-B/16 | 0.5975 | 0.5617 | 0.1248 | 19.6 | 85.8M |
| MobileNet-V3 | 0.5873 | 0.5697 | 0.1233 | 4.5 | 4.2M |
| Finding | Detail |
|---|---|
| Best accuracy | ResNet-50 (64.8%) — recommended for offline analysis pipelines |
| Best calibration | EfficientNet-B3 (ECE=0.098) — best when confidence scores drive decisions |
| Best edge deployment | MobileNet-V3 (4.5ms, 4.2M params) — suitable for on-device inspection |
| ViT underperforms | 85.8M params yet lowest accuracy — data-hungry; needs larger datasets |
| Accuracy–efficiency tradeoff | EfficientNet-B3 trails ResNet-50 by only 2.9pp at 55% fewer parameters |
RDD2022 Japan — Road Damage Dataset 2022, Japan subset. Images captured via smartphone cameras in moving vehicles. Original YOLO bounding-box annotations converted to image-level labels by assigning each image the dominant (most frequent) damage class — standard practice for architecture benchmarking studies.
| Class | Train | Val | Test |
|---|---|---|---|
| D00 Longitudinal crack | 1,616 | 199 | 198 |
| D10 Transverse crack | 1,038 | 141 | 134 |
| D20 Alligator crack | 2,865 | 346 | 360 |
| D40 Pothole | 801 | 104 | 98 |
| Total | 6,320 | 790 | 790 |
Class imbalance (D20 = 45% of training data) addressed via WeightedRandomSampler.
RDD2022 Japan (YOLO format)
│
▼
convert_to_clf.py ← dominant-class label extraction → ImageFolder layout
│
▼
dataloader.py ← WeightedRandomSampler, augmentation, 224×224 resize
│
▼
train.py ← AdamW + CosineAnnealingLR, 10 epochs, Apple MPS
│
▼
models/*.pth ← best checkpoint per architecture
│
▼
benchmark.py ← accuracy, F1, ECE, latency (100 runs), param count
│
▼
results/ ← benchmark_results.csv + benchmark_results.json
pavement_benchmark/
├── scripts/
│ ├── convert_to_clf.py # YOLO → ImageFolder converter
│ ├── dataloader.py # Weighted dataloader pipeline
│ ├── models.py # 4-architecture builder (torchvision + timm)
│ ├── train.py # Unified training loop
│ └── benchmark.py # 5-metric evaluator
├── results/
│ ├── benchmark_results.csv
│ ├── benchmark_results.json
│ └── training_summary.json
├── README.md
└── .gitignore
Prerequisites: Python 3.10+, pip, RDD2022 Japan dataset
git clone https://github.com/DeekshithaKalluri/pavement-benchmark.git
cd pavement-benchmark
python3 -m venv venv && source venv/bin/activate
pip install torch torchvision timm scikit-learn matplotlib \
seaborn pandas numpy tqdm Pillow netcal opencv-python
# Place RDD2022 Japan data under data/raw/
# Expected layout:
# data/raw/images/{train,val,test}/
# data/raw/labels/{train,val,test}/
python3 scripts/convert_to_clf.py # YOLO → ImageFolder
python3 scripts/train.py # trains all 4 models (~3 hrs on Apple MPS)
python3 scripts/benchmark.py # evaluates all models, saves results/| Hyperparameter | Value |
|---|---|
| Input resolution | 224 × 224 |
| Optimiser | AdamW |
| Learning rate | 1e-4 |
| Weight decay | 1e-4 |
| Scheduler | Cosine Annealing (T_max=10) |
| Epochs | 10 |
| Batch size | 32 |
| Augmentation | H/V flip, rotation ±15°, colour jitter |
| Pretrain weights | ImageNet-1K |
| Device | Apple MPS (M-series GPU) |
Architecture selection. Four architectures spanning the CNN-to-transformer spectrum: ResNet-50 (classic residual CNN, 23.5M), EfficientNet-B3 (compound-scaled CNN, 10.7M), ViT-B/16 (pure transformer, 85.8M), MobileNet-V3-Large (lightweight depthwise CNN, 4.2M). All backbone weights initialised from ImageNet-1K; only the final classification head is replaced for the 4-class task.
Calibration metric. Expected Calibration Error (ECE) measures how well a model's confidence aligns with its empirical accuracy, using 10 equal-width bins over [0,1]. Lower ECE = more reliable confidence scores — critical for threshold-based alerting in inspection systems.
Latency measurement. Mean single-image inference time over 100 runs after 10 warm-up passes on Apple MPS backend. Reported as mean ± std (ms).
Class imbalance. D20 alligator crack constitutes ~45% of training images. WeightedRandomSampler equalises effective class frequency per epoch without duplicating images.
YOLO → classification conversion — RDD2022 provides bounding-box annotations, not image-level labels. Resolved by assigning each image the most frequent damage class across its annotations, reducing the detection task to a classification benchmark while preserving label semantics.
ViT data hunger — ViT-B/16 underperformed all CNN baselines despite 8× more parameters than EfficientNet-B3. Self-attention lacks the locality and translation-invariance inductive biases of CNNs, requiring far more data to generalise. With only 6,320 training images, CNNs win decisively.
ECE vs accuracy — EfficientNet-B3 ranks second on accuracy but first on calibration (ECE=0.098 vs ResNet-50's 0.147). For real-world inspection pipelines where a confidence threshold triggers a human review, calibration matters as much as raw accuracy.
Apple MPS backend — torch.backends.mps.is_available() enables GPU-accelerated training on M-series Macs without CUDA, reducing per-epoch time from ~500s (CPU) to ~70-260s depending on model size.
| Layer | Tool |
|---|---|
| Language | Python 3.13 |
| Deep learning | PyTorch 2.x, torchvision |
| ViT backbone | timm |
| Metrics | scikit-learn |
| Device | Apple MPS (M-series GPU) |
| Version control | Git / GitHub |
Dataset: RDD2022 — Arya et al., 2022. A multi-national image dataset for automatic road damage detection. Used strictly for research and educational purposes.
MIT — see LICENSE
Deekshitha Kalluri — GitHub