Semantic Segmentation with PyTorch

Running pretrained CNN-based semantic segmentation on real images — classifying every pixel in a photo, not just labeling the photo as a whole.

Problem

Standard image classification answers "what's in this photo?" Semantic segmentation asks a harder question: "for every pixel in this photo, what does it belong to?" That distinction matters in practice — autonomous driving needs pixel-level road/obstacle boundaries, not just "a car is somewhere in this frame." This project explores that problem hands-on by running a pretrained segmentation model on real images and inspecting exactly which pixels get assigned to which of 150 possible object categories.

Approach

• Set up MIT CSAIL's semantic-segmentation-pytorch framework on Google Colab, using its pretrained model weights rather than training from scratch
• Loaded a ResNet50-dilated encoder paired with a PPM_deepsup (Pyramid Pooling Module) decoder — one of the framework's strongest pretrained configurations, trained on the ADE20K dataset (150 object/scene classes)
• Ran inference on a test image, normalizing it using standard ImageNet statistics before feeding it through the network
• Visualized the full predicted segmentation mask alongside the original image
• Broke down the top 15 predicted classes individually, visualizing which specific pixels the model assigned to each class (e.g. isolating just the "building" pixels, then just the "sky" pixels, etc.)

Results

This was an inference and exploration exercise, not a training run — so there's no accuracy/IoU number of our own to report. What it demonstrates concretely: the pretrained ResNet50dilated + PPM_deepsup model successfully segments real-world photos into coherent per-pixel object regions, and breaking predictions down class-by-class makes it possible to inspect exactly where the model is confident versus where boundaries get blurry (e.g., at object edges or between visually similar categories).

(MIT's own published benchmarks for this exact architecture — Mean IoU ~41.3, Pixel Accuracy ~79.7% on the ADE20K validation set — are the framework's results, not ours, since we didn't train or re-evaluate the model ourselves. Worth citing as context rather than presenting as our own metric.)

Tech Stack

Python, PyTorch, PIL, NumPy, SciPy, torchvision — built on CSAILVision/semantic-segmentation-pytorch

How to Run

pip install yacs torch torchvision numpy scipy pillow

git clone https://github.com/CSAILVision/semantic-segmentation-pytorch.git
cd semantic-segmentation-pytorch
DOWNLOAD_ONLY=1 ./demo_test.sh   # downloads pretrained weights + test image

jupyter notebook Semantic_segmentation.ipynb

Credit

This project runs inference using the pretrained models and framework released by MIT CSAIL:

B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, A. Torralba. Semantic Understanding of Scenes through the ADE20K Dataset. IJCV, 2018.

Original repository: github.com/CSAILVision/semantic-segmentation-pytorch

---

Authors: Khushbu Mahendra Patil, Vafa Khalid University: Ca' Foscari University of Venice Subject: Artificial Intelligence and Image and Video Understanding [CM054]