Bachelor's Thesis · B.Sc. · University of Bremen · 2024
Adapting a state-of-the-art saliency prediction model to forecast visual attention patterns for people with visual impairments - using only the original, unmodified image as input.
Saliency prediction - modelling which regions of an image attract human visual attention - has been studied extensively. However, how visual attention shifts for people with impaired vision remains largely unexplored. This project investigates whether a deep learning model can predict saliency under simulated visual impairments, given only the original image and a flag indicating the type of impairment. No pre-processed or modified images are required at inference time.
Three impairments are simulated to generate reference saliency maps (ground truth):
| Modification | Simulates |
|---|---|
| Contrast reduction | Myopia (near-sightedness) |
| Colour vision deficiency (CVD) | Colour blindness |
| Grey circle overlay | Macular degeneration (central blind spot) |
Combinations of these modifications were also evaluated, even though the model was only trained on single modifications.
The base model is DeepGazeIIE, a state-of-the-art saliency prediction model. The architecture was adapted to accept an additional input - a modification flag - alongside the original image. The modified model files are located in modded_deepgaze_files/ and replace the corresponding files in the DeepGaze library.
- Training: 10,000 images (SALICON dataset)
- Evaluation: 4,012 images (MIT1003 dataset)
- Ground truth saliency maps were generated computationally using the pysaliency library on modified images - not collected from human study participants (see Limitations)
The SIM score (Similarity Metric, following Judd et al., 2012) was used throughout. A score of 1 indicates perfect agreement with ground truth; 0 indicates no overlap.
| Modification | Baseline | Adaptive Model | Oracle (modified image) | GT Similarity |
|---|---|---|---|---|
| Contrast | 0.780 | 0.791 | 0.744 | 0.850 |
| CVD | 0.811 | 0.814 | 0.822 | 0.915 |
| Grey circle | 0.788 | 0.788 | 0.829 | 0.897 |
Baseline: DeepGazeIIE trained without modification awareness. Oracle: model receives the modified image directly (theoretical upper bound). GT Similarity: similarity between original and modified ground truth.
| Combination | Baseline | Adaptive Model | Oracle |
|---|---|---|---|
| Contrast + CVD | 0.761 | 0.761 | 0.729 |
| Contrast + Grey circle | 0.741 | 0.746 | 0.735 |
| CVD + Grey circle | 0.772 | 0.764 | 0.821 |
The model was never exposed to combined modifications during training.
While the adaptive model shows marginal quantitative improvements for contrast and CVD modifications, qualitative inspection reveals that the model does not meaningfully learn to predict modification-specific saliency. The small SIM-score gains reflect only minor changes in the predictions, not a genuine adaptation to the impairments. For the grey circle modification, no improvement over the baseline is observed at all.
The combined-modification results confirm that the model cannot generalise to unseen modification combinations. Passing only one of the two modification flags consistently improves results, suggesting the model cannot process simultaneous modification signals.
Overall: the goal of predicting impairment-specific saliency from the original image alone was not achieved with this approach.
- Ground truth saliency maps are computationally simulated, not collected from participants with actual visual impairments
- The model was trained only on single modifications; combination generalisation was not part of the training objective
- Dataset size and training duration may have been insufficient; a larger dataset or different architecture may yield better results
- Possible directions for future work: contrastive loss functions to better distinguish modifications; real-world ground truth collection with participants
.
├── Bildmanipulation.ipynb # Image modification pipeline (contrast, CVD, grey circle)
├── Model_Training.ipynb # Model training pipeline
├── generate_saliency_maps.ipynb # Ground truth saliency map generation (pysaliency)
├── Vorhersage_Visualisierung.ipynb # Visual prediction output generation
├── plotting_final.ipynb # Result plots used in the thesis
├── modded_deepgaze_files/ # Modified DeepGazeIIE architecture files
│ ├── deepgaze2e.py # Adapted model (accepts modification flag)
│ ├── modules.py # Supporting modules
│ └── __init__.py
├── utils/
│ └── utils.py # Shared utility functions
└── data/ # Data directory structure with placeholder READMEs
├── datasets/ # Original datasets (MIT1003, SALICON)
├── modImages_1aus11/ # Modified images (1 per 11 original)
└── npy/ # Pre-computed saliency maps (.npy format)
The
data/directory contains only the folder structure and README placeholder files - no data is included in this repository. Each subdirectory contains aReadMe.txtdescribing which files belong there and how they should be named.
| Category | Libraries / Tools |
|---|---|
| Deep Learning | PyTorch, torchvision, CUDA |
| Saliency Model | DeepGazeIIE (modified) |
| Ground Truth Generation | pysaliency |
| Image Processing | OpenCV, NumPy |
| Visualisation | Matplotlib |
| MATLAB (legacy) | Required for parts of the saliency toolbox pipeline |
Python dependencies – install via pip:
pip install torch torchvision opencv-python numpy matplotlib pysaliencyMATLAB – required for the saliency toolbox integration:
- Install MATLAB and add the SaliencyToolbox to your MATLAB path
- Install the MATLAB Engine for Python matching your MATLAB version
DeepGaze – install the base library, then replace the relevant files with the modified versions:
pip install deepgaze-pytorch
# Then copy modded_deepgaze_files/*.py into your deepgaze-pytorch installationFollow the ReadMe.txt files inside each data/ subdirectory. They specify which files to place where and the required naming conventions. Remove the ReadMe.txt files once the correct data is in place.
Run the notebooks in this order:
Bildmanipulation.ipynb- generate modified images from originalsgenerate_saliency_maps.ipynb- compute ground truth saliency maps for modified imagesModel_Training.ipynb- train the adaptive modelVorhersage_Visualisierung.ipynb- generate visual prediction outputsplotting_final.ipynb- reproduce result plots
- DeepGazeIIE: Linardos et al., 2021
- pysaliency: Kümmerer et al.
- SaliencyToolbox: Walther & Koch, 2006
- Datasets: MIT1003, SALICON
This project is licensed under the MIT License.