Medical Visual Question Answering

Overview

This project implements a comprehensive system for Medical Visual Question Answering (VQA) using the SLAKE dataset from Hugging Face. The system addresses the challenge of automatically answering natural language questions about medical images, combining computer vision and natural language processing techniques.

Medical VQA is an important task that involves understanding both visual content (medical images) and textual queries to generate accurate answers. This work explores multiple deep learning architectures to achieve robust performance on this challenging multimodal task.

Dataset

The project utilizes the SLAKE dataset, a large scale knowledge enhanced medical visual question answering dataset. SLAKE contains:

• Medical images with corresponding clinical questions and answers
• Diverse medical imaging modalities (X rays, CT scans, etc.)
• Multilingual support with approximately 44% Chinese and 56% English content
• Comprehensive annotations for vision language understanding

The dataset is loaded directly from Hugging Face, making it easy to reproduce and scale the experiments.

Project Structure

The project is organized into the following components:

Core Utilities

• Utils.py: Contains the main dataset loading utilities, the SLAKE Dataset class, and the base architecture (ResNet18 + Multilingual BERT). Implements vocabulary preprocessing, dataloader creation, and evaluation metrics.

• Utils_BLIP.py: Implements the BLIP (Bootstrapping Language Image Pre training) architecture for VQA. BLIP uses Vision Transformer (ViT) for image encoding and BERT for text encoding, specifically fine tuned for the visual question answering task.

• Utils_FLAVA.py: Implements the FLAVA (Foundational Language And Vision Alignment) architecture, a unified multimodal model that jointly processes images and text for improved semantic understanding.

Experiments

• Architecture1.ipynb: Baseline model combining ResNet18 for vision feature extraction and Multilingual BERT for text encoding. This architecture serves as the foundational approach for the medical VQA task.

• Architecture2.ipynb: BLIP based architecture leveraging pre trained vision language models specifically designed for question answering tasks. Implements parameter efficient fine tuning strategies.

• Architecture3.ipynb: FLAVA based architecture utilizing a unified multimodal encoder for improved fusion of visual and textual information.

Architectures

Architecture 1: ResNet18 + mBERT

This baseline architecture combines traditional computer vision and NLP approaches:

• Vision Encoder: ResNet18 pre trained on ImageNet for efficient image feature extraction
• Text Encoder: Multilingual BERT (mBERT) for supporting multiple languages present in the dataset
• Fusion Strategy: Feature concatenation followed by multi layer perceptron (MLP) based classification head
• Fine tuning: Selective unfreezing of final layers while keeping early layers frozen
• Multilingual Support: mBERT is specifically chosen due to the dataset containing approximately 44% Chinese answers alongside English, enabling the model to understand questions and answers in both languages

This approach provides a strong baseline with multilingual capabilities for medical VQA.

Architecture 2: BLIP for Visual Question Answering

BLIP is a specialized vision language model designed for VQA:

• Vision Encoder: Vision Transformer (ViT) from BLIP for high quality image representations
• Text Encoder: BERT based text encoder with cross modal attention
• Pre training: Leverages BLIP's pre training on large scale vision language datasets
• Decoder: Specialized VQA head for answer prediction
• Parameter Efficiency: Fine tunes only the last 2 layers of the text encoder and prediction head

BLIP's design makes it particularly well suited for medical VQA tasks where accurate vision language understanding is critical.

Architecture 3: FLAVA for Multimodal Understanding

FLAVA provides unified multimodal processing:

• Architecture: Unified transformer encoder processing both image and text tokens
• Multimodal Fusion: Joint embedding space for vision and language modalities
• Pre training: Pre trained on diverse vision language datasets with multilingual support
• Language Support: Chosen specifically because FLAVA's language model is pretrained on multilingual data, which aligns well with the dataset composition of approximately 44% Chinese and 56% English answers
• Parameter Efficient Fine tuning: Unfreezes last 2 transformer layers
• Classification Head: Dedicated answer classifier on joint multimodal representations

FLAVA's unified approach and multilingual pretraining capabilities enable better cross modal reasoning and language handling compared to separate encoders.

Key Features

• Modular Design: Separate utility files for different architectures enable easy comparison and experimentation
• Efficient Fine tuning: Implements parameter efficient strategies to reduce computational overhead
• Comprehensive Evaluation: Tracks multiple metrics including accuracy, F1 score, balanced accuracy, and calibration error
• Early Stopping: Prevents overfitting with patience based early stopping mechanism
• Visualization: Includes detailed visualization of training history, evaluation metrics, and confusion matrices
• Reproducibility: All experiments use fixed random seeds and detailed configurations

Dependencies

The project requires the following main packages:

• PyTorch: Deep learning framework
• Transformers: Pre trained models from Hugging Face
• Torchmetrics: Metric computation and evaluation
• Scikit learn: Machine learning utilities for evaluation
• Hugging Face Datasets: SLAKE dataset loading
• PIL/Pillow: Image processing
• Matplotlib/Numpy: Visualization and numerical computing

Usage

Loading the Dataset

from datasets import load_dataset
from Utils import vocab_preprocessing, dataLoaders

# Load SLAKE dataset from Hugging Face
ds = load_dataset("vigsterkr/slake")

# Prepare vocabulary
all_answers, unique_answers, answer_to_idx, idx_to_answer, num_classes = vocab_preprocessing(ds)

# Create dataloaders
train_loader, val_loader, test_loader = dataLoaders(ds, IMG_DIR, answer_to_idx, tokenizer, BLIP=False, FLAVA=False)

Training Architecture 1

from Utils import VQA_Architecture_One, train

model = VQA_Architecture_One(embedding_dim=512, num_classes=num_classes, 
                              pretrained_vision=True, pretrained_text=True)
history = train(model, train_loader, val_loader, device="cuda", epochs=10, lr=1e-4)

Training Architecture 2 (BLIP)

from Utils_BLIP import VQA_Architecture_Two, train_BLIP

model = VQA_Architecture_Two(num_classes=num_classes, PRETRAINED_BLIP=True)
history = train_BLIP(model, train_loader, val_loader, device="cuda", epochs=10, lr=1e-4)

Training Architecture 3 (FLAVA)

from Utils_FLAVA import VQA_Architecture_Three, train

model = VQA_Architecture_Three(num_classes=num_classes, PRETRAINED=True)
history = train(model, train_loader, val_loader, device="cuda", epochs=10, lr=1e-4)

Experimental Results

The three architectures are compared on:

• Training Efficiency: Parameters, computational requirements, and training time
• Model Performance: Accuracy, F1 score, balanced accuracy on test set
• Generalization: Validation performance and overfitting analysis
• Robustness: Calibration error and confusion matrix analysis

Results from experimentation can be visualized using the provided notebook utilities.

Implementation Highlights

• Vocabulary Preprocessing: Automatic extraction and mapping of unique answers to class indices
• Dataset Handling: Robust handling of missing images and data corruption
• Multi format Processing: Support for different tokenizers (BLIP, FLAVA, Multilingual BERT)
• Flexible Architecture: Easy switching between different models and configurations
• Comprehensive Metrics: Multi aspect evaluation beyond simple accuracy

Model Evaluation

Each architecture includes:

• Training and validation loss tracking
• Multiple accuracy metrics (standard, balanced, F1 score)
• Confusion matrix visualization
• Correct vs incorrect prediction analysis
• Early stopping to prevent overfitting
• Learning rate scheduling for optimization

Performance Considerations

• VRAM Requirements: The choice of model depends on available GPU memory
• Training Time: Architecture complexity varies from hours to days
• Inference Speed: Trade offs between accuracy and speed
• Batch Size Effects: Different batch sizes may affect convergence and final performance

Author

Mehdi Benabi