Image to Prompts using OFA, BLIP and GIT with VizWiz Dataset

This project provides a comprehensive pipeline for generating and evaluating image captions using state-of-the-art models. The pipeline leverages the VizWiz dataset and incorporates models like OFA, BLIP, and GIT to generate captions, evaluate their quality, and perform comparative analysis.

Project Overview

This project aims to:

• Load and preprocess the VizWiz dataset
• Initialize and optionally train the OFA, BLIP, and GIT models
• Generate captions for images using all three models
• Evaluate the generated captions against ground truth using various metrics and CLIP similarity
• Perform comparative analysis to visualize and interpret the performance of the models

By following this pipeline, users can assess the quality of captions generated by different models and gain insights into their strengths and weaknesses, particularly for images taken by people who are blind.

Dataset

VizWiz

The VizWiz-Captions dataset consists of 39,181 images originating from people who are blind that are each paired with 5 captions. This dataset represents a real use case where blind people rely on image captioning services to learn about images they take.

Dataset Features:

• Images: Diverse set of images taken by people who are blind
• Captions: Five human-annotated captions per image
• Text Detection: Flag indicating whether text is detected in the image

Dataset Statistics:

• 23,431 training images (117,155 captions)
• 7,750 validation images (38,750 captions)
• 8,000 test images (40,000 captions)

Installation

Before running the scripts, ensure that you have the necessary dependencies installed.

Clone the Repository:

git clone <repository-url>
cd image-captioning-evaluation

Install Dependencies:

# Create and activate conda environments (recommended to use separate environments)
conda create -n ofa_env python=3.8
conda activate ofa_env
pip install -r ofa_env_requirements.txt

conda create -n blip_env python=3.8
conda activate blip_env
pip install -r blip_env_requirements.txt

# Download NLTK data
python -c "import nltk; nltk.download('punkt'); nltk.download('wordnet'); nltk.download('averaged_perceptron_tagger')"

Project Structure

image-captioning-evaluation/
├── data/
│   └── viswiz_images/
├── models/
│   ├── ofa_trained/
│   ├── blip_trained/
│   └── git_trained/
├── results/
│   ├── processed_dataset.csv
│   ├── processed_dataset_train.csv
│   ├── processed_dataset_val.csv
│   ├── prompts_ofa.csv
│   ├── prompts_blip.csv
│   ├── prompts_git.csv
│   ├── evaluation.csv
│   └── analysis_plots/
├── scripts/
│   ├── load_dataset.py
│   ├── initialize_model.py
│   ├── initialize_model_blip.py
│   ├── initialize_model_git.py
│   ├── generate_prompts_ofa.py
│   ├── generate_prompts_blip.py
│   ├── generate_prompts_git.py
│   ├── evaluate_prompts.py
│   ├── train_ofa.py
│   ├── train_blip.py
│   ├── train_git.py
│   └── visualization.py
├── README.md
├── ofa_env_requirements.txt
└── blip_env_requirements.txt

Scripts Documentation

1. load_dataset.py

Description: Loads and preprocesses the VizWiz dataset. It verifies image integrity, saves images to a specified directory, and generates a CSV file containing image paths and their corresponding captions.

Key Features:

• Loading Dataset: Loads the VizWiz dataset from local files
• Image Verification: Checks if each image is valid
• Preprocessing: Filters out samples with invalid images
• Split Creation: Creates separate CSV files for train/validation/test splits
• Output: Saves the processed dataset to processed_dataset.csv

Usage:

python scripts/load_dataset.py --data_dir /path/to/your/data/viswiz --output_csv ./results/processed_dataset.csv --create_splits

2. initialize_model.py

Description: Initializes the OFA and CLIP models. OFA is used to generate captions, while CLIP is utilized to evaluate the similarity between captions and images.

Key Features:

• OFA Initialization: Loads the OFA tokenizer and model
• CLIP Initialization: Loads the CLIP model and processor
• Device Configuration: Automatically detects and utilizes available GPU resources
• Checkpoint Saving: Option to save model checkpoints

Usage:

python scripts/initialize_model.py --save_checkpoint

3. initialize_model_blip.py

Description: Initializes the BLIP model, which is another state-of-the-art model for image captioning.

Key Features:

• BLIP Initialization: Loads the BLIP processor and model
• Device Configuration: Automatically detects and utilizes available GPU resources
• Checkpoint Saving: Option to save model checkpoints

Usage:

python scripts/initialize_model_blip.py --save_checkpoint

4. initialize_model_git.py

Description: Initializes the GIT (Generative Image-to-text Transformer) model, a powerful model for image captioning.

Key Features:

• GIT Initialization: Loads the GIT processor and model
• Device Configuration: Automatically detects and utilizes available GPU resources
• Checkpoint Saving: Option to save model checkpoints

Usage:

python scripts/initialize_model_git.py --save_checkpoint

5. generate_prompts_ofa.py

Description: Generates captions for images using the OFA model and appends them to the dataset.

Key Features:

• Caption Generation: Processes each image to generate a caption using OFA
• Image Preprocessing: Prepares images for the OFA model
• Batch Processing: Efficiently processes large datasets
• Logging: Logs a specified number of samples for inspection
• Output: Saves the captions to prompts_ofa.csv

Usage:

python scripts/generate_prompts_ofa.py --model_path ./models/ofa_trained/final_model

6. generate_prompts_blip.py

Description: Generates captions for images using the BLIP model and appends them to the dataset.

Key Features:

• Caption Generation: Processes each image to generate a caption using BLIP
• Image Preprocessing: Prepares images for the BLIP model
• Batch Processing: Efficiently processes large datasets
• Logging: Logs a specified number of samples for inspection
• Output: Saves the captions to prompts_blip.csv

Usage:

python scripts/generate_prompts_blip.py --model_name ./models/blip_trained/final_model

7. generate_prompts_git.py

Description: Generates captions for images using the GIT model and appends them to the dataset.

Key Features:

• Caption Generation: Processes each image to generate a caption using GIT
• Image Preprocessing: Prepares images for the GIT model
• Batch Processing: Efficiently processes large datasets
• Logging: Logs a specified number of samples for inspection
• Output: Saves the captions to prompts_git.csv

Usage:

python scripts/generate_prompts_git.py --model_name ./models/git_trained/final_model

8. evaluate_prompts.py

Description: Evaluates the generated captions against the ground truth captions using multiple metrics, including BLEU, METEOR, ROUGE-L, and CLIP similarity.

Key Features:

• Multi-Reference Evaluation: Handles multiple ground truth captions per image
• Metric Calculations: Computes various evaluation metrics to assess caption quality:
  • BLEU-1, BLEU-2, BLEU-3, BLEU-4
  • METEOR
  • ROUGE-L
  • CLIP similarity
• Text Preprocessing: Implements tokenization and lemmatization for accurate evaluation
• Multi-Model Evaluation: Evaluates captions from OFA, BLIP, and GIT models
• Batch Processing: Efficiently processes large datasets
• Output: Saves evaluation results to CSV files and provides a comparison table

Usage:

python scripts/evaluate_prompts.py \
    --model1_csv ./results/prompts_blip.csv \
    --model2_csv ./results/prompts_git.csv \
    --model3_csv ./results/prompts_ofa.csv \
    --model1_name blip \
    --model2_name git \
    --model3_name ofa \
    --model1_caption_col blip_caption \
    --model2_caption_col git_caption \
    --model3_caption_col ofa_caption \
    --output_dir ./results/evaluations/

9. train_ofa.py

Description: Trains the OFA model on the VizWiz dataset for image captioning.

Key Features:

• Custom Dataset: Creates a specialized dataset class for VizWiz image captioning with OFA
• Training Loop: Implements a complete training loop with validation
• Optimizer Configuration: Sets up optimizer and learning rate scheduler
• Gradient Clipping: Implements gradient clipping for stable training
• Checkpointing: Saves model checkpoints during training
• Evaluation: Evaluates model performance on validation data
• Logging: Tracks and reports training progress and metrics

Usage:

python scripts/train_ofa.py --train_csv ./results/processed_dataset_train.csv --val_csv ./results/processed_dataset_val.csv --output_dir ./models/ofa_trained --num_epochs 3

10. train_blip.py

Description: Trains the BLIP model on the VizWiz dataset for image captioning.

Key Features:

• Custom Dataset: Creates a specialized dataset class for VizWiz image captioning with BLIP
• Training Loop: Implements a complete training loop with validation
• Optimizer Configuration: Sets up optimizer and learning rate scheduler
• Gradient Clipping: Implements gradient clipping for stable training
• Checkpointing: Saves model checkpoints during training
• Evaluation: Evaluates model performance on validation data
• Logging: Tracks and reports training progress and metrics

Usage:

python scripts/train_blip.py --train_csv ./results/processed_dataset_train.csv --val_csv ./results/processed_dataset_val.csv --output_dir ./models/blip_trained --num_epochs 3

11. train_git.py

Description: Trains the GIT model on the VizWiz dataset for image captioning.

Key Features:

• Custom Dataset: Creates a specialized dataset class for VizWiz image captioning with GIT
• Training Loop: Implements a complete training loop with validation
• Optimizer Configuration: Sets up optimizer and learning rate scheduler
• Gradient Clipping: Implements gradient clipping for stable training
• Checkpointing: Saves model checkpoints during training
• Evaluation: Evaluates model performance on validation data
• Logging: Tracks and reports training progress and metrics

Usage:

python scripts/train_git.py --train_csv ./results/processed_dataset_train.csv --val_csv ./results/processed_dataset_val.csv --output_dir ./models/git_trained --num_epochs 3

12. visualization.py

Description: Creates visualizations for the evaluation results to compare model performance.

Key Features:

• Bar Charts: Creates bar charts comparing average metrics across models
• Individual Metric Charts: Generates individual charts for each metric
• Radar Chart: Creates a radar chart comparing models across all metrics
• Violin Plots: Shows the distribution of each evaluation metric for all models
• Scatter Plots: Visualizes relationships between different metrics
• Dashboard: Creates a comprehensive dashboard with multiple visualizations
• Enhanced Styling: Customizes plots with improved visual design
• Output: Saves plots to the analysis_plots directory

Usage:

python scripts/visualization.py \
    --comparison_csv ./results/evaluations/model_comparison.csv \
    --model1_csv ./results/evaluations/blip_eval.csv \
    --model2_csv ./results/evaluations/git_eval.csv \
    --model3_csv ./results/evaluations/ofa_eval.csv \
    --output_dir ./results/analysis_plots/

Pipeline Workflow

To execute the entire pipeline, follow these steps sequentially:

1. Load and Preprocess the Dataset:

python scripts/load_dataset.py --data_dir /path/to/your/data/viswiz --output_csv ./results/processed_dataset.csv --create_splits

This will generate processed_dataset.csv and split files in the results directory.

2. Initialize Models:

OFA and CLIP:

python scripts/initialize_model.py --save_checkpoint

BLIP:

python scripts/initialize_model_blip.py --save_checkpoint

GIT:

python scripts/initialize_model_git.py --save_checkpoint

3. Train Models:

If you want to train the models from scratch on the VizWiz dataset:

Train OFA model:

python scripts/train_ofa.py --train_csv ./results/processed_dataset_train.csv --val_csv ./results/processed_dataset_val.csv --output_dir ./models/ofa_trained --num_epochs 3

Train BLIP model:

python scripts/train_blip.py --train_csv ./results/processed_dataset_train.csv --val_csv ./results/processed_dataset_val.csv --output_dir ./models/blip_trained --num_epochs 3

Train GIT model:

python scripts/train_git.py --train_csv ./results/processed_dataset_train.csv --val_csv ./results/processed_dataset_val.csv --output_dir ./models/git_trained --num_epochs 3

4. Generate Captions:

Using OFA:

python scripts/generate_prompts_ofa.py --model_path ./models/ofa_trained/final_model

Using BLIP:

python scripts/generate_prompts_blip.py --model_name ./models/blip_trained/final_model

Using GIT:

python scripts/generate_prompts_git.py --model_name ./models/git_trained/final_model

5. Evaluate Generated Captions:

python scripts/evaluate_prompts.py \
    --model1_csv ./results/prompts_blip.csv \
    --model2_csv ./results/prompts_git.csv \
    --model3_csv ./results/prompts_ofa.csv \
    --model1_name blip \
    --model2_name git \
    --model3_name ofa \
    --model1_caption_col blip_caption \
    --model2_caption_col git_caption \
    --model3_caption_col ofa_caption \
    --output_dir ./results/evaluations/

This will produce evaluation results for all three models.

6. Perform Comparative Analysis:

python scripts/visualization.py \
    --comparison_csv ./results/evaluations/model_comparison.csv \
    --model1_csv ./results/evaluations/blip_eval.csv \
    --model2_csv ./results/evaluations/git_eval.csv \
    --model3_csv ./results/evaluations/ofa_eval.csv \
    --output_dir ./results/analysis_plots/

This will generate various plots in the results/analysis_plots directory.

Results

After executing the pipeline, the following results will be available:

• Processed Dataset: results/processed_dataset.csv
• Generated Captions:
  • OFA: results/prompts_ofa.csv
  • BLIP: results/prompts_blip.csv
  • GIT: results/prompts_git.csv
• Evaluation Metrics:
  • Individual model evaluations: results/evaluations/[model]_eval.csv
  • Model comparison: results/evaluations/model_comparison.csv
• Analysis Plots: Stored in results/analysis_plots/
  • Bar charts comparing average metrics
  • Individual metric charts
  • Radar chart comparing models
  • Violin plots showing metric distributions
  • Scatter plots showing metric relationships
  • Comprehensive dashboard

These results provide a comprehensive understanding of how the OFA, BLIP, and GIT models perform in generating image captions for the VizWiz dataset and how they compare across various evaluation metrics.

Model Information

OFA (One For All)

OFA is a unified multimodal pre-trained model that unifies modalities (vision, language, etc.) with a single model. It's trained on multiple tasks including image captioning, making it a powerful model for generating descriptive captions.

BLIP (Bootstrapping Language-Image Pre-training)

BLIP is a vision-language pre-training framework that effectively utilizes the noisy web data by bootstrapping the captions. It achieves state-of-the-art results on various vision-language tasks including image captioning.

GIT (Generative Image-to-text Transformer)

GIT is a generative image-to-text transformer model that excels at generating descriptive text from images. It's designed to handle a wide range of vision-language tasks and is particularly effective for image captioning.

Extending the Project

This project can be extended in several ways:

1. Add more captioning models (e.g., KOSMOS-2, SimVLM)
2. Implement additional evaluation metrics
3. Add fine-tuning capabilities for specific domains
4. Create a web interface for interactive caption generation and evaluation
5. Integrate with other datasets for comparative analysis
6. Implement ensemble methods to combine predictions from multiple models

Acknowledgments

• VizWiz dataset creators
• OFA, BLIP, and GIT model developers
• HuggingFace for providing model implementations