Attention-Focused Approach for Improved Image Perception

This is the official implementation of 'Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory'.

🤖 Overview

Multimodal large language models (MLLMs) frequently suffer from object hallucinations, yet the visual perceptual mechanism underlying this failure remains poorly understood. In this work, we reveal that hallucinations are strongly associated with a human-like attention distraction phenomenon, where humans under divided focus experience degraded visual clarity and produce inaccurate descriptions, while in models the same mechanism manifests as spatial inconsistency in multi-head attention and temporal fading of attention to image tokens during decoding. We further provide theoretical insights that attention dispersion increases model complexity and degrades classification generalization. Motivated by these findings, we propose an Attention-Focused Approach for Improved Image Perception (AFIP), which corrects attention distraction via cross-head attention enrichment and reinforces visual grounding through dynamic historical attention enhancement. Extensive experiments on multiple benchmarks and models validate the effectiveness of AFIP without additional training.

📝 Project Layout

This repository is a cleaned release of our hallucination-indication codebase. The current open-source package focuses on the core experimental path only:

• Supported models: llava-1.5, qwen-vl
• Supported benchmarks: CHAIR, POPE
• Supported method component: forward attention modification (head guide)

Visualization utilities and other model backends from the internal research directory are intentionally excluded from this release branch.

AFIP/
├── qwen_vl_compat/          # Minimal local Qwen-VL compatibility package
├── scripts/
│   ├── run_chair_infer.py
│   ├── run_chair_eval.py
│   ├── run_pope_infer.py
│   ├── eval_pope.py
│   ├── visualize_var_comparison.py
│   └── plot_mvar_trend.py
├── src/ih_open/
│   ├── chair.py
│   ├── constants.py
│   ├── model_manager.py
│   ├── modify_attention.py
│   └── utils.py
├── data/
│   └── chair_questions.example.jsonl
└── requirements.txt

📑 Main Results

• CHAIR hallucination evaluation on five models with max new token set to 512.

🌍 Environment Setup

We recommend Python 3.10 and CUDA-enabled PyTorch.

conda create -n ih-open python=3.10 -y
conda activate ih-open
pip install -r requirements.txt

For llava-1.5, the official LLaVA package must also be available in the environment because we keep the original loading logic:

pip install -e /path/to/LLaVA

If you already maintain a local LLaVA checkout, you can reuse that environment directly.

👾 Model Preparation

You need local checkpoints for the two supported models.

• llava-1.5: a Hugging Face style checkpoint compatible with llava.model.builder.load_pretrained_model
• qwen-vl: the original Qwen-VL checkpoint compatible with trust_remote_code=True

All experiment scripts take --model-path, so no machine-specific hardcoded paths are required.

📊 Data Preparation

CHAIR

You need:

• COCO images, for example val2014/
• COCO annotation directory containing:
  • instances_train2014.json
  • instances_val2014.json
  • captions_train2014.json
  • captions_val2014.json
• A question file in JSONL format like:

{"image_id": 391895, "instruction": "Please describe this image in detail."}

An example file is provided at data/chair_questions.example.jsonl.

POPE

We use a POPE parquet file containing question text and image content. The current script supports the following image storage formats in the parquet rows:

• raw image bytes
• Hugging Face style {"bytes": ...}
• PIL image objects
• image path strings

Ground-truth labels should be stored in JSONL format with fields:

{"question_id": 1, "label": "yes"}

🦿 Running CHAIR

Caption generation

python scripts/run_chair_infer.py \
  --model llava-1.5 \
  --model-path /path/to/llava-1.5-7b-hf \
  --image-dir /path/to/coco/val2014 \
  --question-file data/chair_questions.example.jsonl \
  --output-file outputs/chair/llava_baseline.jsonl \
  --max-tokens 128

With our forward modification enabled:

python scripts/run_chair_infer.py \
  --model llava-1.5 \
  --model-path /path/to/llava-1.5-7b-hf \
  --image-dir /path/to/coco/val2014 \
  --question-file /path/to/chair_questions.jsonl \
  --output-file outputs/chair/llava_guided.jsonl \
  --use-head-guide \
  --alpha 0.1 \
  --threshold 5.0 \
  --heads-num 0.5 \
  --guide-range 0,31

The same command works for qwen-vl by replacing --model and --model-path.

CHAIR evaluation

python scripts/run_chair_eval.py \
  --preds-jsonl outputs/chair/llava_guided.jsonl \
  --coco-annotations /path/to/coco/annotations \
  --cache outputs/chair/chair.pkl \
  --out-json outputs/chair/llava_guided_metrics.json \
  --out-labels-jsonl outputs/chair/llava_guided_labels.jsonl

🦿 Running POPE

Inference

python scripts/run_pope_infer.py \
  --model qwen-vl \
  --model-path /path/to/Qwen-VL \
  --parquet-file /path/to/pope/adversarial-00000-of-00001.parquet \
  --output-file outputs/pope/qwen_answers.jsonl \
  --beams 1

With head guidance:

python scripts/run_pope_infer.py \
  --model qwen-vl \
  --model-path /path/to/Qwen-VL \
  --parquet-file /path/to/pope/adversarial-00000-of-00001.parquet \
  --output-file outputs/pope/qwen_guided_answers.jsonl \
  --use-head-guide \
  --alpha 0.1 \
  --threshold 5.0 \
  --heads-num 0.5 \
  --guide-range 0,31

Evaluation

python scripts/eval_pope.py \
  --gt-files /path/to/pope_ground_truth.jsonl \
  --gen-files outputs/pope/qwen_guided_answers.jsonl

👐 Method Scope

The main method implementation is in modify_attention.py. This release keeps only:

• llama_new_forward and llama_head_guide for llava-1.5
• qwen_new_forward and qwen_head_guide for qwen-vl

Other model-specific branches from the original research workspace were removed to make the project easier to read and maintain.

👓 Visualization

This release now includes the core VAR visualization code used in our analysis.

Native vs Modified VAR Comparison

The script visualize_var_comparison.py reproduces the style of the var_comparison_mean-13b.png figure. It compares:

• native attention forward
• modified attention forward after llama_head_guide

This script is currently intended for LLaVA-style checkpoints. In practice, it can be used for LLaVA-1.5 7B or 13B checkpoints as long as they follow the same loading path.

Example:

python scripts/visualize_var_comparison.py \
  --model-path /path/to/llava-1.5-13b-hf \
  --image-dir /path/to/coco/val2014 \
  --instruction-path /path/to/chair_questions.jsonl \
  --max-tokens 100 \
  --middle-layer-start 10 \
  --middle-layer-end 30 \
  --alpha 0.5 \
  --threshold 5.0 \
  --heads-num 0.5 \
  --smooth-window 9 \
  --output-fig outputs/vis/var_comparison_mean-13b.png

Required inputs:

• --model-path: local LLaVA checkpoint
• --image-dir: COCO image directory
• --instruction-path: JSON or JSONL prompt file with image_id and instruction

Expected output:

• a comparison figure where the orange curve is the original forward and the blue curve is the modified forward

Multi-model VAR Trend Plot

The script plot_mvar_trend.py plots multiple VAR curves from saved .pt files. This is the cleaned version of the trend plotting utility from the original research directory.

Example:

python scripts/plot_mvar_trend.py \
  --input /path/to/token_position_mvar.pt::LLaVA-1.5-7B::#4C72B0 \
  --input /path/to/token_position_mvar_13b.pt::LLaVA-1.5-13B::#55A868 \
  --input /path/to/qwen_token_mvar.pt::Qwen-VL::#C44E52 \
  --output outputs/vis/mvar_trend.pdf \
  --max-length 150

Each .pt file should store a list of per-sample records containing at least:

{
    "image_id": ...,
    "mvar_per_token": [...],
}

If you already have extracted token-level VAR files from your internal pipeline, you can directly reuse them here.

📖 Notes

• This release is organized to preserve the original main logic as much as possible while removing unrelated branches.
• Paths are now passed through CLI arguments instead of hardcoded local machine values.
• The current visualization release focuses on VAR trend figures and native-vs-modified comparison figures.
• Attention-map case-study visualization is still not included in this cleaned version.

Bibtex

@inproceedings{licorrecting,
  title={Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory},
  author={Li, Quanjiang and Liu, Zhiming and Luo, Wei and Luo, Tingjin and Hou, Chenping},
  booktitle={Forty-third International Conference on Machine Learning}
}