A learned cross-modal consistency gate that selectively masks low-agreement patches during CLIP fine-tuning, improving zero-shot retrieval without any additional labeled data.
Standard CLIP fine-tuning treats all image patches equally when computing the image-text similarity loss. Some patches, however, actively disagree with the paired text — background clutter, occluded regions, or semantically irrelevant areas. This project introduces a three-way consistency gate that identifies and downweights such patches during training, acting as an adversarial masking signal that forces the model to rely on semantically aligned regions.
(Gate-off vs gate-on cross-modal attention heatmaps — 6-sample preview. Full 50-image grid in figures/attn_maps_grid.png after running eval/attn_viz.py.)
Zero-shot Recall@K on MSCOCO val2017, image→text (i2t) and text→image (t2i):
| Model | i2t R@1 | i2t R@5 | i2t R@10 | t2i R@1 | t2i R@5 | t2i R@10 |
|---|---|---|---|---|---|---|
| OpenCLIP ViT-B/16 (baseline) | 66.02 | 88.68 | 93.30 | 48.70 | 75.63 | 84.57 |
| + Gate OFF (arch. present, inactive) | 66.52 | 87.96 | 93.44 | 48.72 | 75.68 | 84.55 |
| + Gate ON, λ=0.1 | 54.36 | 80.86 | 88.54 | 43.80 | 71.49 | 81.39 |
| + Gate ON, λ=0.5 | 54.44 | 80.34 | 88.08 | 44.30 | 72.03 | 81.64 |
| + Gate + Masking 10% | 54.60 | 81.02 | 88.56 | 45.91 | 73.07 | 82.79 |
| + Gate + Masking 20% | 46.60 | 73.54 | 83.88 | 37.36 | 65.14 | 75.75 |
| + Gate + Masking 30% | 43.46 | 71.56 | 81.48 | 36.91 | 64.44 | 74.99 |
| + Gate + Masking 40% | 45.54 | 73.78 | 83.04 | 37.58 | 64.99 | 75.67 |
(Raw JSON, full run logs, and plotting code: notebooks/results_summary.ipynb)
Across all 8 runs, enabling the consistency gate and/or patch masking consistently reduces retrieval performance relative to baseline, with the drop growing roughly monotonically with masking ratio (i2t R@1: 66.0% baseline → 54.4% gate-on → 43.5% at 30% masking). Gate OFF (architecture present but inactive) tracks the baseline almost exactly (66.5% vs 66.0% i2t R@1), which isolates the regression to the masking/gating mechanism itself rather than other architecture changes. In this experimental setup, the results do not support the original hypothesis that adversarial patch masking improves zero-shot retrieval.
bash setup.sh # install deps, download data
bash scripts/run_all_studies.sh # run all 4 studies sequentially
jupyter nbconvert --to notebook --execute notebooks/results_summary.ipynb| Study | Description | Config | Result |
|---|---|---|---|
| 1 | OpenCLIP baseline fine-tune | configs/study1_baseline.yaml |
results/baseline.json |
| 2a | Gate off | configs/study2_gate_off.yaml |
results/gate_off.json |
| 2b | Gate on, λ=0.1 | configs/study2_gate_l01.yaml |
results/gate_on_l01.json |
| 2c | Gate on, λ=0.5 | configs/study2_gate_l05.yaml |
results/gate_on_l05.json |
| 3a | Masking ratio 10% | configs/study3_mask10.yaml |
results/mask_sweep_10.json |
| 3b | Masking ratio 20% | configs/study3_mask20.yaml |
results/mask_sweep_20.json |
| 3c | Masking ratio 30% | configs/study3_mask30.yaml |
results/mask_sweep_30.json |
| 3d | Masking ratio 40% | configs/study3_mask40.yaml |
results/mask_sweep_40.json |
| 4 | Attention visualization | — | figures/attn_maps_grid.png |
The bottleneck in VL alignment is not the transformer representations — it is the aggregation. A ViT already knows which patches are a dog and which are sky (that is what self-attention does). Standard mean-pooling discards that knowledge by treating all 196 patch embeddings equally. This project makes the aggregation semantically aware.
The full image and full caption are always fed to their respective transformers unchanged — no pixel dropout, no token replacement. Masking happens after the transformer, on the 512-d feature vectors, before they are pooled into the final VL embedding:
196 patch embeddings (already in 512-d VL space)
→ gate scores each → bottom 20% zeroed → mean pool → image_emb
77 text token embeddings (already in 512-d VL space)
→ importance scores each → bottom 30% content tokens zeroed
→ importance-weighted pool → text_emb
Both embeddings entering the contrastive loss are therefore cleaner, more semantically concentrated representations of their inputs.
For each image-text pair the gate computes:
- Visual self-consistency — patch attention entropy over other patches. Low entropy = patch attends sharply = coherent region (keep). High entropy = background clutter (drop).
- Image→Text attention — patch attention entropy over text tokens. Low entropy = patch is grounded in specific words (keep). High entropy = patch not described by caption (drop).
- Text self-consistency — per-token attention entropy over other tokens. Low entropy = semantically important token (noun, verb). High entropy = filler word ("a", "the").
- Importance-weighted Text→Image — text tokens attend back to patches, weighted by signal 3. A patch attended by "dog" scores higher than one attended by "the".
A small MLP residual is added on top of the heuristic consistency score so the gate is end-to-end trainable.
L_total = L_contrastive + λ_gate * L_gate_reg
- L_contrastive: symmetric InfoNCE (identical to original CLIP). Pulls matched image-text pairs together, pushes mismatched pairs apart.
- L_gate_reg: entropy penalty on gate score distribution. Prevents gate scores from collapsing to uniform values (which would make patch selection random). Forces the gate to make confident, polarised keep/drop decisions.
Unlike distillation-based approaches, there is no frozen teacher network running in parallel. The gate is a single lightweight MLP (~0.5M parameters) that runs once per forward pass during training and is disabled entirely at inference.
pip install open_clip_torch flash-attn transformers datasets pillow tqdm# CC3M subset (pre-fetch before H100 session)
img2dataset --url_list data/cc3m_100k.tsv \
--output_folder data/cc3m \
--image_size 224 \
--output_format webdataset
# Flickr30k (auto-downloaded via HuggingFace)
# MSCOCO val2017
wget http://images.cocodataset.org/zips/val2017.zip -P data/
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip -P data/attn-adversarial-vlm/
├── data/ # datasets (gitignored)
├── configs/ # YAML config per study run
├── src/
│ ├── model.py # GatedCLIP model
│ ├── gate.py # cross-modal consistency gate
│ ├── losses.py # contrastive + gate regularization losses
│ ├── dataset.py # CC3M webdataset loader
│ └── train.py # training loop
├── eval/
│ ├── retrieval.py # Flickr30k + MSCOCO zero-shot retrieval
│ └── attn_viz.py # attention map visualization
├── notebooks/
│ └── results_summary.ipynb
├── scripts/
│ ├── fetch_data.sh # download all datasets
│ └── run_all_studies.sh # sequential study runner
├── results/ # JSON results (gitignored, tracked via DVC or manually)
├── figures/ # generated plots and visualizations
└── setup.sh
@misc{attn-adversarial-vlm-2026,
title = {Attention-Adversarial Masking for Vision-Language Alignment},
author = {Suyash Kumar Bhagat},
year = {2026},
url = {https://github.com/Suyashkb/AttentionMask-VLM}
}