Ambiguity poses persistent challenges in natural language understanding for large language models (LLMs). To better understand how lexical ambiguity can be resolved through the visual domain, we develop an interpretable Visual Word Sense Disambiguation (VWSD) framework. The model leverages CLIP to project ambiguous language and candidate images into a shared multimodal space. We enrich textual embeddings using a dual-channel ensemble of semantic and photo-based prompts with WordNet synonyms, while image embeddings are refined through robust test-time augmentations. We then use cosine similarity to determine the image that best aligns with the ambiguous text. When evaluated on the SemEval-2023 VWSD dataset, enriching the embeddings raises the MRR from 0.7227 to 0.7590 and the Hit Rate from 0.5810 to 0.6220. Ablation studies reveal that dual-channel prompting provides strong, low-latency performance, whereas aggressive image augmentation yields only marginal gains. Additional experiments with WordNet definitions and multilingual prompt ensembles further suggest that noisy external signals tend to dilute semantic specificity, reinforcing the effectiveness of precise, CLIP-aligned prompts for visual word sense disambiguation.
Our paper is a preprint, in the process of submission fo a journal/workshop. It can be found at https://arxiv.org/abs/2602.06799
On a window's system, you can run this code by running python final.py. final.py is the file with the most recent model. All the other files are various scripts we used to debug and test earlier versions of the model.
The versions of Python used to run include Python 3.13.3 and 3.9.6.
The requirements.txt file includes the versions of the packages we used.
The dataset can be found at https://raganato.github.io/vwsd/. It needs to be in the main folder, with subfolders test_v1, train_v1, and trial_v1. Each of those folders should have a folder of images, a text file that provides the ambiguous sentences with candidate images, and another text file that lists the ground truth images.