Vision-Language Model-Based Semantic-Guided Imaging Biomarker for Lung Nodule Malignancy Prediction

Luoting Zhuang, Seyed Mohammad Hossein Tabatabaei, Ramin Salehi-Rad, Linh M. Tran, Denise R. Aberle, Ashley E. Prosper, William Hsu

¹Medical & Imaging Informatics, Department of Radiological Sciences, David Geffen School of Medicine at UCLA, Los Angeles, CA;

²Department of Medicine, Division of Pulmonology and Critical Care, David Geffen School of Medicine at UCLA, Los Angeles, CA

³VA Greater Los Angeles Healthcare System, Los Angeles, 90073, CA, USA

Figure 1. An overview of the proposed framework.

Objective: Machine learning models have utilized semantic features, deep features, or both to assess lung nodule malignancy. However, their reliance on manual annotation during inference, limited interpretability, and sensitivity to imaging variations hinder their application in real-world clinical settings. Thus, this research aims to integrate semantic features derived from radiologists’ assessments of nodules, guiding the model to learn clinically relevant, robust, and explainable imaging features for predicting lung cancer.

Methods: We obtained 938 low-dose CT scans from the National Lung Screening Trial (NLST) with 1,261 nodules and semantic features. Additionally, the Lung Image Database Consortium dataset contains 1,018 CT scans, with 2,625 lesions annotated for nodule characteristics. Three external datasets were obtained from UCLA Health, the LUNGx Challenge, and the Duke Lung Cancer Screening. For imaging input, we obtained 2D nodule slices from nine directions from 50×50×50mm nodule crop. We converted structured semantic features into sentences using Gemini. We fine-tuned a pretrained Contrastive Language-Image Pretraining (CLIP) model with a parameter-efficient fine-tuning approach to align imaging and semantic features and predict the one-year lung cancer diagnosis.

Results: We evaluated the performance of the one-year diagnosis of lung cancer. Our model demonstrated an AUROC of 0.901 and AUPRC of 0.776 in the NLST test set, also outperforming state-of-the-art(SOTA) models on external datasets. Using CLIP, we also obtained predictions on semantic features through zero-shot inference, such as nodule margin (AUROC: 0.807), nodule consistency (0.812), and pleural attachment (0.840).

Conclusion: Our approach surpasses the SOTA models in predicting lung cancer across various datasets from diverse clinical settings, providing explainable outputs, aiding clinicians in comprehending the underlying meaning of model predictions. This approach also prevents the model from learning shortcuts and generalizes across clinical settings.

🚀Getting Started

1. 🛠️Setup Environment

This project supports both GPU (CUDA) and CPU-only environments.

Option 1: Docker container

docker run --shm-size=8g --gpus all -it --rm -p 1234:1234 -v .:/workspace -v /etc/localtime:/etc/localtime:ro nvcr.io/nvidia/pytorch:24.03-py3

• If you use -v .:/workspace as shown above, Docker will map the current directory to /workspace inside the container.
• To map a different folder to a specific path in a docker container, you can replace -v .:/workspace with -v /path/to/local/folder:/path/in/container.
• -p 1234:1234 can be modified to match your preferred host and container ports as needed.

Option 2: Conda environment

conda create -n clip_nodule python=3.10 -y 
conda activate clip_nodule
#gpu
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
#cpu
conda install pytorch torchvision torchaudio cpuonly -c pytorch

2. 📦Clone the Repository and Install Packages

1. Go to the folder you want to store the code and clone the repo

git clone https://github.com/luotingzhuang/CLIP_nodule.git
cd CLIP_nodule

2. Install all of the required Python packages using the following command line.

pip install -r requirements.txt

3. 📁Data Requirement

To prepare a CSV file, list the path to the NIfTI file under the image_path column, along with the corresponding pid and nodule_id. coordX, coordY, and coordZ are the nodule centroid in a global coordinate system. These can be extracted from the nodule mask using the code. If the nodule mask is not available, we recommend using a nodule detection algorithm, such as monai nodule detection to obtain the nodule location from CT scans.

For inference, the CSV file should contain six columns:

pid	nodule_id	image_path	coordX	coordY	coordZ
121389	0	./sample_data/121389/2001-01-02/image.nii.gz	-38.038567	-73.942905	-111.030769

Refer to ./dataset_csv/sample_csv.csv as an example. Note that the malignant column is optional in inference.

Due to data sharing restrictions, we cannot release the in-house annotated semantic features for the NLST dataset. However, we provide synthetic data to illustrate usage. The CSV file is in ./dataset_csv/sample_csv_semantic_feats.csv.

CT scans in our sample data can be downloaded from the link.

# You can also download it using gdown
gdown --folder 'https://drive.google.com/drive/folders/1MhcOCLpG1OrdGyQw9OiwNQELZKfIBGlr?usp=drive_link'

⚠️Note: Try add --fuzzy --no-cookies --no-check-certificate if there is an error running the gdown command. Some institutional Wi-Fi may block Google services. If it still does not work, try downloading the folder using the link via a browser.

🧹 Preprocessing

1. 🖌️ Image Preprocessing

First, we need to crop a 100×100×100 mm bounding box around the nodule and save the resulting cropped volume as a .pt file for later use.

python crop_nodule.py --dataset_path ./dataset_csv/sample_csv.csv --save_path ./cropped_img --crop_size 100

Important Arguments

Argument	Type	Default	Description
--crop_size	int	100	Size of the crop for the nodules.

Note: The crop size should typically be set slightly larger than the target size to accommodate data augmentation during training. A final cropping step will be applied afterward.

The nodule crop will be saved with the format {pid}_{nodule_id}.pt.

2. 📄Semantic Features Preprocessing (Only for Training)

We need to convert semantic features, which were originally in tabular format, into texts. The code for transformation is shown in ./notebook/tabular2text.ipynb.

Download report_generation from the link and put it under ./CLIP_nodule.

# You can also download it using gdown
gdown --folder 'https://drive.google.com/drive/folders/1LO3t7r6xZ6WakMFTt17snKRggsNCvL21?usp=sharing'

Code for extracting LIDC semantic features and mapping them to UCLA's semantic features is available in ./notebook/lidc_preprocess.ipynb.

3. 🧩Create Cross-Validation Splits (Only for Training)

In our training, we used k-fold CV. The splits folder should have the following structure.

splits/
├── fold_0/
│   ├── train_pid.csv
│   └── val_pid.csv
├── ...

Each CSV file must contain a single column named pid, which lists all patient IDs included in that split. You can find example CSV files in the splits folder of the repository. We also provide sample code to generate the splits in the ./notebook/create_splits.ipynb.

🧠Model Training and Inference

1. ▶️Run Training

CUDA_VISIBLE_DEVICES=0 python train.py \
  --dataset_path ./dataset_csv/sample_csv_with_semantic_feats.csv \
  --result_dir ./results \
  --img_dir ./cropped_img \
  --text_dir ./report_generation \
  --split_dir ./splits \
  --n_splits 5

Important Arguments

Argument	Type	Default	Description
--crop_size	int	50	Size of the crop for the images.
--batch_size	int	4	Batch size for training.
--lr	float	1e-4	Learning rate for the optimizer.
--weight_decay	float	1e-1	Weight decay for the optimizer.
--epochs	int	100	Number of epochs for training.
--es_warmup	int	0	Warmup epochs for early stopping.
--es_patience	int	5	Patience for early stopping.
--dropout	float	0.1	Dropout rate for the model.
--ga	int	1	Gradient accumulation steps.
--tuning	str	'ft'	Tuning method (ft, pt, or lora).
--clip_loss_weight	float	1.0	Weight for the CLIP loss.
--img_loss_weight	float	1.0	Weight for the image loss.
--text_loss_weight	float	1.0	Weight for the text loss.
--weighted	str	'diagnosis'	Weighted sampling method (diagnosis, semantic).
--tau	float	0.07	Temperature for the CLIP loss.
--out_dim	int	256	Output dimension of last layer before classifier.
--position	str	'all'	Where to put the LoRA modules.
--encoder	str	'both'	Encoder to apply LoRA to (text, vision, both).
--params	list[str]	['q','k','v']	Attention matrices to apply LoRA to.
--r	int	2	Rank of the low-rank matrices.
--alpha	int	1	Scaling factor (see LoRA paper).
--dropout_rate	float	0.25	Dropout rate applied before the LoRA module.

Additional arguments, such as those related to data augmentation, can be found in the script.

The checkpoints and arguments are saved in the result_dir, under folders named in the format experiment_YYYYMMDD_HHMMSS.

experiment_YYYYMMDD_HHMMSS/
├── args.txt               # All training arguments saved as plain text
├── log.txt                # Console output and training logs
├── fold_0/
│   ├── best_both.pt       # Checkpoint with best combined CLIP + prediction loss
│   ├── best_clip.pt       # Checkpoint with best CLIP loss
│   ├── best_pred.pt       # Checkpoint with best prediction loss
│   ├── ckpt.pt            # Latest checkpoint
│   └── events.out.tfevents...  # TensorBoard logs
├── ...

The training logs, including loss and evaluation metrics, are saved. You can visualize them with TensorBoard by running:

tensorboard --logdir=./results --port=1234

Then open http://<IP Address>:1234 in your browser.

2. ▶️Run Inference

CUDA_VISIBLE_DEVICES=0 python evaluate.py \
  --model_path ./results/experiment_YYYYMMDD_HHMMSS \
  --dataset_path ./dataset_csv/sample_csv.csv \
  --img_dir ./cropped_img \
  --num_workers 4 \
  --save_path ./results_csv \
  --ckpt_file best_both.pt

The output CSV file will be saved at {save_path}/experiment_YYYYMMDD_HHMMSS_best_both_result.csv. This file includes:

• pid: the unique patient or sample ID
• raw_X: the predicted probability from fold X
• ensemble: the average predicted probability across all folds Each row corresponds to a sample and its associated prediction results.

3. ▶️Run Inference Using Our Pretrained Model

Download ckpt from the link and put it under ./CLIP_nodule.

gdown --folder 'https://drive.google.com/drive/folders/1V1bUAt3Hl2WNh5eZmQCZHDqQmEd1FT7W?usp=sharing'

CUDA_VISIBLE_DEVICES=0 python evaluate.py \
  --model_path ./ckpt \
  --dataset_path ./dataset_csv/sample_csv.csv \
  --img_dir ./cropped_img \
  --num_workers 4 \
  --save_path ./results_csv \
  --ckpt_file best_both.pt \
  --calibrate

In our study, we performed beta calibration to prevent the model from being overconfident and ensure that the model's predicted probabilities reflect the true likelihoods. To enable calibration, add the --calibrate flag when running the evaluation script. This will generate additional columns in the output:

• calibrated_X: the calibrated probability from fold X
• calibrated_ensemble: the average of the calibrated probabilities across all folds

Acknowledgements

This project is based on the code from the following repository:

• CLIP
• CLIP-LoRA
• foundation-cancer-image-biomarker

CITATION

@article{zhuang2025vision,
    title = {Vision-language model-based semantic-guided imaging biomarker for lung nodule malignancy prediction},
    journal = {Journal of Biomedical Informatics},
    volume = {172},
    pages = {104947},
    year = {2025},
    issn = {1532-0464},
    doi = {https://doi.org/10.1016/j.jbi.2025.104947},
    url = {https://www.sciencedirect.com/science/article/pii/S1532046425001765},
    author = {Luoting Zhuang and Seyed Mohammad Hossein Tabatabaei and Ramin Salehi-Rad and Linh M. Tran and Denise R. Aberle and Ashley E. Prosper and William Hsu}
}

CONTACT

If you have any questions, please don't hesitate to contact us at luotingzhuang@g.ucla.edu.