Cross-Modal Perception and Contrastive Learning for Object Detection in Endoscopic Thyroid Surgery Videos(CPCL).

CPCL is an enhanced version of TSCD_Plus, designed to alleviate the ineffectiveness of feature aggregation and feature matching caused by semantic similarity degradation. TSCD_Plus is an enhanced version of TSCD, which is pytorch implementation for the paper "Task-Specific SpatioTemporal Context-Aware Decoupling for Occluded Video Object Detection"(2025 ICIP).

Introduction

Cross-Modal Perception and Contrastive Learning for Object Detection in Endoscopic Thyroid Surgery Videos(CPCL) is a video object detector for Endoscopic Thyroid Surgery videos, which improves upon the deficiencies of TSCD_Plus. Due to the weak and highly fragmented feature representations of small objects under complex backgrounds and occlusion, their semantic similarity is significantly degraded, which renders the aggregation mechanism based on feature similarity alignment ineffective. To address this issue, we enhance feature recognition and matching for small objects in complex backgrounds, thereby improving video object detection performance.

• Text-Guided SpatioTemporal Context-Aware (TGCA) achieves cross-modal matching between spatiotemporal contextual features and textual features, guiding the model to focus on semantic regions consistent with textual descriptions across multiple frames, thereby improving the effectiveness of subsequent feature aggregation.
• Additionally, Spatiotemporal Context-Aware Feature Matching (CAFM) driven by contrastive learning enhances the robustness of small object feature matching.

This repo is an implementation of PyTorch version CPCL based on TSCD_Plus, TSCD, YOLOV and YOLOX.

Quick Start

Installation

Install CPCL from source.

git clone git@github.com:KaihongLi/CPCL.git
cd CPCL

Create conda env.

conda create -n cpcl python=3.8.19

conda activate cpcl

pip install -r requirements.txt

pip3 install -v -e .

Reproduce our results on Thyroid (Reference to TSCD_Plus, TSCD, YOLOV++)

Endoscopic Thyroid Surgery Videos is used to validate the effect of detecting small objects under complex backgrounds and occlusion.

Step 1. Organize the dataset as follows:

path to your datasets/thyroid

Step 2. Change the "data_dir" field in exp files to [path to your datasets].

Training:

1. The baseline YOLOX detector was initialized with COCO-pretrained weights from the YOLOX official repository and fine-tuned on the Thyroid dataset.

python tools/train.py -f exps/thyroid_default/thyroid_yolox_l.py -c [yolox_l pretrained weights on COCO] -b [batch size] -d [your devices] --fp16

2. Initialize the CPCL with finetuned weights obtained by the up step.

python tools/tscd_train.py -f exps/CPCL_Thyroid/thyroid_cpcl_large.py -c [path to your weights] --fp16

Evaluation:

python tools/tscd_plus_eval.py -f exps/CPCL_Thyroid/thyroid_cpcl_large.py -c path to your weights/thyroid_cpcl_large.pth --dataset thyroid --fp16

Visualization:

python tools/tscd_plus_demo.py -f exps/CPCL_Thyroid/thyroid_tscd_large.py -c [path to your weights]/thyroid_cpcl_large.pth --dataset thyroid --path [path to your video] --conf 0.25 --nms 0.5 --tsize 576 --save_result True

(For yolox models, please use python tools/demo.py for inference.)

Acknowledgements

• TSCD
• TSCD_Plus
• YOLOV
• YOLOX
• YOLOE
• DVIS_Plus
• Robust and efficient post-processing for video object detection