BYOL-VS: Self Supervised Learning on lingual Ultrasound Video Data for Speech Sound Disorder

Overview

Speech sound disorder (SSD) affects many school-aged children and can lead to long-term academic, social, and psychosocial difficulties. Lingual ultrasound provides noninvasive visualization of tongue motion for real-time articulatory feedback but requires labor-intensive expert annotation of complex spatiotemporal patterns, limiting scalability in routine clinical workflows.

We present a self-supervised learning study of within-phoneme articulatory structure in the American English rhotic from lingual ultrasound video using a ViViT encoder trained with a BYOL objective and an adversarial module to suppress speaker-related variation. Even without task specific fine-tuning, the model demonstrates accurate discrimination of perceptual accuracy and articulatory subtype (retroflex vs. bunched) with representations aligned to expert-labeled tongue shapes.

👅🖥️ We train our model on PERCEPT-US, which uniquely provides expert labeled within-rhotic categories. (To be released)

📊 Even without task specific fine-tuning, we achieve mean ACC: 0.974, mean MCC: 0.941 on determining perceptual accuracy and mean ACC: 0.961, mean MCC: 0.897 on discriminating articulatory subtypes of perceptually accurate rhotics.

Usage

This package was developed using Python 3.12 and Pytorch 2.6.0.

To install the package, please run the following lines:

git clone $BYOL-VS repo$
cd $PATH to BYOL-VS repo$
pip install -r requirements.txt

To use:

python -m src.main --mode_train {train or inference} 
                   --base_path {data_path} 
                   --output_dir {output_path}
                   --device cuda 
                   --batch_size 16 
                   --num_epochs 200 
                   --lr 2e-4 
                   --warmup_epochs 20 
                   --min_lr 1e-6 
                   --grl_lambda 0.1 
                   --speaker_hidden_size 256

Citing our work

Accepted to MICCAI 2026, TBA