Isolated Sign Language Recognition with ODE-Transformers using Skeleton based Modality

Project Overview

This repository contains the codebase for an experimental setup designed to evaluate the performance of Ordinary Differential Equation (ODE) integrated Transformers on Isolated Sign Language Recognition. The experiments are conducted on the AUTSL (Ankara University Turkish Sign Language) dataset using MediaPipe landmark data.

📊 Dataset & Landmark Extraction

The data used in this project consists of pre-extracted MediaPipe landmarks from the original AUTSL dataset videos. If you wish to extract these normalized landmarks from the raw AUTSL videos yourself from scratch, you can use the following open-source extraction tool provided by the author: 👉 MediaPipe Normalized Landmark Extractor

Label Files (*_labels.csv): The dataset is accompanied by label files (e.g., train_labels.csv, val_labels.csv). These files map each .pt landmark file to its corresponding integer class label representing the specific sign language gesture (226 classes).

🌟 Theoretical Background & Value Proposition

Standard Transformers rely on discrete layer-to-layer transitions (e.g., $x_{l+1} = x_l + F(x_l)$), which can be interpreted as a simple Euler discretization of a continuous-time Ordinary Differential Equation (ODE).

In this project, we implement and test advanced continuous-time ODE formulations within the Transformer's Encoder architecture. Instead of standard layer progression, we introduce advanced ODE solvers such as Runge-Kutta (RK2, RK4) at the architectural level.

What do we achieve with this modification?

• Enhanced Feature Representation: By integrating ODE solvers, the network learns continuous dynamic representations of sign language landmarks, which naturally capture the fluid, temporal nature of human gestures better than discrete steps.
• Robustness & Stability: Advanced numerical methods like RK4 offer better stability and deeper feature extraction without necessarily increasing the sheer number of physical layers.
• Architectural Flexibility: We implemented different gating mechanisms and learnable coefficients (e.g., Learnable RK, Standard RK, Initialization RK) directly inside the Transformer Encoder (ode_transformer_encoder.py and fairseq_ode_transformer.py).

This project aims to answer: Does modeling the hidden state transitions as a continuous differential equation improve the accuracy and robustness of Isolated Sign Language Recognition?

🧠 Architecture & Evaluation

• Architecture: The model consists of a linear embedding layer that projects the flattened 48-point landmark features into the hidden dimension (d_model), followed by sinusoidal positional encoding. The core is the ODETransformerEncoder which replaces standard discrete self-attention layers with Runge-Kutta continuous solvers. A final linear classifier outputs predictions for the 226 AUTSL sign classes.
• Evaluation Metric: Due to potential class imbalances and the multi-class nature of the problem, the models are evaluated primarily on Macro F1 Score alongside standard Accuracy to ensure balanced performance across all gestures.

---

🛠️ Repository Structure

• train.py: Main training script for the standard (or custom) models using PyTorch Lightning.
• train_fairseq.py: Alternative training script utilizing the Fairseq framework's backend components.
• run_experiments.py: Automation script to train the model across various hyperparameters without ODE implementations, saving baseline results to a CSV.
• run_ode_experiments.py: Automation script that reads the best performing hyperparameters from the baseline experiments and applies different combinations of ODE implementations (enc_calculate_num, rk_type) to find the optimal continuous-time architecture.
• evaluate_model.py: Script to load checkpoints and evaluate the model on the test set, outputting detailed metrics (Accuracy, Macro F1, Inference time).
• run_evaluations.py: Automation script to run evaluations iteratively across multiple model versions.
• generate_results_table.py: Utility script to parse YAML evaluation logs and compile them into structured CSV/Excel tables for comparative analysis against the baselines.
• ode_transformer_encoder.py: Core implementation of the ODE Transformer Encoder in native PyTorch.
• fairseq_ode_transformer.py: Core implementation of the ODE Transformer adapted for the Fairseq ecosystem.

🚀 Setup & Installation

1. Clone the repository:

   git clone <your-repo-url>
   cd <your-repo-name>

2. Install requirements: It is recommended to use a virtual environment.

   pip install -r requirements.txt

3. Data Preparation: Ensure that the AUTSL landmark .pt files and corresponding .csv labels are placed in your desired data directories. Update the dataset paths in the scripts (train.py, evaluate_model.py, etc.) as needed.

💻 Usage

1. Baseline Hyperparameter Search

To search for the best model parameters without ODE integration:

python run_experiments.py

This will generate training_results.csv.

2. ODE Integration Experiments

After identifying the best baseline configurations (ensure they are saved as best_experiment_results_without_ODE_implementation.csv), run the ODE experiments:

python run_ode_experiments.py

This script will test various enc_calculate_num (1, 2, 3, 4) and rk_type (standard, learnable, initialization) combinations.

3. Single Model Training

To train a specific configuration manually:

python train.py --num_frames 16 --d_model 512 --nhead 4 --num_layers 4 --enc_calculate_num 2 --rk_type learnable

4. Evaluation

To evaluate a single trained checkpoint and generate metrics:

python evaluate_model.py --version <version_number>

To run automated evaluations across multiple trained model versions sequentially:

python run_evaluations.py

To compile all evaluation YAML logs into a comprehensive Excel table:

python generate_results_table.py

---

📖 Citations

If you use this code in your research, please cite the following works:

1. Thesis / Project Citation:

@mastersthesis{isik2026iterative,
  author  = {Işık, Ö. F.},
  title   = {An Iterative Transformer Framework for Isolated {Turkish} Sign Language Recognition},
  school  = {Hacettepe Üniversitesi, Fen Bilimleri Enstitüsü},
  year    = {2026},
  type    = {Yüksek Lisans Tezi},
  address = {Ankara, Türkiye}
}

2. Original ODE Transformer:

@article{li2021ode,
  title={Ode transformer: An ordinary differential equation-inspired model for neural machine translation},
  author={Li, Bei and Du, Quan and Zhou, Tao and Zhou, Shuhan and Zeng, Xin and Xiao, Tong and Zhu, Jingbo},
  journal={arXiv preprint arXiv:2104.02308},
  year={2021}
}