Scuba-Gesture-Dataset (SGD11)

The aim of the SCUBA Gesture dataset is to further the state-of-the-art (SOTA) in underwater human-robot-interaction (UHRI) research by providing a robust source of training data for static and dynamic action/gesture recogntion algorithms, and a standardized set of test data for offline evaluation of diver gesture recognition pipeline performance, assessed across a dynamic set of gestures (for more information on pipeline evaluation, see pipeline evaluation).

The datset currently consists of 1100 videos of diver actions/gestures of 11 different classes. This includes including 220 videos with synchronized diver pose data.​

The majority the the current dataset are video sequences of 2 seconds in length, where each video contains only one gesture.

Depiction of each available gesture

Access the full dataset here!

https://www.dropbox.com/scl/fo/5rxj3gthms3urg4lqajwk/AFigP8iOmR46MKk3KaxDoWs?rlkey=r4qctmr5j0p9akvsiqmynhown&st=zmd7s5of&dl=0

Dataset Folder Structure

Public_dataset/
   - OKAY
      - OKAY_TANK_VERTICAL_1f8s0f8an4.MP4
      - OKAY_TANK_VERTICAL_1f8s0f8an4.MP4
      - OKAY_TANK_VERTICAL_1f8s0f8an4.MP4
      ...
   - BUDDY_UP
      - BUDDYUP_TANK_VERTICAL_1f8s0f8an4.MP4
      - BUDDYUP_TANK_VERTICAL_1f8s0f8an4.MP4
      - BUDDYUP_TANK_VERTICAL_1f8s0f8an4.MP4
      ...
   - RIGHT
      ...
    ...

Synchronized Diver Pose + Video Frames (SYNCHRONIZED_DIVER_DATA/)

In addition to raw gesture video data, the dataset includes a structured subset named SYNCHRONIZED_DIVER_DATA/, located within the main Dropbox folder. This directory contains synchronized 3D diver pose data (from Qualisys QTM) paired with corresponding camera frames from the BlueROV2. It is intended for training and evaluating multimodal models that combine visual and motion cues for more robust underwater gesture recognition.

Unlike the Public_dataset/ folder, which contains raw videos, SYNCHRONIZED_DIVER_DATA/ follows a new format where each gesture is organized into:

• A gesture folder (e.g., YOU/, BUDDY_UP/, etc.)
• Ten numbered subfolders (1/ through 10/) per gesture, each containing:
  • A frames/ folder of synchronized image frames.
  • A .json file with 3D marker data and (when possible) 6D rigid body poses.

Folder Layout Example

SYNCHRONIZED_DIVER_DATA/
├── YOU/
│   ├── 1/
│   │   ├── frames/
│   │   └── YOU_1.json
│   ├── 2/
│   │   ├── frames/
│   │   └── YOU_2.json
│   └── ...
├── BUDDY_UP/
│   ├── 1/
│   │   ├── frames/
│   │   └── BUDDY_UP_1.json
│   └── ...

JSON Data Format

Each JSON file is exported from QTM using the following settings:

Rigid bodies (if present) are labeled as:

• DIVER: created from multiple tracked diver markers or from a single stable marker if insufficient.
• BR2: created from BlueROV2 markers or a representative point.

This structure ensures a clean alignment between pose and visual data for each short gesture clip, making the dataset suitable for pose-supervised gesture classification and tracking tasks.

Tools

The tools directory contains an expanding collection of tools for working with and transforming the dataset. Contributions are welcome.

Training

The videos can be used for model training. Simply download the videos you need for training your model. We provide an example class for loading specific data of interest from the dataset.

Pipeline Evaluation

A key initiative of this work is to establish a large, open-source dataset to serve as a standard benchmark for diver action and gesture recognition. By providing a publicly available dataset along with standardized evaluation procedures and metrics, we aim to create a consistent and reproducible framework for assessing different recognition pipelines.

Currently, it is challenging to systematically determine what contributes to improvements in diver gesture recognition. Ambiguities exist regarding whether variations in performance are due to differences in image quality, model training strategies, underwater image enhancement techniques (or lack thereof), or the choice of model architecture (e.g., CNNs, LSTMs, Transformers). By establishing a common dataset and evaluation criteria, we enable researchers to scientifically compare different approaches, identify key factors driving improvements, and ultimately advance the robustness and reliability of diver gesture recognition systems.

Provided is an example jupyter notebook for loading the desired component of the dataset you would like to test over, loading the data, and running inference on a pretrained model.

Model results are evaluated using a confusion matrix over the classes tested, with results averaged over 10 runs.

Multi-class F-score is also computed as a metric, also averaged over 10 runs.

How to Contribute your own data

Please contribute your scuba diving gesture data! Follow the instructions in 'dataset_contribution_guide.md'

This work is made available for academic use under CC BY-NC-SA.

For commercial use and queries please contact jane.shin@ufl.edu