beast

Behavioral analysis via self-supervised pretraining of transformers

beast is a package for pretraining vision transformers on unlabeled data to provide backbones for downstream tasks like pose estimation, action segmentation, and neural encoding.

See the ICLR paper here.

Installation

Step 1: Install ffmpeg

First, check to see if you have ffmpeg installed by typing the following in the terminal:

ffmpeg -version

If not, install:

sudo apt install ffmpeg

Step 2: Create a conda environment

First, install anaconda.

Next, create and activate a conda environment:

conda create --yes --name beast python=3.10
conda activate beast

Step 3: Download and install

Move to your home directory (or wherever you would like to download the code) and install via Github clone or through PyPI.

For Github cloning:

git clone https://github.com/paninski-lab/beast
cd beast
pip install lightning poetry-core
pip install -e . --no-build-isolation

For installation through PyPI:

pip install lightning poetry-core
pip install beast-backbones --no-build-isolation

Note: beast depends on a custom fork of gsplat that must be compiled from source. The gsplat build requires torch (provided by lightning) and the build backend requires poetry-core. Installing these first and using --no-build-isolation lets the build find them in your environment.

Usage

The commands below are for the single-view BEAST model. For multi-view 3D data, see the BEAST3D documentation.

beast comes with a simple command line interface. To get more information, run

beast -h

Extract frames

Extract frames from a directory of videos to train beast with.

beast extract --input <video_dir> --output <output_dir> [options]

Type "beast extract -h" in the terminal for details on the options.

Train a model

You will need to specify a config path; see the configs directory for examples.

beast train --config <config_path> [options]

Type "beast train -h" in the terminal for details on the options.

Run inference

Inference on a single video or a directory of videos:

beast predict --model <model_dir> --input <video_path> [options]

Inference on (possibly nested) directories of images:

beast predict --model <model_dir> --input <video_path> [options]

Type "beast predict -h" in the terminal for details on the options.

---

Contributing

See CONTRIBUTING.md for guidelines on setting up a development environment, code style, and submitting pull requests.

Funding

We are grateful for support from the following:

• Gatsby Charitable Foundation GAT3708
• NIH R50NS145433
• NIH U19NS123716
• NSF 1707398
• The NSF AI Institute for Artificial and Natural Intelligence
• Simons Foundation
• Wellcome Trust 216324
• Zuckerman Institute (Columbia University) Team Science