Results published: Gosztolai, A., Gunel, S., Lobato-Rios, V., Abrate, M. P., Morales, D., Rhodin, H., Fua, P., & Ramdya, P. (2021). LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animals. Nature methods.

Tool available on this GitHub page.

Predicting Drosophila melanogaster 3D pose from the 2D pose

This is the code for the semester project

Predicting Drosophila melanogaster 3D pose from the 2D pose

Student: Marco Pietro Abrate

Supervisor: Semih Gunel

EPFL Lab: Neuroengineering Laboratory (RAMDYA)

The aim of this project is to reduce the number of cameras mounted on the platform that are used by DeepFly3D to predict the 3D pose of Drosophila melanogaster.

Please consider reading the report.pdf for a detailed explanation of this project.

Dependencies (Python 3.7)

• tensorflow 1.0 or later
• matplotlib

First of all

Clone this repository and get the data. The data must be downloaded in the right folders. The name of the training and testing files can be found in the project report, Appendix A.

git clone https://github.com/marcoabrate/predict3Dpose_drosophila
cd 3d-pose-baseline
mkdir flydata_train
mkdir flydata_test

Show results

For showing the results of the final model, you have to go in the 3d-pose-baseline folder, and run

python flysrc/predict_3dpose.py --residual --batch_norm --dropout 0.5 --max_norm --epochs 200 --origin_bc --camera_frame --test --load 266800

This will produce an animation similar to this video and a visualization similar to this:

Training

To train a model from scratch, run (always from the 3d-pose-baseline folder):

python flysrc/predict_3dpose.py --residual --batch_norm --dropout 0.5 --max_norm --epochs 200 --origin_bc --camera_frame --train_dir "new_model"