Scene-Aware Social Transformer

[Paper] [arXiv] [Supplementary Material]

Implementation for the ABAW@ECCV 24 workshop paper "Massively Multi-Person 3D Human Motion Forecasting with Scene Context".

Usage

This code was tested with Python 3.10. Install all dependencies with

pip install -r requirements.txt

Download the Humans in Kitchens and unpack its content to data/, such that data/ contains poses/, scenes/, and body_models/.

Preprocess the dataset using

python sast/data/multi_person_data.py hik SAST.yaml 

This will load pose information from Humans in Kitchens and store them at data/hik_[ABC].

Train the model with

python train.py SAST.yaml

Generate model outputs for all sequences in the Humans in Kitchens evaluation set using hik.eval.Evaluator.

python eval.py path/to/model data/

This will create a file eval.pkl that can be analyzed using Humans in Kitchens evaluation code.

metric_calculation.py documents how we calculate the metrics reported in the paper based on the eval files of our and baseline models. You probably need to adjust paths to files to run the script.

Reference

If you found this repository useful, please cite

@inproceedings{mueller2024sast,
  author       = {Felix B. Mueller and
                  Julian Tanke and
                  Juergen Gall},
  title        = {Massively Multi-person 3D Human Motion Forecasting with Scene Context},
  booktitle    = {Computer Vision - {ECCV} 2024 Workshops - Milan, Italy, September
                  29-October 4, 2024, Proceedings, Part {XV}},
  series       = {Lecture Notes in Computer Science},
  volume       = {15637},
  pages        = {130--147},
  publisher    = {Springer},
  year         = {2024},
  url          = {https://doi.org/10.1007/978-3-031-91581-9\_10},
  doi          = {10.1007/978-3-031-91581-9\_10},
}