Sebastian Bock1,2 · Leonie Schüßler1,2 · Krishnakant Singh1 · Simone Schaub-Meyer1,2,3 · Stefan Roth1,2,3
1 TU Darmstadt 2 ELIZA 3 hessian.AI
MUFASA is a lightweight, plug-and-play framework for unsupervised object segmentation that
- 🔍 Computes slot attention across multiple ViT feature layers.
- 🧩 Fuses multi-layer slots into a unified object-centric representation.
- 📈 Improves segmentation quality across multiple benchmarks.
- 🏆 Sets a new state of the art when integrated into existing OCL methods.
- ⚡ Accelerates training convergence with only minimal inference overhead.
conda create -n mufasa python=3.9.16
conda activate mufasa
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu117MUFASA works on top of slot-attention-based models. We provide sample implementations for DINOSAUR and SPOT, both of which are implemented as instances of the BaselineModel class.
To use MUFASA with a different model, you need to wrap your model with the MUFASA wrapper in train.py. You may adapt the BaselineModel class, but this is not required. The only requirements that the MUFASA wrapper expects are:
- Implements a
forward_encoder()method that returns a tuple(multi_layer_features, reconstruction_target). - Contains a slot attention module which is invoked immediately after
forward_encoder().
# Create the model and wrap MUFASA around it
model = MUFASA(model, args)The MUFASA wrapper overrides the base model's forward_encoder() so that it returns features of multiple layers of the encoder, along with the reconstruction target. If your model performs additional operations inside forward_encoder(), you may need to modify the wrapper accordingly.
In addition, the wrapper replaces the standard slot attenion module with the multi-layer version. By default, the wrapper assumes that the slot attention module in the base model is named "slot_attn". If your model uses a different name, set the sa_module_name parameter accordingly.
We provide checkpoints for the following models:
| Model | Dataset | mBOc | mBOi | mIoU | Checkpoint |
|---|---|---|---|---|---|
| SPOT-M | VOC | 59.7 | 51.6 | 49.7 | Download |
| SPOT-M | COCO | 45.4 | 34.7 | 32.5 | Download |
| SPOT-M | MOVi-C | - | 49.3 | 48.3 | Download |
| DINOSAUR-M | VOC | 58.5 | 50.1 | 48.1 | Download |
| DINOSAUR-M | COCO | 43.2 | 33.1 | 31.0 | Download |
| DINOSAUR-M | MOVi-C | - | 49.2 | 48.2 | Download |
Note that these values may deviate slightly from results reported in the paper, as we average scores across three seeds in the paper, whereas the values stated above are computed using the respective checkpoint.
To evaluate MUFASA, run the following commands.
| Dataset | Command |
|---|---|
| COCO | python eval.py --dataset coco --data_path /path/to/coco --num_slots 7 --init_method shared_gaussian --use_second_encoder False --eval_permutations standard --train_permutations standard --checkpoint_path /path/to/weights/model.pt |
| PASCAL VOC | python eval.py --dataset voc --data_path /path/to/voc --num_slots 6 --init_method shared_gaussian --use_second_encoder False --eval_permutations standard --train_permutations standard --checkpoint_path /path/to/weights/model.pt |
| MOVi-C | python eval.py --dataset movi --data_path /path/to/movi --num_slots 11 --init_method shared_gaussian --use_second_encoder False --eval_permutations standard --train_permutations standard --val_mask_size 128 --checkpoint_path /path/to/weights/model.pt |
| Dataset | Command |
|---|---|
| COCO | python eval.py --dataset coco --data_path /path/to/coco --num_slots 7 --eval_permutations standard --checkpoint_path /path/to/weights/model.pt |
| PASCAL VOC | python eval.py --dataset voc --data_path /path/to/voc --num_slots 6 --eval_permutations standard --checkpoint_path /path/to/weights/model.pt |
| MOVi-C | python eval.py --dataset movi --data_path /path/to/movi --num_slots 11 --eval_permutations standard --val_mask_size 128 --checkpoint_path /path/to/weights/model.pt |
For the dataset preparation, we follow SPOT's instructions:
Download COCO dataset (2017 Train images,2017 Val images,2017 Train/Val annotations) from here and place them following this structure:
COCO2017
├── annotations
│ ├── instances_train2017.json
│ ├── instances_val2017.json
│ └── ...
├── train2017
│ ├── 000000000009.jpg
│ ├── ...
│ └── 000000581929.jpg
└── val2017
├── 000000000139.jpg
├── ...
└── 000000581781.jpgDownload PASCAL VOC 2012 dataset from http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar, extract the files and copy trainaug.txt in VOCdevkit/VOC2012/ImageSets/Segmentation. The final structure should be the following:
VOCdevkit
└── VOC2012
├── ImageSets
│ └── Segmentation
│ ├── train.txt
│ ├── trainaug.txt
│ ├── trainval.txt
│ └── val.txt
├── JPEGImages
│ ├── 2007_000027.jpg
│ ├── ...
│ └── 2012_004331.jpg
├── SegmentationClass
│ ├── 2007_000032.png
│ ├── ...
│ └── 2011_003271.png
└── SegmentationObject
├── 2007_000032.png
├── ...
└── 2011_003271.pngTo download MOVi-C/E datasets, uncomment the last two rows in requirements.txt to install the tensorflow_datasets package. Then, run the following commands:
python download_movi.py --level c --split train
python download_movi.py --level c --split validation
python download_movi.py --level e --split train
python download_movi.py --level e --split validationThe structure should be the following:
MOVi
├── c
│ ├── train
│ │ ├── 00000000
│ │ │ ├── 00000000_image.png
│ │ │ └── ...
│ │ └── ...
│ └── validation
│ ├── 00000000
│ │ ├── 00000000_image.png
│ │ ├── 00000000_mask_00.png
│ │ └── ...
│ └── ...
└── e
├── train
│ └── ...
└── validation
└── ...All models use the default MUFASA parameters:
- layer_selection="8,9,10,11"
- slot_fusion="mfusion"
- mask_fusion="learned"
| Dataset | Command |
|---|---|
| COCO | python train.py --dataset coco --data_path /path/to/COCO2017 --epochs 100 --num_slots 7 --train_permutations standard --eval_permutations standard --log_path /path/to/logs/coco |
| PASCAL VOC | python train.py --dataset voc --data_path /path/to/voc --epochs 1120 --num_slots 6 --train_permutations standard --eval_permutations standard --log_path /path/to/logs/voc |
| MOVi-C | python train.py --dataset movi --data_path /path/to/movi --epochs 95 --num_slots 11 --train_permutations standard --eval_permutations standard --log_path /path/to/logs/movic --val_mask_size 128 --lr_main 0.0002 --lr_min 0.00004 |
SPOT is trained in two stages: Stage 1: 👩🏫 MUFASA Teacher
| Dataset | Command |
|---|---|
| COCO | python train.py --dataset coco --data_path /path/to/COCO2017 --epochs 50 --num_slots 7 --train_permutations random --eval_permutations standard --log_path /path/to/logs/coco |
| PASCAL VOC | python train.py --dataset voc --data_path /path/to/voc --epochs 560 --num_slots 6 --train_permutations random --eval_permutations standard --log_path /path/to/logs/voc |
| MOVi-C | python train.py --dataset movi --data_path /path/to/movi --epochs 65 --num_slots 11 --train_permutations random --eval_permutations standard --log_path /path/to/logs/movic --val_mask_size 128 --lr_main 0.0002 --lr_min 0.00004 |
Stage 2: 🧑🎓 MUFASA Student
| Dataset | Command |
|---|---|
| COCO | python train_student.py --dataset coco --data_path /path/to/COCO2017 --epochs 50 --num_slots 7 --train_permutations random --eval_permutations standard --teacher_train_permutations random --teacher_eval_permutations random --factor_warmup_epochs 3 --lr_main 0.0003 --teacher_checkpoint_path /path/to/logs/spot_teacher_coco/weights.pt --log_path /path/to/logs/spot_coco |
| PASCAL VOC | python train_student.py --dataset voc --data_path /path/to/voc --epochs 560 --num_slots 6 --train_permutations random --eval_permutations standard --teacher_train_permutations random --teacher_eval_permutations random --teacher_checkpoint_path /path/to/logs/spot_teacher_voc/weights.pt --log_path /path/to/logs/spot_voc |
| MOVi-C | python train_student.py --dataset movi --data_path /path/to/movi --epochs 30 --num_slots 11 --train_permutations random --eval_permutations standard --teacher_train_permutations random --teacher_eval_permutations random --teacher_checkpoint_path /path/to/logs/spot_teacher_movi/weights.pt --log_path /path/to/logs/spot_movi --val_mask_size 128 --lr_main 0.0002 --lr_min 0.00015 --predefined_movi_json_paths train_movi_paths.json |
This repository is based upon SPOT. We thank them for their work!
If you find this repository useful, please consider giving a star ⭐ and citation:
@InProceedings{Bock_2026_CVPR,
author = {Bock, Sebastian and Sch\"u{\ss}ler, Leonie and Singh, Krishnakant and Schaub-Meyer, Simone and Roth, Stefan},
title = {MUFASA: A Multi-Layer Framework for Slot Attention},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2026},
pages = {27750-27760}
}