Pyramid-VAE-GAN

Pyramid-VAE-GAN: Transferring Hierarchical Latent Variables for Image Inpainting

Huiyuan Tian, Li Zhang, Shijian Li, Min Yao and Gang Pan

Introduction

Our contributions made in this paper are summarized as follows:

• To the best of our knowledge, the Pyramid-VAE backbone is used in the domain of image inpainting firstly. Our model augments the details of generated pictures by pyramid structure while improving rationality of structure via VAE.
• We propose a latent variable transfer module for the first time. The long-term correlations in high-level latent variables can be transferred to low-level latent variables with more details, thus it is capable of generating both visually and semantically convincing images.
• Our VAE-GAN hybrid model improves the robustness of the training process and enhances the sharpness of the generated images.

Example Results

Run

Our work is mainly based on PEN-Net-for-Inpainting.

1. Environments:
   1. Install python3.6
   2. Install PyTorch (tested on Release 1.7.1)
2. Preparation
   1. Download training/valid/test images file lists and put them into flist folder
   2. Download datasets and put them into datazip folder
   3. Keep flist, datazip and Pyramid-VAE-GAN folder in the same level directory
3. Training: For different datasets, the following commands can be used for training respectively.

# facade
python train.py -c configs/facade.json -n pvg_facade -m square -s 256
# dtd
python train.py -c configs/dtd.json -n pvg_dtd -m square -s 256
# celebahq 
python train.py -c configs/celebahq.json -n pvg_celebahq -m square -s 256

4. Testing: For different datasets, the following commands can be used for testing respectively.

# facade
python test.py -c configs/facade.json -n pvg_facade -m square -s 256
# dtd
python test.py -c configs/dtd.json -n pvg_dtd -m square -s 256
# celebahq
python test.py -c configs/celebahq.json -n pvg_celebahq -m square -s 256

5. Evaluating:

python eval.py --real_dir [ground truths] --fake_dir [inpainting results] --metric mae psnr ssim fid

Pretrained Models

1. Download the pretrained models named pvg_celebahq_celebahq_square256 for CELEBA-HQ and put it under release_model/
2. DTD and Facade are similar.

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@article{tian2023pyramid,
  title={Pyramid-VAE-GAN: Transferring hierarchical latent variables for image inpainting},
  author={Tian, Huiyuan and Zhang, Li and Li, Shijian and Yao, Min and Pan, Gang},
  journal={Computational Visual Media},
  pages={1--15},
  year={2023},
  publisher={Springer}
}