Question regarding Coarse Stage

[sauradip]

Hi ,

I was curious if we can overfit the model to multiple frames in the coarse stage instead of the middle frame of the video.
I guess this is done ( by NERF based approaches) so that the deformation can be estimated in the refined stage w.r.t canonical shape. So did you try with multiple frames and is there any reason not to use multiple reference shape during refined shape ?

[zeng-yifei]

Of course you can fit multiple images in the coarse stage. But this means you need to turn on the deformation network in this stage, which will make the 'coarse' stage not that coarse and fast. Anyway, you can try it and may update some results here. We are happy to see that.

[sauradip]

Thank You ! I also observed the back view is very bad , but front view is good. Any pruning issue here ? I feel it is mainly because of some incorrect optimization. What can be the possible causes ? Zero-1-2-3 is the issue ? Did you try ImageDream ?

[zeng-yifei]

In the version before June 20, the optimization code was actually something wrong as you said. We fix the loss after that. In some cases where zero123 or zero123++ does not good, our method will also fail to generate a stable result.

For ImageDream (or Stable Zero123), we tested it in a very early version and found it not very helpful. Maybe you can try it out by yourself.