Were there any image generation tests with text guidance?

[Guanyu-Zhou]

Were there any image generation tests with text guidance? When I trained the image generation with text guidance using the script that could normally train Controlnet and UNet, the training crashed. This problem was not solved in MONAI or the scripts provided in this repository, even though I adjusted the hyperparameters such as the learning rate and the crash still occurred. The main manifestation was that the loss fluctuated little, and the training crashed after a certain number of epochs.

[Guanyu-Zhou]

It would be greatly appreciated if there were a set of code that could be used as a guide for learning.

[Can-Zhao]

We have paper Text2CT which includes some training details. https://arxiv.org/pdf/2505.04522 @guopengf

[gerd-bizi]

@Can-Zhao is there any plan to release the repository for Text2CT?
If there isn't a question that I have is about hooking up text embeddings into the generation process.
If I am keeping the DiffUNet frozen, then should I only hook up my text embeddings to the ControlNet?
Or would it make sense to unfreeze the DiffUNet? If I unfreeze the DiffUNet, I assume I'd also have to unfreeze the VAE as well. Any insight would be greatly appreciated.

[gerd-bizi]

Hi @Can-Zhao, just following up on this!

@Can-Zhao is there any plan to release the repository for Text2CT? If there isn't a question that I have is about hooking up text embeddings into the generation process. If I am keeping the DiffUNet frozen, then should I only hook up my text embeddings to the ControlNet? Or would it make sense to unfreeze the DiffUNet? If I unfreeze the DiffUNet, I assume I'd also have to unfreeze the VAE as well. Any insight would be greatly appreciated.

[Can-Zhao]

For VAE, from the feedbacks we received, usually VAE can be frozen, unless you are focusing on the soft tissue in CT head data.

For DiffUNet, usually you can also freeze DiffUNet, unless you find your dataset are pediatric or has disease that is not covered in NV-Generate training data.