where do we need the image_embeddings_cls in training_step

[garychan22]

hi, thanks for your excellent work here!

I am reading the code and a little bit confused by the image_embeddings_cls in the training_step. The attention layer accurately takes the image_embeddings as inputs, leaving the image_embeddings_cls to be recorded by self.image_infos and then unset (Line 556~596).

image_embeddings_cls, image_embeddings = self.encode_images(
            batch["entity_images"], batch["image_token_idx_mask"], batch["bboxes"], latents.device)
...
self.image_infos["image_embedding"] = image_embeddings_cls[batch["image_token_idx_mask"]]
...
objects = image_embeddings
self.image_infos["image_embedding"] = None

[Ed-ivan]

While the cls _embeddings is used to replace the text-embeddings , which you can find in the replace_clip_embedding.py