Training the model

[zarashomme]

Hello,

I am aiming to train EarthMind for greater natural disaster capabilities. However, it is unclear what formatting for segmentation 'instances.json' or vqa data_path='data/vrsbenchvqa/annotations/vqatrain.json' actually include. It would be a lot easier for me to make fine-tuning if I didn't have to reverse engineer these parameters. If these example files could be provided, that would be greatly appreciated.

Thanks.

[shuyansy]

Hi, i think the easiest way is to download my training dataset and understanding the format of data. Actually, vqa data is formatted like LLaVA format (https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md), where you should specify the image path and qa pairs. Segmentation data is formatted as Refcoco (https://paperswithcode.com/dataset/refcoco), where you should include instances.json (including image path, annotations...) and refs(unc).p (including text prompt and dataset split).

[shuyansy]

BTW, you can directly download the RRSISD data (https://drive.google.com/drive/folders/1Xqi3Am2Vgm4a5tHqiV9tfaqKNovcuK3A) which have formatted data. And you can try https://github.com/lichengunc/refer to visualize and check the refcoco-like data.