[Benchmark] Support CVQA

[timothycdc]

Adds support for the Cultural VQA (CVQA) benchmark.
Despite 'VQA' being in the name, CVQA is actually multiple-choice.

I used the dataset's region+language column (e.g. Japan, Japanese) as l2-category split.

Original dataset link
TSV converted version

[FangXinyu-0913]

HI @timothycdc, Thanks for your contribution to our community. Here are the results of the test on our side for your reference. If you feel okay, we will merge this PR.

---

split test
Overall 0.3877964141122036
('Amharic', 'Ethiopia') 0.24786324786324787
('Bengali', 'India') 0.18181818181818182
('Breton', 'France') 0.2962962962962963
('Bulgarian', 'Bulgaria') 0.3692722371967655
('Chinese', 'China') 0.4212218649517685
('Chinese', 'Singapore') 0.3915094339622642
('Egyptian_Arabic', 'Egypt') 0.1724137931034483
('Filipino', 'Philippines') 0.4482758620689655
('Hindi', 'India') 0.3283582089552239
('Igbo', 'Nigeria') 0.325
('Indonesian', 'Indonesia') 0.4029126213592233
('Irish', 'Ireland') 0.4110429447852761
('Japanese', 'Japan') 0.3103448275862069
('Javanese', 'Indonesia') 0.3602693602693603
('Kinyarwanda', 'Rwanda') 0.33617021276595743
('Korean', 'South Korea') 0.3724137931034483
('Malay', 'Malaysia') 0.473015873015873
('Marathi', 'India') 0.24257425742574257
('Minangkabau', 'Indonesia') 0.350597609561753
('Mongolian', 'Mongolia') 0.22115384615384615
('Norwegian', 'Norway') 0.5418060200668896
('Oromo', 'Ethiopia') 0.38317757009345793
('Portuguese', 'Brazil') 0.5985915492957746
('Romanian', 'Romania') 0.5198675496688742
('Russian', 'Russia') 0.55
('Sinhala', 'Sri_Lanka') 0.21777777777777776
('Spanish', 'Argentina') 0.5471698113207547
('Spanish', 'Chile') 0.5897435897435898
('Spanish', 'Colombia') 0.5311203319502075
('Spanish', 'Ecuador') 0.5552486187845304
('Spanish', 'Mexico') 0.4613003095975232
('Spanish', 'Spain') 0.660377358490566
('Spanish', 'Uruguay') 0.44126984126984126
('Sundanese', 'Indonesia') 0.35
('Swahili', 'Kenya') 0.37362637362637363
('Tamil', 'India') 0.18691588785046728
('Telugu', 'India') 0.27
('Urdu', 'India') 0.2
('Urdu', 'Pakistan') 0.10648148148148148
Brands / products / companies 0.3910355486862442
Cooking and food 0.35877862595419846
Geography / buildings / landmarks 0.4
Objects / materials / clothing 0.3337531486146096
People and everyday life 0.4195624195624196
Plants and animal 0.36418816388467373
Public Figure and pop culture 0.41467576791808874
Sports and recreation 0.41372141372141374
Traditions / art / history 0.39231738035264485
Vehicles and Transportation 0.4074074074074074

---

BTW, the SYSPROMPT in the PR is not used in the actual evaluation. If you want it to be used by all models, you can write a build_prompt function in combination with this prompt and put it under the class CVQA.

[timothycdc]

Thanks @FangXinyu-0913. Which model did you use for this? The score seems a bit lower than normal.

I will double-check the accuracy calculation code and fix the SYSPROMPT.

[FangXinyu-0913]

Thanks @FangXinyu-0913. Which model did you use for this? The score seems a bit lower than normal.

I will double-check the accuracy code and fix the SYSPROMPT.

I use the llava_v1.5_7b model and get this result.

[kennymckormick]

BTW, a large proportion of VLMs may override the build_prompt method for MCQ benchmarks. So if you would like to force all VLMs to use the official prompt, you may consider change the DATASET_TYPE to CVQA_MCQ, etc.

HI @timothycdc, Thanks for your contribution to our community. Here are the results of the test on our side for your reference. If you feel okay, we will merge this PR.

split test Overall 0.3877964141122036 ('Amharic', 'Ethiopia') 0.24786324786324787 ('Bengali', 'India') 0.18181818181818182 ('Breton', 'France') 0.2962962962962963 ('Bulgarian', 'Bulgaria') 0.3692722371967655 ('Chinese', 'China') 0.4212218649517685 ('Chinese', 'Singapore') 0.3915094339622642 ('Egyptian_Arabic', 'Egypt') 0.1724137931034483 ('Filipino', 'Philippines') 0.4482758620689655 ('Hindi', 'India') 0.3283582089552239 ('Igbo', 'Nigeria') 0.325 ('Indonesian', 'Indonesia') 0.4029126213592233 ('Irish', 'Ireland') 0.4110429447852761 ('Japanese', 'Japan') 0.3103448275862069 ('Javanese', 'Indonesia') 0.3602693602693603 ('Kinyarwanda', 'Rwanda') 0.33617021276595743 ('Korean', 'South Korea') 0.3724137931034483 ('Malay', 'Malaysia') 0.473015873015873 ('Marathi', 'India') 0.24257425742574257 ('Minangkabau', 'Indonesia') 0.350597609561753 ('Mongolian', 'Mongolia') 0.22115384615384615 ('Norwegian', 'Norway') 0.5418060200668896 ('Oromo', 'Ethiopia') 0.38317757009345793 ('Portuguese', 'Brazil') 0.5985915492957746 ('Romanian', 'Romania') 0.5198675496688742 ('Russian', 'Russia') 0.55 ('Sinhala', 'Sri_Lanka') 0.21777777777777776 ('Spanish', 'Argentina') 0.5471698113207547 ('Spanish', 'Chile') 0.5897435897435898 ('Spanish', 'Colombia') 0.5311203319502075 ('Spanish', 'Ecuador') 0.5552486187845304 ('Spanish', 'Mexico') 0.4613003095975232 ('Spanish', 'Spain') 0.660377358490566 ('Spanish', 'Uruguay') 0.44126984126984126 ('Sundanese', 'Indonesia') 0.35 ('Swahili', 'Kenya') 0.37362637362637363 ('Tamil', 'India') 0.18691588785046728 ('Telugu', 'India') 0.27 ('Urdu', 'India') 0.2 ('Urdu', 'Pakistan') 0.10648148148148148 Brands / products / companies 0.3910355486862442 Cooking and food 0.35877862595419846 Geography / buildings / landmarks 0.4 Objects / materials / clothing 0.3337531486146096 People and everyday life 0.4195624195624196 Plants and animal 0.36418816388467373 Public Figure and pop culture 0.41467576791808874 Sports and recreation 0.41372141372141374 Traditions / art / history 0.39231738035264485 Vehicles and Transportation 0.4074074074074074

BTW, the SYSPROMPT in the PR is not used in the actual evaluation. If you want it to be used by all models, you can write a build_prompt function in combination with this prompt and put it under the class CVQA.

[FangXinyu-0913]

---

split test
Overall 0.5496433391170233
('Amharic', 'Ethiopia') 0.5170940170940171
('Bengali', 'India') 0.5244755244755245
('Breton', 'France') 0.38271604938271603
('Bulgarian', 'Bulgaria') 0.5148247978436657
('Chinese', 'China') 0.5466237942122186
('Chinese', 'Singapore') 0.6273584905660378
('Egyptian_Arabic', 'Egypt') 0.5221674876847291
('Filipino', 'Philippines') 0.5270935960591133
('Hindi', 'India') 0.6716417910447762
('Igbo', 'Nigeria') 0.52
('Indonesian', 'Indonesia') 0.5097087378640777
('Irish', 'Ireland') 0.6073619631901841
('Japanese', 'Japan') 0.4039408866995074
('Javanese', 'Indonesia') 0.5286195286195287
('Kinyarwanda', 'Rwanda') 0.5063829787234042
('Korean', 'South Korea') 0.5413793103448276
('Malay', 'Malaysia') 0.5365079365079365
('Marathi', 'India') 0.6485148514851485
('Minangkabau', 'Indonesia') 0.5219123505976095
('Mongolian', 'Mongolia') 0.4551282051282051
('Norwegian', 'Norway') 0.6053511705685619
('Oromo', 'Ethiopia') 0.514018691588785
('Portuguese', 'Brazil') 0.6338028169014085
('Romanian', 'Romania') 0.5794701986754967
('Russian', 'Russia') 0.555
('Sinhala', 'Sri_Lanka') 0.6088888888888889
('Spanish', 'Argentina') 0.5547169811320755
('Spanish', 'Chile') 0.594017094017094
('Spanish', 'Colombia') 0.5767634854771784
('Spanish', 'Ecuador') 0.5524861878453039
('Spanish', 'Mexico') 0.4613003095975232
('Spanish', 'Spain') 0.6792452830188679
('Spanish', 'Uruguay') 0.4158730158730159
('Sundanese', 'Indonesia') 0.485
('Swahili', 'Kenya') 0.663003663003663
('Tamil', 'India') 0.6261682242990654
('Telugu', 'India') 0.59
('Urdu', 'India') 0.6818181818181818
('Urdu', 'Pakistan') 0.6435185185185185
Brands / products / companies 0.5440494590417311
Cooking and food 0.5332606324972737
Geography / buildings / landmarks 0.5277628032345013
Objects / materials / clothing 0.570528967254408
People and everyday life 0.637065637065637
Plants and animal 0.6130500758725341
Public Figure and pop culture 0.514505119453925
Sports and recreation 0.5176715176715176
Traditions / art / history 0.5396725440806045
Vehicles and Transportation 0.5855379188712522

---

Hi @timothycdc. This is the result of my evaluation using the latest code, the evaluation command is torchrun --nproc-per-node=2 run.py --model llava_v1.5_7b --data CVQA_EN --verbose. If the result is fine, I will merge this pull request within 2 days.

[timothycdc]

Thanks for this, the results look much better. You can merge it.