Hi guys, first of all, congrats for the beautiful dataset!! 馃憦 馃憦
I have a question about the dataset for tasks 1 and 2 (crag_task_1_and_2_dev_v4.jsonl.bz2): we are currently using the dataset to evaluate our RAG system, using our own evaluation pipeline. Now, for task 1 and 2 "5 pages are randomly selected from the top-10 pages. These pages are likely relevant to the question, but relevance is not guaranteed.".
The question is then: how can we use the gold-standard answer for the crag_task_1_and_2_dev_v4.jsonl.bz2 dataset, as we don't know if the 5 randomly selected pages contain the right context from which the gold-standard answer has been obtained?
Perhaps I misunderstood something, but since the url of the web page used to generate the gold-standard answer is not included in the crag_task_1_and_2_dev_v4.jsonl.bz2 file, we cannot infer whether a question can/cannot be correctly answered using the 5 web pages.
Thanks for any hint!
Hi guys, first of all, congrats for the beautiful dataset!! 馃憦 馃憦
I have a question about the dataset for tasks 1 and 2 (
crag_task_1_and_2_dev_v4.jsonl.bz2): we are currently using the dataset to evaluate our RAG system, using our own evaluation pipeline. Now, for task 1 and 2 "5 pages are randomly selected from the top-10 pages. These pages are likely relevant to the question, but relevance is not guaranteed.".The question is then: how can we use the gold-standard answer for the
crag_task_1_and_2_dev_v4.jsonl.bz2dataset, as we don't know if the 5 randomly selected pages contain the right context from which the gold-standard answer has been obtained?Perhaps I misunderstood something, but since the url of the web page used to generate the gold-standard answer is not included in the
crag_task_1_and_2_dev_v4.jsonl.bz2file, we cannot infer whether a question can/cannot be correctly answered using the 5 web pages.Thanks for any hint!