IBERtaQA

This repository contains the official configuration files to evaluate large language models (LLMs) on the IBERtaQA benchmark using EleutherAI's LM Evaluation Harness.

IBERtaQA is a parallel, human-curated multilingual benchmark designed to evaluate cultural and factual knowledge across multiple Iberian languages and cultural communities. It covers 6 cultural regions (global, plus 5 local territories) across 5 parallel languages (Basque, Catalan, Galician, English, Spanish, and Valencian).

For more information on IBERtaQA, please refer to the Hugging Face dataset card: https://huggingface.co/datasets/HiTZ/IBERtaQA

---

Prerequisites

1. To use these configurations, you must install the LM Evaluation Harness framework (see its own prerequisites and dependencies too!):

   git clone https://github.com/EleutherAI/lm-evaluation-harness
   cd lm-evaluation-harness
   pip install -e .

2. Clone this repository or just download the ibertaqa folder.

Setup

To evaluate a model using IBERtaQA tasks, the framework needs to know where to find the configuration files located inside the ibertaqa folder of this repo. You can integrate them in one of two ways (both are fine!):

Option 1: Direct integration

Copy or symlink the ibertaqa folder directly into your local installation of the framework:

cp -r ibertaqa/ /path/to/lm-evaluation-harness/lm_eval/tasks/

When using this method, you can call tasks natively without any extra flags.

Option 2: Using the include path flag

Keep this repository separate and explicitly tell the evaluation engine where to look using the --include_path flag point to the ibertaqa folder:

--include_path /path/to/ibertaqa/

This option does not require you to do anything now; we'll see examples of how to run evaluations in the next section.

---

How to run evaluations

The setup is highly granular, allowing you to run individual subsets, group evaluations by language, group evaluations by cultural territory, or evaluate the entire benchmark at once.

Notice that we provide minimal examples for each of these options. Please refer to the LM Evaluation Harness documentation for more information on how to run evaluations.

1. Evaluate the entire benchmark

To evaluate the model across every single language and cultural subset simultaneously using the master group alias:

# If using Option 1 (Direct Integration):
lm_eval --model hf \
    --model_args pretrained=your-model-hf-path \
    --tasks ibertaqa

# If using Option 2 (Include Path):
lm_eval --model hf \
    --model_args pretrained=your-model-hf-path \
    --tasks ibertaqa \
    --include_path /path/to/ibertaqa/

2. Grouped by language

Isolate the evaluation to see how a model performs in a specific linguistic context using the language-specific group tags (e.g., ibertaqa_eu, ibertaqa_gl, ibertaqa_ca, ibertaqa_va, ibertaqa_es, ibertaqa_en):

# Evaluate all cultural subsets exclusively in Galician (gl)
lm_eval --model hf \
    --model_args pretrained=your-model-hf-path \
    --tasks ibertaqa_gl

3. Grouped by cultural / origin set

Evaluate a model's knowledge regarding a specific community across all its available translations using the territory group keys (e.g., ibertaqa_local-ca, ibertaqa_local-eu, etc.):

# Evaluate the Valencian Community cultural subset across all parallel languages
lm_eval --model hf \
    --model_args pretrained=your-model-hf-path \
    --tasks ibertaqa_local-va

4. Run an individual subset

For precise, fine-grained benchmarking, target a single cultural dataset configuration paired with a specific language split using the standard naming pattern ibertaqa_[cultural-set]_[language]:

# Evaluate only the global knowledge subset translated into Catalan
lm_eval --model hf \
    --model_args pretrained=your-model-hf-path \
    --tasks ibertaqa_global_ca

---

Authors

The benchmark dataset and configuration profiles were curated and developed by the following institutions:

• HiTZ Center, University of the Basque Country (EHU)

  • Contact: naiara.perez@ehu.eus

• Barcelona Supercomputing Center (BSC)

  • Contact: paula.rivera@bsc.es

• CiTIUS, University of Santiago de Compostela (USC)

  • Contact: silvia.paniagua.suarez@usc.e

• GPLSI, University of Alicante (UA)

  • Contact: ivan.martinezmurillo@ua.es

Funding

This work has been supported and funded by the Ministerio para la Transformación Digital y de la Función Pública and the Plan de Recuperación, Transformación y Resiliencia – funded by the EU through NextGenerationEU within the project Modelos de Lenguaje ALIA.