A multilingual trustworthiness evaluation framework for comparing direct generation with a modular Generator → Verifier → Reviser pipeline on parallel XQuAD question-answering data.
The project evaluates not only answer accuracy, but also evidence grounding, explanation faithfulness, multilingual performance disparity, and cross-lingual consistency using a single instruction-tuned language model.
Multilingual language models can achieve strong average performance while behaving inconsistently across languages or producing explanations that are insufficiently grounded in the provided context.
This project evaluates these issues using parallel XQuAD examples in five languages:
- English (
en) - Arabic (
ar) - German (
de) - Spanish (
es) - Turkish (
tr)
Two inference strategies are compared:
The model receives a context and question and directly produces:
- Answer
- Explanation
- Supporting evidence
- Confidence score
The same initial answer is passed through an explicit verification workflow:
Question + Context
│
▼
Generator
│
▼
Verifier
/ \
Supported Unsupported
│ │
│ ▼
│ Reviser
│ │
└──────┬──────┘
▼
Final Answer
The verifier determines whether the proposed answer is sufficiently supported by the context. Unsupported answers are sent to the reviser for correction.
The same underlying model is used throughout the pipeline to keep the comparison controlled.
Language Model
Qwen/Qwen2.5-0.5B-Instruct
Dataset
google/xquad
XQuAD contains parallel translations of question-answering examples across multiple languages. The evaluation keeps question IDs shared across all selected languages, enabling controlled cross-lingual comparison.
The experiments use:
- 150 aligned question IDs
- 5 languages
- 750 language-question pairs
- Exact Match — normalized exact agreement with the gold answer
- Token F1 — token-level overlap with the gold answer
- Fuzzy Match — approximate normalized string similarity
- Evidence-in-Context — whether generated evidence approximately appears in the source context
- Answer-in-Explanation — whether the generated explanation explicitly contains the predicted answer
- Faithfulness — combined evidence and explanation grounding score
- Per-language performance
- English vs. non-English performance gap
- Best- vs. worst-language gap
- Standard deviation across languages
- Worst-language performance
- Cross-lingual consistency across parallel questions
| Metric | Direct | Modular |
|---|---|---|
| Exact Match | 0.355 | 0.371 |
| Token F1 | 0.454 | 0.471 |
| Fuzzy Match | 0.568 | 0.585 |
| Faithfulness | 0.674 | 0.767 |
| Evidence Rate | 0.824 | 0.848 |
| Answer-in-Explanation | 0.524 | 0.685 |
The modular pipeline produces modest improvements in answer quality, while the strongest gains appear in trustworthiness-related metrics.
Faithfulness increases from 0.674 to 0.767, and Answer-in-Explanation increases from 0.524 to 0.685.
Performance differs substantially across languages.
English is generally the strongest language, while Turkish is the weakest language for several answer-quality metrics.
In the modular configuration:
- English Exact Match: 0.493
- Turkish Exact Match: 0.280
The effect of modular verification is not uniform across languages. Exact Match improves for Arabic, English, Spanish, and Turkish, while German decreases slightly.
The English vs. non-English Exact Match gap changes from:
Direct: 0.157
Modular: 0.153
Evidence-grounding disparity decreases more noticeably:
Direct: 0.112
Modular: 0.090
These results suggest that modular verification can reduce some forms of multilingual grounding disparity, although substantial language-dependent differences remain.
Because the same semantic questions appear across multiple languages, the project also evaluates whether correctness remains stable across translations.
| Mode | Mean Consistency | Fully Consistent Rate |
|---|---|---|
| Direct | 0.816 | 0.360 |
| Modular | 0.805 | 0.347 |
The modular pipeline improves several aggregate quality and faithfulness metrics but does not improve cross-lingual consistency.
This distinction is important: better average performance does not necessarily imply more stable behavior across languages.
multilingual-llm-trust-evaluation/
├── assets/
│ ├── 01_token_f1_by_language_and_mode.png
│ ├── 02_faithfulness_by_language_and_mode.png
│ ├── 03_language_disparity_best_vs_worst_gap.png
│ ├── 04_language_disparity_standard_deviation.png
│ └── 05_cross_lingual_consistency_by_mode.png
├── multilingual_evaluation/
│ └── multilingual_trust_evaluation.ipynb
├── reports/
│ └── final_report.md
├── results/
│ ├── 01_direct_results.csv
│ ├── 02_modular_results.csv
│ ├── 03_combined_results.csv
│ ├── 04_aggregate_results.csv
│ ├── 05_aggregate_by_mode.csv
│ ├── 06_language_fairness_metrics.csv
│ ├── 07_best_worst_language_gap_compact.csv
│ ├── 08_standard_deviation_across_languages.csv
│ ├── 09_worst_language_compact.csv
│ ├── 10_cross_lingual_consistency_by_question_id.csv
│ └── 11_cross_lingual_consistency_summary.csv
├── .gitignore
├── README.md
└── requirements.txt
Clone the repository and install the dependencies:
git clone https://github.com/Hamidreza-Talei/multilingual-llm-trust-evaluation.git
cd multilingual-llm-trust-evaluation
pip install -r requirements.txtA CUDA-capable GPU is recommended for faster inference.
Open:
multilingual_evaluation/multilingual_trust_evaluation.ipynb
and execute the notebook sequentially.
The workflow:
- Loads aligned multilingual XQuAD examples.
- Loads Qwen2.5-0.5B-Instruct.
- Runs the direct QA baseline.
- Runs the Generator–Verifier–Reviser pipeline.
- Computes answer-quality and faithfulness metrics.
- Aggregates results by language and inference mode.
- Measures multilingual performance disparity.
- Computes cross-lingual consistency.
- Generates evaluation visualizations.
- Exports evaluation artifacts to
results/.
The experiments highlight several important properties of multilingual LLM evaluation:
- Modular verification improves overall answer quality, but the gains are relatively modest.
- Improvements in faithfulness and explanation grounding are considerably stronger than improvements in accuracy.
- English remains substantially stronger than several non-English languages.
- Turkish is particularly challenging for answer-quality metrics.
- Modular verification reduces some grounding-related language disparities.
- Improvements are not uniform across every language and metric.
- Better aggregate performance does not guarantee better cross-lingual consistency.
- Using the same model as generator, verifier, and reviser limits verifier independence.
Overall, the results demonstrate why multilingual LLM evaluation should consider more than average accuracy alone.
Several limitations should be considered when interpreting the results:
- The same model acts as generator, verifier, and reviser, creating a risk of self-confirmation.
- Qwen2.5-0.5B-Instruct is relatively small and may have limited multilingual reasoning and verification capacity.
- The faithfulness metrics are automatic approximations rather than complete semantic-grounding measures.
- The evaluation covers five languages rather than the full multilingual XQuAD collection.
- The modular architecture requires additional model calls and therefore increases inference cost.
- The experiments rely on automatic evaluation without human assessment of explanation quality.
Potential extensions include independent verifier models, larger multilingual models, additional languages, semantic faithfulness metrics, human evaluation, statistical significance testing, and component-level ablation studies.
A more detailed discussion of the experimental results, multilingual disparities, limitations, and possible extensions is available in:
-
XQuAD — Cross-lingual Question Answering Dataset
https://github.com/google-deepmind/xquad -
XQuAD on Hugging Face
https://huggingface.co/datasets/google/xquad -
Qwen2.5-0.5B-Instruct
https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct -
HELM: Holistic Evaluation of Language Models
https://arxiv.org/abs/2211.09110 -
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
https://arxiv.org/abs/2306.11698 -
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
https://arxiv.org/abs/2303.08896
Python · PyTorch · Hugging Face Transformers · Hugging Face Datasets · Qwen2.5 · XQuAD · Pandas · NumPy · RapidFuzz · Matplotlib