This repository provides a unified framework for evaluating Question Answering (QA) systems using multiple evaluation metrics and LLM-based judges.
The framework supports traditional lexical metrics, semantic similarity metrics, and LLM-based evaluation methods.
- BLEU
- Google BLEU
- SacreBLEU
- ROUGE-1
- ROUGE-2
- ROUGE-L
- METEOR
- Exact Match
- F1
- BERTScore
- Sentence Transformer similarity
- LLM Score Judge
- LLM Pairwise Judge
Python 3.9+
Recommended installation:
pip install -r requirements.txtTypical dependencies include:
pandas
numpy
transformers
sentence-transformers
sacrebleu
bert-score
tabulate
Additional dependencies may be required depending on the selected judge.
The main script is run.py.
You must specify the judge type and the dataset file path.
python run.py <judge_type> <file_path>Example:
python run.py bleu datasets/wiki-1c-1t.jsonl| Argument | Description |
|---|---|
judge_type |
Evaluation method to use |
file_path |
Path to dataset file |
| Argument | Description |
|---|---|
--evaluate_qa |
Enables QA evaluation mode |
--include_question |
Includes the question in the evaluation context |
--model |
LLM model used by LLM-based judges |
--prompt |
Prompt template ID used by LLM judges |
--v70 |
Use Llama v70 configuration |
--metric_score |
Returns the raw metric score instead of accuracy |
--max_queries |
Maximum number of LLM queries |
bleu
exact_match
f1
meteor
rouge_1
rouge_2
rouge_L
sacrebleu
bertscore
google_bleu
sentence_transformer
llm_score
llm_pairwise
Some evaluation metrics require separate execution scripts and environments due to dependency conflicts between libraries. Certain metrics rely on specific versions of frameworks (e.g., TensorFlow vs. PyTorch, or incompatible transformers versions), which makes running them in the same Python environment impractical.
To avoid these conflicts, the following metrics are executed using dedicated scripts and isolated environments:
run_bartscore.pyrun_bleurt.pyrun_discoscore.py
Each of these scripts should be executed within its corresponding environment, where the required dependencies are installed.
Example:
python run_bartscore.py <file_path>These scripts follow the same general evaluation philosophy as run.py, but are separated to maintain compatibility with their respective library requirements.
MIT License