Authors: Emre Kılıç, Kristina Trajkovski, Bilge Kaan Güneyli
The LLM Evaluator. An evaluator model which analyses and scores an LMM in terms of their performance in the summarization task.
We use SummEval project in our tests. It uses the CNN and Daily Mail stories as data sources. The summaries of the stories are created by different LLMs and annotated by human evaluators. To recreate the dataset, follow the instructions:
- Download CNN Stories and Daily Mail Stories from here.
- Create a cnndm directory and unpack downloaded files into the directory.
- Download the model outputs and human annotations from here and add the json file into the cnndm directory.
- Put the cnndm directory into the datasets directory.
We use Ollama to form our evaluation models. To recreate the EvalLM models, you must install Ollama. You'll be able to do so from here.
We also use the Ollama's python package to run the tests. You can install it by running:
pip install ollamaWe store the descriptions of the models in the models directory. To recreate the models, you can run the following command:
bash models/<version>/model-compose.shAfter recreating the dataset and composing a model, you can run the model's test to ensure everything goes well. To run the test of a model, you can use the following command:
python3 models/<version>/model_test.pyTBW