llm-eval-framework

Simple framework for evaluating LLM outputs against reference answers. Supports multiple models, providers and evaluation metrics. Designed to be extensible and easy to use for various NLP tasks.

Installation

Option 1: with uv (recommended)

uv sync

Option 2: editable install with pip

python -m venv .venv
source .venv/bin/activate
pip install -e .

Dataset format

Each sample in JSON file should include input as dictionary and a reference answer:

[
	{
		"input": {
			"question": "What is the capital of France?",
			"context": "France is a country in Europe. Its capital is Paris."
		},
		"reference": "Paris"
	}
]

Repository layout

llm_eval/
	core/           # Data structures (sample, prediction, output, results)
	datasets/       # Dataset loading
	evaluation/     # Evaluator
	metrics/        # All metrics used for evaluation
	models/         # Models supported by the framework
	preprocessing/  # Text normalization utilities
	tasks/          # Task-specific evaluation logic (QA, summarization, etc.)