LLM Benchmark – Python Exercise Generation

A benchmarking tool that sends pedagogical prompts to multiple LLM providers and collects their responses for comparison. The use case is generating Python programming exercises for beginner students.

What it does

1. Loads a system prompt and a user prompt from the prompts/ directory
2. Sends both to each configured model across all providers
3. Saves each response as a text file under responses/
4. Appends timing and word-count statistics to statistics.txt

Providers

Runner	Models
OpenAI	gpt-5.4, gpt-5.3-codex
Anthropic	claude-opus-4-7, claude-sonnet-4-6
PublicAI (Swiss AI)	apertus-70b-instruct, apertus-8b-instruct
Infomaniak	mistral3 (Mistral-Small-3.2-24B)
Ollama (local)	mistral 7B

Project structure

prompts/
  system_prompt.txt          # Meta-prompt defining the exercise format
  user_prompt_1.txt          # User prompt variant 1
  user_prompt_2.txt          # User prompt variant 2
  user_prompt_3.txt          # User prompt variant 3
responses/
  user_prompt_<id>/
    <runner>/
      <model>/
        <model>_response_1.txt
        ...
statistics.txt               # ElapsedTime and word count per run
config.py                    # API keys

Setup

1. Create a virtual environment with mamba

mamba create -n didinfo python=3.11
mamba activate didinfo

If mamba is not installed, you can use conda instead (conda create ..., conda activate ...), or a standard Python virtual environment:

python -m venv .venv
source .venv/bin/activate   # macOS/Linux
.venv\Scripts\activate      # Windows

2. Install Python dependencies

pip install -r requirements.txt

3. Install Ollama (for local models)

1. Download and install Ollama from ollama.com
2. Start the Ollama service
3. Pull the required models:

ollama pull mistral

Set your API keys in config.py:

api_keys = {
    'OPENAI_API_KEY': '...',
    'ANTHROPIC_API_KEY': '...',
    'INFOMANIAK_API_KEY': '...',
    'PUBLICAI_API_KEY': '...',
}

Usage

Run all providers with a given prompt index (default: 1):

python run.py -p 1

Run a single provider:

python run_anthropic.py -p 2
python run_openai.py -p 3

Each run appends a new numbered response file, preserving previous runs.

Prompts

The system prompt defines a pedagogical framework for generating Python function exercises for beginners. It specifies fixed constraints (no loops, no conditions, no imports) and variation parameters (scaffolding level, pedagogical progression, context/theme).

Each exercise is structured in four sections: Consignes (instructions), Code de base (starter code), Solution, Explication.