Generating Italian Poems with GePpeTto

Fine-tuning the Italian GPT-2 model GePpeTto to generate poems conditioned on author and century of production.

---

Project structure

src/
├── archive/
│   └── poems.json              # Wikisource dataset (cleaned)
├── poems_manager/
│   ├── poem.py                     # Poem dataclass, PoemText, token helpers
│   ├── poem_collection.py          # I/O, preprocessing, train/val/test split
│   ├── poem_dataset.py             # Torch Dataset with inverse-frequency sampling
│   ├── poem_similarity.py          # Cosine similarity + heatmap (style analysis)
│   ├── year_parser.py              # Parses raw year strings to integers
│   └── utils.py                    # PoemFields enum
├── fine_tuning/
│   ├── fine_tuner.py               # Training (RepetitionPenaltyTrainer)
│   └── evaluator.py                # BERTScore evaluation + per-poem heatmaps
├── evaluation/
│   ├── cosine_similarity/
│   │   ├── heatmap_utils.py            # Shared utilities (generation, centroids, plotting)
│   │   ├── heatmap_eval.py             # Style-attribution heatmaps for top-20 poets (28×8, one PNG per poet)
│   │   └── heatmap_eval_squared.py     # Square 28×28 heatmap (author-only + century-only queries)
│   └── bert_score/
│       ├── bert_score_utils.py         # Generation, BERTScore, saving, plotting utilities
│       └── bert_score_eval.py          # Standalone evaluation script
├── start_finetuning.py             # Entry point: preprocessing → training → evaluation
├── main.py                         # Interactive generation / CLI
└── resolve_years_manually.py       # One-shot script to clean year data in the JSON

data/                               # Generated by preprocessing
├── train.txt
├── val.txt
├── test.txt
├── train.json                      # Same splits in JSON format (author/year/title/text)
├── val.json
└── test.json

model_output/                       # Generated by training
├── config.json
├── model.safetensors
├── checkpoint-*/                   # Intermediate checkpoints
└── tokenizer files...

results/
├── heatmaps/                       # Generated by heatmap_eval*.py
│   ├── heatmap_{Autore}.png        # One PNG per poet (heatmap_eval.py)
│   └── heatmap_squared.png         # Single 28×28 heatmap (heatmap_eval_squared.py)
└── bert_score/                     # Generated by bert_score_eval.py
    ├── scores.csv
    ├── summary.json
    ├── best_worst/
    └── plots/

---

Installation

uv sync

---

Quick start

# 1. Preprocess + fine-tune + evaluate in one shot
uv run python ./src/start_finetuning.py

# 2. List available author and century tokens
uv run python src/main.py --list-tokens --poems-json ./src/archive/poems.json

# 3. Generate poems
uv run python src/main.py --author "<Giacomo_Leopardi>" --century 19 --num 3

---

Pipeline

0. Dataset preparation (one-shot)

Before training, clean the raw JSON to fix missing/unparseable years and discard authors with too few poems:

uv run python src/resolve_years_manually.py

Edit EXTRA_DISCARD and YEAR_OVERRIDES at the top of the file to customise. The script saves a cleaned new_poems.json in the same directory as the source.

NOTE: src/archive/poems.json is the already-cleaned version.

---

1. Preprocessing and fine-tuning

Run the full pipeline (preprocessing → training → evaluation) with the same parameters used on the SLURM cluster:

uv run python ./src/start_finetuning.py \
    --dataset              ./src/archive/poems.json \
    --output-dir           ./data \
    --min-poems-per-author 10 \
    --max-poems-per-author 400 \
    --target-size          20000 \
    --seed                 45

Argument	Default	Description
--dataset	./src/archive/poems.json	Path to source poems JSON
--output-dir	./data	Where to write the dataset splits
--val-ratio	0.1	Fraction of poems used for validation
--test-ratio	0.1	Fraction of poems used for testing
--min-poems-per-author	10	Authors below this threshold are excluded
--max-poems-per-author	400	Authors above this threshold are downsampled
--max-poems	None	Hard cap on total poems (no limit by default)
--target-size	None	Training examples to sample (default: full train set)
--seed	42	Random seed for reproducibility

Training poems are sampled with inverse-frequency weighting: rare authors are upsampled relative to prolific ones so no single author dominates the training signal.

The trainer adds a soft repetition penalty to the standard cross-entropy loss (controlled by rep_penalty_weight in fine_tuner.py, default 0.1), which mirrors the hard repetition_penalty applied at inference time.

Each poem is formatted as:

<POEM>
<Giacomo_Leopardi> <19th_century>
TITLE: L'infinito

TEXT:
<STANZA>
Sempre caro mi fu quest'ermo colle,
...
</STANZA>
</POEM>

On the SLURM cluster:

sbatch ./geppetto.slurm

Notes for CPU:

• Training is slow; for a quick test use --max-poems 200 and set EPOCHS = 1 in fine_tuner.py
• The fine-tuned model is saved to model_output/

---

2. BERTScore evaluation

src/evaluation/bert_score/bert_score_eval.py evaluates the fine-tuned model against the test split and saves full results to results/bert_score/.

For each test poem: builds the prompt (author + century + title), generates the body, then computes BERTScore (Italian BERT) against the original.

Output:

File	Content
scores.csv	Per-poem precision / recall / F1
summary.json	Averages: global, by author, by century
best_worst/	Top-3 and bottom-3 poems per author (text comparison)
plots/boxplot_f1_by_author.png	F1 distribution per author
plots/boxplot_f1_by_century.png	F1 distribution per century
plots/bar_avg_f1_by_author.png	Average F1 per author, sorted
plots/scatter_precision_recall.png	Precision vs Recall per poem
plots/heatmap_f1_author_century.png	Avg F1 grid (author × century)
plots/histogram_f1.png	Global F1 distribution

Evaluation is restricted to the top-20 most prolific authors by default, consistent with the heatmap evaluation scope.

Argument	Default	Description
--dataset	./src/archive/poems.json	Path to source poems JSON (used for author ranking)
--test-dataset	./data/test.json	Path to the pre-saved test split JSON
--batch-size	32	Generation batch size
--top-n	3	Best/worst poems to save per author
--top-authors	20	Restrict evaluation to the N most prolific authors
--seed	42	Random seed

uv run python -m src.evaluation.bert_score.bert_score_eval
# On the cluster:
sbatch ./bertscore.slurm

---

3. Style-attribution heatmaps

Two scripts evaluate whether the model has correctly learned author style and century conditioning, both using paraphrase-multilingual-MiniLM-L12-v2 embeddings.

---

3a. Per-poet heatmaps — heatmap_eval.py

For each of the top-20 most prolific poets, generates 10 poems conditioned on (poet, century) for every century, then measures cosine similarity against all author and century centroids.

Layout — one PNG per source poet (28 rows × 8 cols):

Columns : all unique centuries  →  10 poems generated as (source_poet, C)

Rows    : ┬ section 1 — 20 author centroids
          ├─── dividing line ───
          └ section 2 — 8 century centroids

uv run python -m src.evaluation.cosine_similarity.heatmap_eval [--reload] [--n-poems N]
# Output: results/heatmaps/heatmap_{Autore}.png  (20 files)

---

3b. Square heatmap — heatmap_eval_squared.py

Produces a single 28×28 matrix using single-tag queries (author-only or century-only), separating the two conditioning signals cleanly.

Layout:

Rows    (28) : 20 author centroids  +  8 century centroids  (reference corpus)
Columns (28) : 20 author queries    +  8 century queries    (generated poems)

Query prompts:
  author  query j : <POEM>\n<Author_j>\nTITLE:
  century query j : <POEM>\n<Century_j>\nTITLE:

cell[i, j] = cos_sim(avg_embedding(poems from query j), centroid i)

Divider lines: horizontal at row 20, vertical at column 20

uv run python -m src.evaluation.cosine_similarity.heatmap_eval_squared [--reload] [--n-poems N]
# Output: results/heatmaps/heatmap_squared.png

Both scripts accept:

Argument	Default	Description
--reload	false	Reuse precomputed centroids from src/archive/centroids/centroids.pkl
--n-poems	50	Number of poems generated per query; shown in the PNG title

The --n-poems value is set via the N_POEMS variable in heatmaps.slurm.

On the SLURM cluster both scripts run sequentially via heatmaps.slurm:

sbatch ./heatmaps.slurm

---

4. Generation

Before generating, list the available tokens to find valid author/century values:

uv run python src/main.py --list-tokens --poems-json ./src/archive/poems.json

Interactive mode (no arguments):

uv run python src/main.py

CLI mode:

# Generate 3 poems by Leopardi
uv run python src/main.py --author "<Giacomo_Leopardi>" --century 19 --num 3

# Generate a titled poem by Dante
uv run python src/main.py --author "<Dante_Alighieri>" --century 14 --title "Nuova canzone"

# Use the original GePpeTto without fine-tuning
uv run python src/main.py --base-model

# Use a specific checkpoint
uv run python src/main.py --author "<Giacomo_Leopardi>" --century 19
# (edit FINETUNED_DIR in main.py to point to e.g. model_output/checkpoint-5000)

Argument	Default	Description
--author	""	Author token (e.g. <Giacomo_Leopardi>)
--century	0	Century as integer (e.g. 19)
--title	""	Optional poem title
--num	1	Number of poems to generate
--max-tokens	200	Maximum generation length
--base-model	false	Use original GePpeTto instead of fine-tuned
--poems-json	""	Path to poems JSON (required for --list-tokens)
--list-tokens	false	Print all available tokens and exit

Generation parameters (temperature, top_p, repetition_penalty, etc.) are defined in GENERATION_DEFAULTS at the top of src/main.py.