Multi-seed training variance: are gains robust to LoRA initialization seed?

[sebasmos]

Why this matters for the paper

The current plan uses 5 different eval draws (5 seeds × 200 samples from 1K Hard) to estimate evaluation variance. But there is a separate source of variance: the LoRA weight initialization seed. Two runs with identical data but different init seeds can produce models with meaningfully different behavior, especially with small training sets. If gains appear in one init seed but not another, the result is not reliable.

Scope

• Train 3 LoRAs on identical data with different init seeds (seeds 0, 1, 2).
• Evaluate each on the same 5-draw holdout protocol.
• Report std across init seeds alongside std across eval draws.
• If std across init seeds > std across eval draws, the result is init-sensitive and must be reported honestly.

Done when

• 3 LoRAs trained with different init seeds
• Per-seed-init eval scores recorded
• RESULTS.md Section 1 gains a training-variance sub-note
• Paper §4 acknowledges or dismisses init-seed variance with evidence

Effort

Medium (3 training runs, same eval pipeline). Run after main result is confirmed.

Good for

Anyone running the next cluster batch.