REPRODUCTION_NOTES.md

SPV-MIA Reproduction Notes

Paper: "SPV-MIA: Membership Inference Attack with Self-Prompted Variation" (NeurIPS 2024)

---

Successful Run

Date: 2026-05-02T10:39:10
Hardware: Apple Silicon Mac (ALFA-Mac), 192GB unified RAM, MPS backend, Python 3.9
Result file: results/results_gpt2_wikitext_20260502_103910.json

Results

Metric	Our Run	Paper (Table 1/2)
AUC	0.9771	0.975
ASR	0.933	—
TPR@1%FPR	0.906	0.673
TPR@5%FPR	0.930	—

---

Exact Configuration

Model & Dataset

Parameter	Value
Model	gpt2 (base, 124M params)
Dataset	wikitext / wikitext-103-raw-v1
Block size	128 tokens
Packing	True

Target Model Fine-tuning

Parameter	Value
Checkpoint	ft_llms/gpt2/wikitext-103-raw-v1/target/checkpoint-31250
PEFT	Full fine-tuning (--disable_peft)
Epochs	10
Batch size	16
Learning rate	1e-4
Optimizer	AdamW
Training samples	0 – 50,000
Eval samples	0 – 5,000
Gradient accumulation steps	1

Reference Model Fine-tuning

Parameter	Value
Checkpoint	ft_llms/gpt2/wikitext-103-raw-v1/refer/checkpoint-2500
PEFT	LoRA (default)
Epochs	4
Batch size	16
Learning rate	5e-5
Training data	Self-prompted data generated by target model (samples 10,000–20,000)
Training samples	0 – 50,000
Eval samples	0 – 5,000

Attack (from configs/config.yaml)

Parameter	Value
Attack kind	stat
Maximum samples	1,000 members + 1,000 non-members
Perturbation number	1
Sample number	10
Span length	2
Mask pct	0.3
Buffer size	1
Mask top-p	1.0
Mask filling model	t5-base (forced to CPU)
Calibration	True
Random seed	48
Eval batch size	1

---

Key Deviations from Paper

1. Full fine-tuning instead of LoRA for target model. The paper states LoRA as default PEFT. In our experiments, LoRA on GPT-2 base produced AUC ≈ 0.52 (random) because the memorization signal was indistinguishable from the reference model. Full fine-tuning produced AUC = 0.977, matching the paper's reported numbers.

2. T5 mask-filling runs on CPU. On MPS (Apple GPU), T5 generation produced empty fills, breaking the perturbation step. Moving T5 to CPU fixed this. This is a Mac-specific workaround and would not apply on NVIDIA hardware.

3. 50,000 training samples. The paper does not specify exact training data size. 10,000 samples with LoRA was insufficient; 50,000 with full fine-tuning produced results matching the paper.

---

Failed Configurations

Run	Model	PEFT	Dataset	Train Samples	AUC	Root Cause
1	gpt2-xl	Full fine-tune	AG News	10,000	1.000	Model too large, trivial memorization
2	gpt2 base	LoRA	Wikitext-103	10,000	0.536	LoRA memorization too weak
3	gpt2 base	LoRA	Wikitext-103	50,000	0.518	Same — data size not the cause

---

How to Reproduce from Scratch

# On ALFA-Mac
cd ~/SPV-MIA
tmux new -s train
bash _run_pipeline.sh

The pipeline runs all 5 steps automatically:

1. Fine-tune target model (gpt2, full fine-tuning, 10 epochs, 50k samples)
2. Generate self-prompt reference data from target model
3. Fine-tune reference model (LoRA, 4 epochs, 50k samples)
4. Update configs/config.yaml with checkpoint paths
5. Run SPV-MIA attack and print AUC

Expected runtime: ~10–12 hours on Apple Silicon 192GB.

Save results after completion:

python3 save_results.py

Results saved to results/results_gpt2_wikitext_<timestamp>.json.