Value Drifts: Tracing Value Alignment During LLM Post-Training

Code and data for the TACL 2026 paper "Value Drifts: Tracing Value Alignment During LLM Post-Training" by Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, and Siva Reddy.

TL;DR

A model's values are largely decided during SFT — not preference optimization.

Instead of only probing fully post-trained models, we trace when and how a model acquires its values across post-training. Studying Llama-3 and Qwen-3 and disentangling the roles of the algorithm and the data, we find:

• ① SFT sets the values. A model's value alignment is locked in during supervised fine-tuning, often very early in training.
• ② Preference optimization barely moves them. PPO, DPO, and SimPO rarely move the values SFT already set. The reason: standard preference data has near-identical value distributions across preference pairs, a minimal value gap for the optimizer to exploit.
• ③ But the algorithm does matter — once there's a gap. When we synthetically inject a controlled value gap between the preference pairs, the three algorithms diverge sharply, producing markedly different alignment outcomes on the same data.

Repository structure

value-drifts/
├── configs/
│   └── ds_config.json          # DeepSpeed ZeRO-2 config for SFT
├── data/
│   ├── README.md               # dataset docs (HF configs, columns, polarity)
│   ├── load_data.py            # helpers around load_dataset(...)
│   └── vprism_questions.csv     # local copy of the vPRISM eval prompts
├── training/
│   ├── sft/                    # supervised fine-tuning (run_sft.py / .sh)
│   ├── dpo/                    
│   ├── simpo/                  
│   └── ppo/                    # reward modeling + PPO
│       ├── reward_modeling.py  
│       ├── run_ppo.py
│       └── *.sh
└── inference/
    ├── run_batch_base.py       # base (pre-SFT) model, "Response:" continuation
    ├── run_batch_sft.py        # generate k=5 responses per checkpoint over vPRISM
    ├── run_batch_dpo.py
    ├── run_batch_simpo.py
    ├── run_batch_ppo.py
    └── prompts.py              # eval / synthetic-gen prompts

Installation

pip install -r requirements.txt
# flash-attn must be installed after torch:
pip install flash-attn --no-build-isolation

The paper's runs used CUDA 11.8, torch==2.7.1 (cu118), accelerate==1.8.1, and vllm==0.7.3. Training uses flash_attention_2 and (for SFT) DeepSpeed ZeRO-2.

Data

HuggingFace: McGill-NLP/value-drifts.

from datasets import load_dataset

vprism = load_dataset("McGill-NLP/value-drifts", "vprism", split="train")               # 550 eval prompts
prefs  = load_dataset("McGill-NLP/value-drifts", "synthetic_preference_data", split="train")  # 13,443 pairs

See data/README.md for column descriptions and an important note on the chosen/rejected polarity swap the training scripts perform. The training and inference scripts load from the Hub by default.

Pipeline

The experiments follow three stages: SFT → preference optimization → evaluation. The .sh files are SLURM examples — edit the placeholder path variables (SFT_CKPT, OUTPUT_DIR, ACCEL_CONFIG, …) at the top of each for your cluster. Set WANDB_ENTITY in your environment to enable Weights & Biases logging.

1. Supervised fine-tuning

cd training/sft
sbatch run_sft.sh          # or: accelerate launch run_sft.py

Model, dataset (wildchat / alpaca), and hyperparameters are configured at the bottom of run_sft.py.

2. Preference optimization

All three consume synthetic_preference_data and start from an SFT checkpoint (--model_name_or_path).

# DPO
cd training/dpo   && sbatch run_dpo.sh

# SimPO
cd training/simpo && sbatch run_simpo.sh

# PPO (train a reward model first, then run PPO against it)
cd training/ppo
sbatch run_reward.sh       # -> reward model checkpoint
sbatch run_ppo.sh          # uses --reward_model_path + --sft_model_path

3. Batch inference over vPRISM

Generate k=5 responses for every checkpoint in a run, then evaluate value alignment on the generations.

cd inference

# Base model (before SFT)
python run_batch_base.py \
    --model-path meta-llama/Llama-3.1-8B \
    --output-csv ./prism_responses/base/prism_base.csv

# A post-trained run (one CSV per checkpoint)
python run_batch_sft.py \
    --checkpoints-dir /path/to/llama3_3b_sft_alpaca \
    --output-dir      ./prism_responses/sft_alpaca

# run_batch_dpo.py / run_batch_simpo.py / run_batch_ppo.py take the same flags.

Each script defaults to the HF vprism config; pass --input-csv data/vprism_questions.csv to use the local copy instead.

Citation

@article{bhatia2025valuedrifts,
  title   = {Value Drifts: Tracing Value Alignment During LLM Post-Training},
  author  = {Bhatia, Mehar and Nayak, Shravan and Kamath, Gaurav and
             Mosbach, Marius and Sta\'nczak, Karolina and Shwartz, Vered and
             Reddy, Siva},
  journal = {Transactions of the Association for Computational Linguistics (TACL)},
  year    = {2026},
  url     = {https://arxiv.org/abs/2510.26707}
}