About the trainer.

[wanghanone]

Hello, may I ask why in the code I see you are using ppotrainer to update the strategy, but in the paper, it says you are using the dpo training method?

[PrasannS]

Hi! Thanks for taking a look. We actually do a hack where we override the code in ppo_trainer to use the DPO loss (among a few other changes). I see in the README (this gets mentioned in the "Training with RLHF" part) the path to those files is outdated, right now I think the correct file is (https://github.com/PrasannS/d2po/blob/main/rlhfutils/rlhfutils/dpoplus_trainer_v2.py). Right now it requires copy-pasting to override trl source ppo_trainer.py(can do an editable TRL source install to make that), but I'm probably going to try to monkey-patch at some point. Sorry for the confusion.