Feat: marker-based intervention positions and first-class ReFT DPO trainer

[Rakshitha-Ireddi]

Summary

This PR introduces marker-based intervention targeting and first-class DPO support for ReFT.

Marker-based intervention positions

Users can wrap spans in prompts using:

<> ... >

Markers are stripped before the model input and intervention locations are automatically mapped to the corresponding token indices. Optionally, only the last token of each span can be used.

First-class ReFT DPO

Adds native DPO support for preference training with ReFT interventions.

Components added:

• ReftDPOTrainer
• ReftDPODataCollator
• make_dpo_data_module

trl can be installed via:

pip install pyreft[dpo]

Changes

dataset.py

• _parse_marker_spans
• _span_ranges_to_token_indices
• make_marker_supervised_data_module
• ReftDPODataCollator
• make_dpo_data_module

reft_trainer.py

• ReftDPOTrainer

setup.py

• extras_require={"dpo": ["trl"]}

tests/

• marker parsing tests
• dataset tests
• DPO collator tests

Authors

• Ireddi Rakshitha
• Yashwanth Devavarapu