sotopia-lab · robotMonkeyButler · Jul 9, 2025 · Jul 9, 2025
diff --git a/README.md b/README.md
@@ -1,22 +1,69 @@
 ![sotopia-rl](assets/sotopia-rl-title.png)
 
-# sotopia-rl
+<h1 align="center"> Sotopia-RL: Reward Design for Social Intelligence</h1>
 
-## Install
+[![Project Page](https://img.shields.io/badge/Project-Page-green.svg)](https://rl.sotopia.world/)![Paper PDF](https://img.shields.io/badge/Paper-PDF-red.svg)[![huggingface](https://img.shields.io/badge/%F0%9F%A4%97-Model-orange)](https://huggingface.co/ulab-ai/sotopia-rl-qwen-2.5-7B-grpo)[![Python 3.10](https://img.shields.io/badge/python-%E2%89%A53.10-blue)](https://www.python.org/downloads/release/python-3109/)[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://pre-commit.com/) <a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-blue.svg)
 
-Create a conda environment with python 3.10
-```
+
+
+## 📚 Table of Contents
+- [Introduction](#introduction)
+- [Step 0 - Environment Setup](#step-0---️-environment-setup)
+- [Step 1 - Generating LLM Annotations](#step-1---generating-llm-annotations)
+- [Step 2 - Model Training](#step-2---model-training)
+  - [Behavior Cloning (SFT)](#21-behavior-cloning-supervised-fine-tuning-sft)
+  - [Reward Model Training](#22-reward-model-training)
+  - [GRPO Training](#23-grpo-training)
+- [Step 3 - Automatic Evaluation](#step-3---automatic-evaluation)
+
+## Introduction
+
+**Sotopia-RL** is an utterance-level, attribution-based, and multi-dimensional social reward design method, trained using single-turn online RL It achieves state-of-the-art performance on social goal completion tasks in the SOTOPIA benchmark. 
+
+We first attribute episode-level rewards for multi-turn social interactions to in- dividual utterances with LLMs. Then, we construct a combined reward that includes multiple dimensions of rewards besides goal completion, allowing us to regularize the optimization pro- cess for goal completion. These rewards are used to guide the RL training of social agents.
+
+
+
+![sotopia-rl](assets/sotopia_method.pdf)
+
+## Step 0 - 🛠️ Environment Setup
+
+We recommend using `conda` to manage a clean Python environment for `sotopia-rl`.
+
+#### 1. Create and activate the environment
+
+```bash
 conda create -n sotopia-rl python=3.10
 conda activate sotopia-rl
 ```
 
-Then install poetry and use it to install the dependencies. Currently the package is under development so it's recommended to use the `--no-root` flag to avoid installing the package itself.
+#### 2. Install Poetry (Python package manager)
+
+```bash
+curl -sSL https://install.python-poetry.org | python3
+export PATH="$HOME/.local/bin:$PATH"
+```
+#### 3. Environment Variables
+
+##### Set Redis URL
+
+A Redis database needs to be set up to connect to a Redis DB for loading and saving environment/session data and run this repo. For detailed instructions of setting up Redis database, please refer to [this tutorial](https://github.com/sotopia-lab/sotopia-pi/tree/main/data_generate#setting-up-redis-database). Make sure to set up Redis OM url in conda environment
+
+```bash
+conda env config vars set REDIS_OM_URL="redis://:PASSWORD@server_name:port_num"
 ```
-pip install poetry
-poetry install --no-root
+
+**Set OpenAI API Key** 
+
+```bash
+conda env config vars set OPENAI_API_KEY="<your_openai_api_key>"
 ```
 
-## Generating LLM Annotations
+After setting these environment variables, run `conda deactivate && conda activate sotopia-rl` to apply them.
+
+
+
+## Step 1 - 🧠 Generating LLM Annotations
 
 To generate LLM annotations, you need to download the original sotopia-pi episodes file from the [huggingface repository](https://huggingface.co/datasets/cmu-lti/sotopia-pi/tree/main) and place it in the `data` folder. Then run the following command:
 ```
@@ -32,5 +79,130 @@ python sample_episodes_and_annotate.py --data_dir /workspace/sotopia-rl/data --l
 The annotations will need to be furuther processed into the format required by the training script. This can be done by running the following command:
 ```
 cd ../data_process
-python process_annotation_direct_attribution.py --data_dir /workspace/sotopia-rl/data --input_file sotopia_pi_bc_episodes_annotated.jsonl --reward_output_file sotopia_pi_bc_episodes_reward.json --ppo_output_file sotopia_pi_bc_episodes_ppo.json
+python process_annotation_direct_attribution.py --data_dir /workspace/sotopia-rl/data --input_file sotopia_pi_bc_episodes_annotated.jsonl --reward_output_file sotopia_pi_bc_episodes_reward.json --grpo_output_file sotopia_pi_bc_episodes_grpo.json
+```
+
+
+
+## Step 2 - 🤖 Model Training
+
+### 2.0 Preparation
+
+Before training, make sure you have configured **Accelerate** correctly. Save your configurations in the following files under `scripts/`:
+
+- `accelerate_config_sft.yaml`
+- `accelerate_config_rm.yaml`
+- `accelerate_config_grpo.yaml`
+
+All training scripts should be run from the `scripts/` directory.
+
+### 2.1 Behavior Cloning (SFT)
+
+We first collect self-play dialogue data using **GPT-4o** in the SOTOPIA environment. Then we fine-tune a **Qwen2.5-7B-Instruct** model using supervised fine-tuning (SFT) with LoRA.
+
+#### 2.1.1: Data Collection
+
+Conversation data can be collected via Redis.
+
+Use `scripts/data_process/serialize.py` to extract and serialize the logs.
+
+To load data from Redis and save it as `.jsonl`:
+
+```bash
+from sotopia.database.logs import EpisodeLog
+episode = EpisodeLog.get(pk = 'xxxxx')
+episodes_to_jsonl(episodes, "xxxxx.jsonl")
+```
+
+#### 2.1.2: Train with Behavior Cloning (SFT)
+
+This command performs supervised fine-tuning using LoRA on Qwen2.5-7B-Instruct.
+
+```bash
+export MODEL_PATH="<your_model_path>"
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 accelerate launch \
+  --config_file ./accelerate_config_sft.yaml \
+  --main_process_port 29512 \
+    ./train_sft.py \
+    --model_name $MODEL_PATH \
+    --learning_rate 1e-4 \
+    --max_length 4096 \
+    --train_batch_size 2 \
+    --val_batch_size 1 \
+    --accumulation_steps 8 \
+    --num_epochs 500 \
+    --use_lora \ # Enable LoRA for parameter-efficient fine-tuning
+    --evaluation_steps 5 \ # Evaluate every 5 training steps
+    --sft_data_path ../data/sft_data_path.json \
+    --template_path ../evals/qwen2.5-7b.jinja \
+    --checkpoint_dir ../sft_checkpoints_qwen2.5-7b
+```
+
+### 2.2 Reward Model Training
+
+```bash
+export MODEL_PATH="<your_model_path>"
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 accelerate launch \
+  --config_file ./accelerate_config_rm.yaml \
+  --main_process_port 29500 \
+  ./scripts/train_rm.py \
+  --model_name $MODEL_PATH \
+  --learning_rate 1e-5 \
+  --max_length 4096 \
+  --train_batch_size 1 \
+  --val_batch_size 1 \
+  --accumulation_steps 8 \
+  --num_epochs 30 \
+  --evaluation_steps 50 \
+  --reward_data_path ../data/rm_data_path.json \
+  --template_path ../evals/qwen2.5-7b.jinja \
+  --checkpoint_dir ../rm_checkpoints_qwen2.5-7b
 ```
+
+### 2.3 GRPO Training
+
+We now use the behavior cloning model to generate self-play dialogues and train the agent using **GRPO(Group Reward Policy Optimization)**
+
+#### 2.3.1: Data Collection
+
+Conversation data can be collected via Redis after running self-play using GPT-4o.
+
+Use `scripts/data_process/serialize.py` to extract and serialize the logs.
+
+To load data from Redis and save it as `.jsonl`:
+
+```bash
+from sotopia.database.logs import EpisodeLog
+episode = EpisodeLog.get(pk = 'xxxxx')
+episodes_to_jsonl(episodes, "xxxxx.jsonl")
+```
+#### 2.3.2: Training with GRPO
+
+```bash
+export MODEL_PATH="<your_model_path>"
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 accelerate launch \
+  --config_file ./accelerate_config_grpo.yaml \
+  --main_process_port 29511 \
+  ./train_grpo.py \
+  --model_name $MODEL_PATH \
+  --policy_adapter_path ../sft_checkpoints_qwen2.5-7b/best-checkpoints \
+  --reward_adapter_path ../rm_checkpoints_qwen2.5-7b/best-checkpoints \
+  --learning_rate 5e-6 \
+  --per_device_train_batch_size 4 \
+  --per_device_eval_batch_size 4 \
+  --gradient_accumulation_steps 8 \
+  --grpo_data_path ../data/grpo_data_path.json \
+  --template_path ../evals/qwen2.5-7b.jinja \
+  --num_grpo_epochs 2 \
+  --use_lora_train_grpo \
+  --num_generations 16 \
+  --output_dir ../grpo_checkpoints
+```
+
+
+
+## Step 3 - Automatic Evaluation
+
+We first deploy SFT and GRPO model using vllm and, deploy reward model using danjgo, then we evaluate our model based on SOTOPIA-EVAL framework.
+
+For details please see this section.
diff --git a/assets/sotopia_method.pdf b/assets/sotopia_method.pdf
diff --git a/evals/grpo_serving.sh b/evals/grpo_serving.sh
@@ -0,0 +1,67 @@
+export REPO_FOLDER_NAME="$(cd "$(dirname "$0")/.." && pwd)"
+export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct"
+export SFT_GPU=0
+export PPO_GPU=1
+export SFT_PORT=7010
+export GRPO_PORT=7020
+export SFT_MODEL_FOLDER_NAME="sft_checkpoints_qwen2.5-7b"
+export GRPO_MODEL_FOLDER_NAME="grpo_checkpoints_qwen2.5-7b"
+export SFT_MODEL_CKPT_STEP=1600
+export GRPO_MODEL_CKPT_STEP=1600
+export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/"
+export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/"
+export ENV_MODEL="gpt-4o"
+export CHAT_TEMPLATE="${REPO_FOLDER_NAME}/evals/qwen2.5-7b.jinja"
+
+
+
+export TAG="${GRPO_MODEL_FOLDER_NAME}_step_${GRPO_MODEL_CKPT_STEP}_vs_${SFT_MODEL_FOLDER_NAME}_step_${SFT_MODEL_CKPT_STEP}"
+export SFT_MODEL_NAME="${SFT_MODEL_FOLDER_NAME}-gpu${SFT_GPU}"
+export GRPO_MODEL_NAME="${GRPO_MODEL_FOLDER_NAME}-gpu${GRPO_GPU}"
+export MODEL_A=custom/${PPO_MODEL_NAME}@http://localhost:${PPO_PORT}/v1
+export MODEL_B=custom/${SFT_MODEL_NAME}@http://localhost:${SFT_PORT}/v1
+
+# Command 1: Launch the VLLM API server with LoRA enabled.
+CUDA_VISIBLE_DEVICES=$SFT_GPU python -m vllm.entrypoints.openai.api_server \
+    --model $MODEL_PATH \
+    --port "$SFT_PORT" \
+    --max-lora-rank 64 \
+    --chat-template $CHAT_TEMPLATE \
+    --served-model-name qwen25-7b-instruct \
+    --enable-lora \
+    --lora-modules "$SFT_MODEL_NAME=$SFT_MODEL_PATH"
+
+# Command 2: Launch the VLLM API server with LoRA enabled.
+CUDA_VISIBLE_DEVICES=$GRPO_GPU python -m vllm.entrypoints.openai.api_server \
+    --model $MODEL_PATH \
+    --port "$GRPO_PORT" \
+    --max-lora-rank 64 \
+    --chat-template $CHAT_TEMPLATE \
+    --served-model-name qwen25-7b-instruct \
+    --enable-lora \
+    --lora-modules "$GRPO_MODEL_NAME=$GRPO_MODEL_PATH"
+
+# Command 3: Run experiment evaluations.
+python examples/experiment_eval.py \
+  --gin_file sotopia_conf/generation_utils_conf/generate.gin \
+  --gin_file sotopia_conf/server_conf/server.gin \
+  --gin_file sotopia_conf/run_async_server_in_batch.gin \
+  --gin.BATCH_SIZE=20 \
+  --gin.PUSH_TO_DB=True \
+  '--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]' \
+  "--gin.ENV_MODEL='${ENV_MODEL}'" \
+  "--gin.AGENT1_MODEL='${MODEL_A}'" \
+  "--gin.AGENT2_MODEL='${MODEL_B}'" \
+  "--gin.TAG='${TAG}'" \
+&& \
+python examples/experiment_eval.py \
+  --gin_file sotopia_conf/generation_utils_conf/generate.gin \
+  --gin_file sotopia_conf/server_conf/server.gin \
+  --gin_file sotopia_conf/run_async_server_in_batch.gin \
+  --gin.BATCH_SIZE=20 \
+  --gin.PUSH_TO_DB=True \
+  '--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]' \
+  "--gin.ENV_MODEL='${ENV_MODEL}'" \
+  "--gin.AGENT2_MODEL='${MODEL_A}'" \
+  "--gin.AGENT1_MODEL='${MODEL_B}'" \
+  "--gin.TAG='${TAG}'"