diff --git a/README.md b/README.md index b48c5f7..5aaf961 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,69 @@ ![sotopia-rl](assets/sotopia-rl-title.png) -# sotopia-rl +

Sotopia-RL: Reward Design for Social Intelligence

-## Install +[![Project Page](https://img.shields.io/badge/Project-Page-green.svg)](https://rl.sotopia.world/)![Paper PDF](https://img.shields.io/badge/Paper-PDF-red.svg)[![huggingface](https://img.shields.io/badge/%F0%9F%A4%97-Model-orange)](https://huggingface.co/ulab-ai/sotopia-rl-qwen-2.5-7B-grpo)[![Python 3.10](https://img.shields.io/badge/python-%E2%89%A53.10-blue)](https://www.python.org/downloads/release/python-3109/)[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://pre-commit.com/) Code style: black![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-blue.svg) -Create a conda environment with python 3.10 -``` + + +## 📚 Table of Contents +- [Introduction](#introduction) +- [Step 0 - Environment Setup](#step-0---️-environment-setup) +- [Step 1 - Generating LLM Annotations](#step-1---generating-llm-annotations) +- [Step 2 - Model Training](#step-2---model-training) + - [Behavior Cloning (SFT)](#21-behavior-cloning-supervised-fine-tuning-sft) + - [Reward Model Training](#22-reward-model-training) + - [GRPO Training](#23-grpo-training) +- [Step 3 - Automatic Evaluation](#step-3---automatic-evaluation) + +## Introduction + +**Sotopia-RL** is an utterance-level, attribution-based, and multi-dimensional social reward design method, trained using single-turn online RL It achieves state-of-the-art performance on social goal completion tasks in the SOTOPIA benchmark. + +We first attribute episode-level rewards for multi-turn social interactions to in- dividual utterances with LLMs. Then, we construct a combined reward that includes multiple dimensions of rewards besides goal completion, allowing us to regularize the optimization pro- cess for goal completion. These rewards are used to guide the RL training of social agents. + + + +![sotopia-rl](assets/sotopia_method.pdf) + +## Step 0 - 🛠️ Environment Setup + +We recommend using `conda` to manage a clean Python environment for `sotopia-rl`. + +#### 1. Create and activate the environment + +```bash conda create -n sotopia-rl python=3.10 conda activate sotopia-rl ``` -Then install poetry and use it to install the dependencies. Currently the package is under development so it's recommended to use the `--no-root` flag to avoid installing the package itself. +#### 2. Install Poetry (Python package manager) + +```bash +curl -sSL https://install.python-poetry.org | python3 +export PATH="$HOME/.local/bin:$PATH" +``` +#### 3. Environment Variables + +##### Set Redis URL + +A Redis database needs to be set up to connect to a Redis DB for loading and saving environment/session data and run this repo. For detailed instructions of setting up Redis database, please refer to [this tutorial](https://github.com/sotopia-lab/sotopia-pi/tree/main/data_generate#setting-up-redis-database). Make sure to set up Redis OM url in conda environment + +```bash +conda env config vars set REDIS_OM_URL="redis://:PASSWORD@server_name:port_num" ``` -pip install poetry -poetry install --no-root + +**Set OpenAI API Key** + +```bash +conda env config vars set OPENAI_API_KEY="" ``` -## Generating LLM Annotations +After setting these environment variables, run `conda deactivate && conda activate sotopia-rl` to apply them. + + + +## Step 1 - 🧠 Generating LLM Annotations To generate LLM annotations, you need to download the original sotopia-pi episodes file from the [huggingface repository](https://huggingface.co/datasets/cmu-lti/sotopia-pi/tree/main) and place it in the `data` folder. Then run the following command: ``` @@ -32,5 +79,130 @@ python sample_episodes_and_annotate.py --data_dir /workspace/sotopia-rl/data --l The annotations will need to be furuther processed into the format required by the training script. This can be done by running the following command: ``` cd ../data_process -python process_annotation_direct_attribution.py --data_dir /workspace/sotopia-rl/data --input_file sotopia_pi_bc_episodes_annotated.jsonl --reward_output_file sotopia_pi_bc_episodes_reward.json --ppo_output_file sotopia_pi_bc_episodes_ppo.json +python process_annotation_direct_attribution.py --data_dir /workspace/sotopia-rl/data --input_file sotopia_pi_bc_episodes_annotated.jsonl --reward_output_file sotopia_pi_bc_episodes_reward.json --grpo_output_file sotopia_pi_bc_episodes_grpo.json +``` + + + +## Step 2 - 🤖 Model Training + +### 2.0 Preparation + +Before training, make sure you have configured **Accelerate** correctly. Save your configurations in the following files under `scripts/`: + +- `accelerate_config_sft.yaml` +- `accelerate_config_rm.yaml` +- `accelerate_config_grpo.yaml` + +All training scripts should be run from the `scripts/` directory. + +### 2.1 Behavior Cloning (SFT) + +We first collect self-play dialogue data using **GPT-4o** in the SOTOPIA environment. Then we fine-tune a **Qwen2.5-7B-Instruct** model using supervised fine-tuning (SFT) with LoRA. + +#### 2.1.1: Data Collection + +Conversation data can be collected via Redis. + +Use `scripts/data_process/serialize.py` to extract and serialize the logs. + +To load data from Redis and save it as `.jsonl`: + +```bash +from sotopia.database.logs import EpisodeLog +episode = EpisodeLog.get(pk = 'xxxxx') +episodes_to_jsonl(episodes, "xxxxx.jsonl") +``` + +#### 2.1.2: Train with Behavior Cloning (SFT) + +This command performs supervised fine-tuning using LoRA on Qwen2.5-7B-Instruct. + +```bash +export MODEL_PATH="" +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 accelerate launch \ + --config_file ./accelerate_config_sft.yaml \ + --main_process_port 29512 \ + ./train_sft.py \ + --model_name $MODEL_PATH \ + --learning_rate 1e-4 \ + --max_length 4096 \ + --train_batch_size 2 \ + --val_batch_size 1 \ + --accumulation_steps 8 \ + --num_epochs 500 \ + --use_lora \ # Enable LoRA for parameter-efficient fine-tuning + --evaluation_steps 5 \ # Evaluate every 5 training steps + --sft_data_path ../data/sft_data_path.json \ + --template_path ../evals/qwen2.5-7b.jinja \ + --checkpoint_dir ../sft_checkpoints_qwen2.5-7b +``` + +### 2.2 Reward Model Training + +```bash +export MODEL_PATH="" +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 accelerate launch \ + --config_file ./accelerate_config_rm.yaml \ + --main_process_port 29500 \ + ./scripts/train_rm.py \ + --model_name $MODEL_PATH \ + --learning_rate 1e-5 \ + --max_length 4096 \ + --train_batch_size 1 \ + --val_batch_size 1 \ + --accumulation_steps 8 \ + --num_epochs 30 \ + --evaluation_steps 50 \ + --reward_data_path ../data/rm_data_path.json \ + --template_path ../evals/qwen2.5-7b.jinja \ + --checkpoint_dir ../rm_checkpoints_qwen2.5-7b ``` + +### 2.3 GRPO Training + +We now use the behavior cloning model to generate self-play dialogues and train the agent using **GRPO(Group Reward Policy Optimization)** + +#### 2.3.1: Data Collection + +Conversation data can be collected via Redis after running self-play using GPT-4o. + +Use `scripts/data_process/serialize.py` to extract and serialize the logs. + +To load data from Redis and save it as `.jsonl`: + +```bash +from sotopia.database.logs import EpisodeLog +episode = EpisodeLog.get(pk = 'xxxxx') +episodes_to_jsonl(episodes, "xxxxx.jsonl") +``` +#### 2.3.2: Training with GRPO + +```bash +export MODEL_PATH="" +CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 accelerate launch \ + --config_file ./accelerate_config_grpo.yaml \ + --main_process_port 29511 \ + ./train_grpo.py \ + --model_name $MODEL_PATH \ + --policy_adapter_path ../sft_checkpoints_qwen2.5-7b/best-checkpoints \ + --reward_adapter_path ../rm_checkpoints_qwen2.5-7b/best-checkpoints \ + --learning_rate 5e-6 \ + --per_device_train_batch_size 4 \ + --per_device_eval_batch_size 4 \ + --gradient_accumulation_steps 8 \ + --grpo_data_path ../data/grpo_data_path.json \ + --template_path ../evals/qwen2.5-7b.jinja \ + --num_grpo_epochs 2 \ + --use_lora_train_grpo \ + --num_generations 16 \ + --output_dir ../grpo_checkpoints +``` + + + +## Step 3 - Automatic Evaluation + +We first deploy SFT and GRPO model using vllm and, deploy reward model using danjgo, then we evaluate our model based on SOTOPIA-EVAL framework. + +For details please see this section. diff --git a/assets/sotopia_method.pdf b/assets/sotopia_method.pdf new file mode 100644 index 0000000..f01df55 Binary files /dev/null and b/assets/sotopia_method.pdf differ diff --git a/evals/grpo_serving.sh b/evals/grpo_serving.sh new file mode 100644 index 0000000..0eb1b85 --- /dev/null +++ b/evals/grpo_serving.sh @@ -0,0 +1,67 @@ +export REPO_FOLDER_NAME="$(cd "$(dirname "$0")/.." && pwd)" +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" +export SFT_GPU=0 +export PPO_GPU=1 +export SFT_PORT=7010 +export GRPO_PORT=7020 +export SFT_MODEL_FOLDER_NAME="sft_checkpoints_qwen2.5-7b" +export GRPO_MODEL_FOLDER_NAME="grpo_checkpoints_qwen2.5-7b" +export SFT_MODEL_CKPT_STEP=1600 +export GRPO_MODEL_CKPT_STEP=1600 +export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" +export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/" +export ENV_MODEL="gpt-4o" +export CHAT_TEMPLATE="${REPO_FOLDER_NAME}/evals/qwen2.5-7b.jinja" + + + +export TAG="${GRPO_MODEL_FOLDER_NAME}_step_${GRPO_MODEL_CKPT_STEP}_vs_${SFT_MODEL_FOLDER_NAME}_step_${SFT_MODEL_CKPT_STEP}" +export SFT_MODEL_NAME="${SFT_MODEL_FOLDER_NAME}-gpu${SFT_GPU}" +export GRPO_MODEL_NAME="${GRPO_MODEL_FOLDER_NAME}-gpu${GRPO_GPU}" +export MODEL_A=custom/${PPO_MODEL_NAME}@http://localhost:${PPO_PORT}/v1 +export MODEL_B=custom/${SFT_MODEL_NAME}@http://localhost:${SFT_PORT}/v1 + +# Command 1: Launch the VLLM API server with LoRA enabled. +CUDA_VISIBLE_DEVICES=$SFT_GPU python -m vllm.entrypoints.openai.api_server \ + --model $MODEL_PATH \ + --port "$SFT_PORT" \ + --max-lora-rank 64 \ + --chat-template $CHAT_TEMPLATE \ + --served-model-name qwen25-7b-instruct \ + --enable-lora \ + --lora-modules "$SFT_MODEL_NAME=$SFT_MODEL_PATH" + +# Command 2: Launch the VLLM API server with LoRA enabled. +CUDA_VISIBLE_DEVICES=$GRPO_GPU python -m vllm.entrypoints.openai.api_server \ + --model $MODEL_PATH \ + --port "$GRPO_PORT" \ + --max-lora-rank 64 \ + --chat-template $CHAT_TEMPLATE \ + --served-model-name qwen25-7b-instruct \ + --enable-lora \ + --lora-modules "$GRPO_MODEL_NAME=$GRPO_MODEL_PATH" + +# Command 3: Run experiment evaluations. +python examples/experiment_eval.py \ + --gin_file sotopia_conf/generation_utils_conf/generate.gin \ + --gin_file sotopia_conf/server_conf/server.gin \ + --gin_file sotopia_conf/run_async_server_in_batch.gin \ + --gin.BATCH_SIZE=20 \ + --gin.PUSH_TO_DB=True \ + '--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]' \ + "--gin.ENV_MODEL='${ENV_MODEL}'" \ + "--gin.AGENT1_MODEL='${MODEL_A}'" \ + "--gin.AGENT2_MODEL='${MODEL_B}'" \ + "--gin.TAG='${TAG}'" \ +&& \ +python examples/experiment_eval.py \ + --gin_file sotopia_conf/generation_utils_conf/generate.gin \ + --gin_file sotopia_conf/server_conf/server.gin \ + --gin_file sotopia_conf/run_async_server_in_batch.gin \ + --gin.BATCH_SIZE=20 \ + --gin.PUSH_TO_DB=True \ + '--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]' \ + "--gin.ENV_MODEL='${ENV_MODEL}'" \ + "--gin.AGENT2_MODEL='${MODEL_A}'" \ + "--gin.AGENT1_MODEL='${MODEL_B}'" \ + "--gin.TAG='${TAG}'" diff --git a/evals/ppo_serving.sh b/evals/ppo_serving.sh deleted file mode 100644 index f225d38..0000000 --- a/evals/ppo_serving.sh +++ /dev/null @@ -1,132 +0,0 @@ -export SFT_GPU=8 -export PPO_GPU=9 -export SFT_PORT=8090 -export PPO_PORT=8095 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b" -export PPO_MODEL_FOLDER_NAME="ppo_qwen25_7b_reward_only_response_gpt-4o" -export SFT_MODEL_CKPT_STEP=1000 -export PPO_MODEL_CKPT_STEP=1500 -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/" - -export SFT_GPU=8 -export PPO_GPU=9 -export SFT_PORT=8070 -export PPO_PORT=8075 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b" -export PPO_MODEL_FOLDER_NAME="ppo_qwen25_7b_reward_utterance_quality_gpt-4o" -export SFT_MODEL_CKPT_STEP=1000 -export PPO_MODEL_CKPT_STEP=1500 -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/" - - -export SFT_GPU=2 -export PPO_GPU=3 -export SFT_PORT=8070 -export PPO_PORT=8075 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b_sft_round_1_bc_data_top_2" -export SFT_MODEL_CKPT_STEP=1500 -export PPO_MODEL_FOLDER_NAME="ppo_qwen25_7b_reward_utterance_quality_gpt-4o" -export PPO_MODEL_CKPT_STEP=2400 -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/" -export ENV_MODEL="gpt-4o" - - -export SFT_GPU=2 -export PPO_GPU=3 -export SFT_PORT=7019 -export PPO_PORT=7009 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b_sft_round_1_bc_data_top_2" -export PPO_MODEL_FOLDER_NAME="grpo_rm_0504_annotated_direct_default" -export SFT_MODEL_CKPT_STEP=1500 -export PPO_MODEL_CKPT_STEP=1750 -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/" -export ENV_MODEL="gpt-4o" - - -export SFT_GPU=6 -export PPO_GPU=7 -export SFT_PORT=7020 -export PPO_PORT=7029 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b_sft_round_1_bc_data_top_2" -export PPO_MODEL_FOLDER_NAME="grpo_rm_0504_annotated_direct_default" -export SFT_MODEL_CKPT_STEP=1500 -export PPO_MODEL_CKPT_STEP=1600 -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/" -export ENV_MODEL="gpt-4o" - - -export SFT_GPU=7 -export PPO_GPU=8 -export SFT_PORT=5010 -export PPO_PORT=5020 -export SFT_MODEL_FOLDER_NAME="sft_round_1_bc_data_top_2_with_aligned_format_instruction_prompt_weight_decay_0_official_qwen_lora_0509" -export PPO_MODEL_FOLDER_NAME="sft_round_1_bc_data_top_2_with_aligned_format_instruction_prompt_weight_decay_0_official_qwen_lora_0509" -export SFT_MODEL_CKPT_STEP=1600 -export PPO_MODEL_CKPT_STEP=1600 -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export PPO_MODEL_PATH="${REPO_FOLDER_NAME}/${PPO_MODEL_FOLDER_NAME}/checkpoint-${PPO_MODEL_CKPT_STEP}/" -export ENV_MODEL="gpt-4o" - - -export TAG="${PPO_MODEL_FOLDER_NAME}_step_${PPO_MODEL_CKPT_STEP}_vs_${SFT_MODEL_FOLDER_NAME}_step_${SFT_MODEL_CKPT_STEP}-0505" -export SFT_MODEL_NAME="${SFT_MODEL_FOLDER_NAME}-gpu${SFT_GPU}" -export PPO_MODEL_NAME="${PPO_MODEL_FOLDER_NAME}-gpu${PPO_GPU}" -export MODEL_A=custom/${PPO_MODEL_NAME}@http://localhost:${PPO_PORT}/v1 -export MODEL_B=custom/${SFT_MODEL_NAME}@http://localhost:${SFT_PORT}/v1 -export REDIS_OM_URL="redis://:QzmCUD3C3RdsR@35.232.108.130:6379" - -# Command 1: Launch the VLLM API server with LoRA enabled. -CUDA_VISIBLE_DEVICES=$SFT_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ - --port "$SFT_PORT" \ - --max-lora-rank 64 \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ - --served-model-name qwen25-7b-instruct \ - --enable-lora \ - --lora-modules "$SFT_MODEL_NAME=$SFT_MODEL_PATH" - -# Command 2: Launch the VLLM API server with LoRA enabled. -CUDA_VISIBLE_DEVICES=$PPO_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ - --port "$PPO_PORT" \ - --max-lora-rank 64 \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ - --served-model-name qwen25-7b-instruct \ - --enable-lora \ - --lora-modules "$PPO_MODEL_NAME=$PPO_MODEL_PATH" - -# Command 3: Run experiment evaluations. -python examples/experiment_eval.py \ - --gin_file sotopia_conf/generation_utils_conf/generate.gin \ - --gin_file sotopia_conf/server_conf/server.gin \ - --gin_file sotopia_conf/run_async_server_in_batch.gin \ - --gin.BATCH_SIZE=20 \ - --gin.PUSH_TO_DB=True \ - '--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]' \ - "--gin.ENV_MODEL='${ENV_MODEL}'" \ - "--gin.AGENT1_MODEL='${MODEL_A}'" \ - "--gin.AGENT2_MODEL='${MODEL_B}'" \ - "--gin.TAG='${TAG}'" \ -&& \ -python examples/experiment_eval.py \ - --gin_file sotopia_conf/generation_utils_conf/generate.gin \ - --gin_file sotopia_conf/server_conf/server.gin \ - --gin_file sotopia_conf/run_async_server_in_batch.gin \ - --gin.BATCH_SIZE=20 \ - --gin.PUSH_TO_DB=True \ - '--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]' \ - "--gin.ENV_MODEL='${ENV_MODEL}'" \ - "--gin.AGENT2_MODEL='${MODEL_A}'" \ - "--gin.AGENT1_MODEL='${MODEL_B}'" \ - "--gin.TAG='${TAG}'" diff --git a/evals/rej_sampling_serving.sh b/evals/rej_sampling_serving.sh index e6d166f..25c5732 100644 --- a/evals/rej_sampling_serving.sh +++ b/evals/rej_sampling_serving.sh @@ -1,144 +1,44 @@ #!/bin/bash -export VLLM_GPU=4 -export DJANGO_GPU=5 -export VLLM_PORT=8035 -export DJANGO_PORT=8047 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b_sft_round_1_bc_data_top_2" -export SFT_MODEL_CKPT_STEP=1500 -export RM_FOLDER_NAME="rm_reward_direct_default_without_that_n_error_as_the_end" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-4000" -export ENV_MODEL="gpt-4o" - - -export VLLM_GPU=0 -export DJANGO_GPU=1 -export VLLM_PORT=8001 -export DJANGO_PORT=8008 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b" -export RM_FOLDER_NAME="rm_reward_mixed_direct_o3_only_response" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-1000/" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-4600" -export ENV_MODEL="gpt-4o" - -export VLLM_GPU=2 -export DJANGO_GPU=3 -export VLLM_PORT=8013 -export DJANGO_PORT=8024 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b" -export RM_FOLDER_NAME="rm_reward_direct_default_gpt-4o" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-1000/" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-4400" - - +export REPO_FOLDER_NAME="$(cd "$(dirname "$0")/.." && pwd)" +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" export VLLM_GPU=0 export DJANGO_GPU=1 -export VLLM_PORT=8035 -export DJANGO_PORT=8047 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b" -export RM_FOLDER_NAME="rm_reward_only_response_no_goal_gpt-4o" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-1000/" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-4400" - -export VLLM_GPU=4 -export DJANGO_GPU=5 -export VLLM_PORT=8005 -export DJANGO_PORT=8017 +export VLLM_PORT=8010 +export DJANGO_PORT=8020 export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b" -export RM_FOLDER_NAME="rm_reward_utterance_quality_no_goal_gpt-4o" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-1000/" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-3200" - -export VLLM_GPU=2 -export DJANGO_GPU=3 -export VLLM_PORT=8015 -export DJANGO_PORT=8027 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b" -export RM_FOLDER_NAME="rm_reward_direct_default_no_goal_gpt-4o" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-1000/" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-3200" - - - - - -#rej 89 -export VLLM_GPU=8 -export DJANGO_GPU=9 -export VLLM_PORT=8015 -export DJANGO_PORT=8027 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_qwen25_7b_sft_round_1_bc_data_top_2" -export SFT_MODEL_CKPT_STEP=1500 -export RM_FOLDER_NAME="rm_reward_key_utterance_no_goal_gpt-4o" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-4000" +export SFT_MODEL_FOLDER_NAME="sft_checkpoints_qwen2.5-7b" +export RM_FOLDER_NAME="rm_checkpoints_qwen2.5-7b" +export SFT_MODEL_CKPT_STEP=1000 +export RM_MODEL_CKPT_STEP=4600 +export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" +export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-${RM_MODEL_CKPT_STEP}" export ENV_MODEL="gpt-4o" +export CHAT_TEMPLATE="${REPO_FOLDER_NAME}/evals/qwen2.5-7b.jinja" -#0 sotopia-hard-70 discounting rm_reward_discounting_rej_sampling_num10_vs_sft_qwen25_7b_sft_round_1_bc_data_top_2_0328 -export VLLM_GPU=2 -export DJANGO_GPU=3 -export VLLM_PORT=8000 -export DJANGO_PORT=8015 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="new_sft_default_0506" -export SFT_MODEL_CKPT_STEP=100 -export RM_FOLDER_NAME="rm_all_the_same_0507" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-7180" -export ENV_MODEL="gpt-4o" -export VLLM_GPU=7 -export DJANGO_GPU=8 -export VLLM_PORT=8035 -export DJANGO_PORT=8045 -export REJ_SAMPLING_NUM=10 -export SFT_MODEL_FOLDER_NAME="sft_round_1_bc_data_top_2_with_aligned_format_instruction_prompt_0509" -export SFT_MODEL_CKPT_STEP=500 -export RM_FOLDER_NAME="rm_knowledge_0507" -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}" -export RM_MODEL_PATH="${REPO_FOLDER_NAME}/${RM_FOLDER_NAME}/checkpoint-6800" -export ENV_MODEL="gpt-4o" - -export TAG="${RM_FOLDER_NAME}_rej_sampling_num${REJ_SAMPLING_NUM}_vs_${SFT_MODEL_FOLDER_NAME}_0509_v2" -export SFT_MODEL_NAME="sft_qwen25_7b_sft_round_1_bc_data_top_2-gpu${VLLM_GPU}" +export TAG="${RM_FOLDER_NAME}_step_${RM_MODEL_CKPT_STEP}_rej_sampling_num${REJ_SAMPLING_NUM}_vs_${SFT_MODEL_FOLDER_NAME}_step_${SFT_MODEL_CKPT_STEP}" +export SFT_MODEL_NAME="${SFT_MODEL_FOLDER_NAME}-gpu${VLLM_GPU}" export MODEL_A=custom/${RM_FOLDER_NAME}_rejsampling_num${REJ_SAMPLING_NUM}@http://localhost:${DJANGO_PORT}/sotopia export MODEL_B=custom/${SFT_MODEL_NAME}@http://localhost:${VLLM_PORT}/v1 -export REDIS_OM_URL="redis://:QzmCUD3C3RdsR@35.232.108.130:6379" export SFT_MODEL_VLLM_API_URL="http://localhost:${VLLM_PORT}/v1/completions" # Command 1: Launch the VLLM API server with LoRA enabled. CUDA_VISIBLE_DEVICES=$VLLM_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ + --model $MODEL_PATH \ --port "$VLLM_PORT" \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ + --chat-template $CHAT_TEMPLATE \ --served-model-name qwen25-7b-instruct \ --enable-lora \ --lora-modules "$SFT_MODEL_NAME=$SFT_MODEL_PATH" # Command 2: Start the Django server with the specified configuration. -CUDA_VISIBLE_DEVICES=$DJANGO_GPU python /data/haofeiy2/sotopia-rl/serves/manage.py start_with_config \ +CUDA_VISIBLE_DEVICES=$DJANGO_GPU python $REPO_FOLDER_NAME/serves/manage.py start_with_config \ --sft_model_name "$SFT_MODEL_NAME" \ --sft_model_vllm_api_url "$SFT_MODEL_VLLM_API_URL" \ --reward_model_path "$RM_MODEL_PATH" \ - --reward_model_name "/mnt/data_from_server1/models/Qwen2.5-7B-Instruct" \ - --template_path "/data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja" \ + --reward_model_name $MODEL_PATH \ + --template_path $CHAT_TEMPLATE \ --max_responses "$REJ_SAMPLING_NUM" \ --max_length 4096 \ --port "$DJANGO_PORT" \ @@ -169,5 +69,3 @@ python examples/experiment_eval.py \ "--gin.AGENT2_MODEL='${MODEL_A}'" \ "--gin.AGENT1_MODEL='${MODEL_B}'" \ "--gin.TAG='${TAG}'" - -# rm_reward_direct_default_without_that_n_error_as_the_end \ No newline at end of file diff --git a/evals/self_play.sh b/evals/self_play.sh index c9ee4e9..e859edc 100644 --- a/evals/self_play.sh +++ b/evals/self_play.sh @@ -1,30 +1,34 @@ -export SFT_SELFPLAY_TAG="selfplay_new_sft_sotopia_rl_gpt-4o_0507_v7" -export SFT1_GPU=6 -export SFT2_GPU=7 +export REPO_FOLDER_NAME="$(cd "$(dirname "$0")/.." && pwd)" +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" +export SFT_SELFPLAY_TAG="selfpaly_tag" +export SFT1_GPU=0 +export SFT2_GPU=1 +export SFT1_PORT=7010 +export SFT2_PORT=7020 +export SFT1_MODEL_NAME="sft_checkpoints_qwen2.5-7b" +export SFT2_MODEL_NAME="sft_checkpoints_qwen2.5-7b" +export SFT1_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT1_MODEL_NAME}/best-checkpoint" +export SFT2_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT2_MODEL_NAME}/best-checkpoint" export ENV_MODEL="gpt-4o" -export SFT1_PORT=7040 -export SFT1_MODEL_NAME="new_sft_default_0506" -export SFT2_PORT=7050 -export SFT2_MODEL_NAME="new_sft_default_0506" +export CHAT_TEMPLATE="${REPO_FOLDER_NAME}/evals/qwen2.5-7b.jinja" + export MODEL_A=custom/${SFT1_MODEL_NAME}@http://localhost:${SFT1_PORT}/v1 export MODEL_B=custom/${SFT2_MODEL_NAME}@http://localhost:${SFT2_PORT}/v1 -export SFT1_MODEL_PATH="/data/haofeiy2/sotopia-rl/${SFT1_MODEL_NAME}/checkpoint-1000" -export SFT2_MODEL_PATH="/data/haofeiy2/sotopia-rl/${SFT2_MODEL_NAME}/checkpoint-1000" -export REDIS_OM_URL="redis://:QzmCUD3C3RdsR@35.232.108.130:6379" - +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" +export CHAT_TEMPLATE="${REPO_FOLDER_NAME}/evals/qwen2.5-7b.jinja" CUDA_VISIBLE_DEVICES=$SFT1_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ + --model $MODEL_PATH \ --port "$SFT1_PORT" \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ + --chat-template $CHAT_TEMPLATE \ --served-model-name qwen25-7b-instruct \ --enable-lora \ --lora-modules "$SFT1_MODEL_NAME=$SFT1_MODEL_PATH" CUDA_VISIBLE_DEVICES=$SFT2_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ + --model $MODEL_PATH \ --port "$SFT2_PORT" \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ + --chat-template $CHAT_TEMPLATE \ --served-model-name qwen25-7b-instruct \ --enable-lora \ --lora-modules "$SFT2_MODEL_NAME=$SFT2_MODEL_PATH" diff --git a/evals/sft_serving.sh b/evals/sft_serving.sh index abf7888..bea5b62 100644 --- a/evals/sft_serving.sh +++ b/evals/sft_serving.sh @@ -1,36 +1,38 @@ +export REPO_FOLDER_NAME="$(cd "$(dirname "$0")/.." && pwd)" +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" export SFT_GPU=2 export ORI_GPU=3 export SFT_PORT=9050 export ORI_PORT=9080 export SFT_MODEL_FOLDER_NAME="new_sft_default_0506" export SFT_MODEL_CKPT_STEP=200 -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export ORI_MODEL_PATH="/mnt/data_from_server1/models/Qwen2.5-7B-Instruct" +export ORI_MODEL_PATH="${MODEL_PATH}" export ENV_MODEL="gpt-4o" +export CHAT_TEMPLATE="${REPO_FOLDER_NAME}/evals/qwen2.5-7b.jinja" -export TAG="Qwen2.5-7B-Instruct_vs_${SFT_MODEL_FOLDER_NAME}_step_${SFT_MODEL_CKPT_STEP}-0504" + +export TAG="Qwen2.5-7B-Instruct_vs_${SFT_MODEL_FOLDER_NAME}_step_${SFT_MODEL_CKPT_STEP}" export SFT_MODEL_NAME="${SFT_MODEL_FOLDER_NAME}-gpu${SFT_GPU}" export ORI_MODEL_NAME="Qwen2.5-7B-Instruct-gpu${ORI_GPU}" export MODEL_A=custom/${ORI_MODEL_NAME}@http://localhost:${ORI_PORT}/v1 export MODEL_B=custom/${SFT_MODEL_NAME}@http://localhost:${SFT_PORT}/v1 -export REDIS_OM_URL="redis://:QzmCUD3C3RdsR@35.232.108.130:6379" # Command 1: Launch the VLLM API server with LoRA enabled. CUDA_VISIBLE_DEVICES=$SFT_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ + --model $MODEL_PATH \ --port "$SFT_PORT" \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ + --chat-template $CHAT_TEMPLATE \ --served-model-name qwen25-7b-instruct \ --enable-lora \ --lora-modules "$SFT_MODEL_NAME=$SFT_MODEL_PATH" # Command 2: Launch the VLLM API server with LoRA enabled. CUDA_VISIBLE_DEVICES=$ORI_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ + --model $MODEL_PATH \ --port "$ORI_PORT" \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ + --chat-template $CHAT_TEMPLATE \ --served-model-name $ORI_MODEL_NAME # Command 3: Run experiment evaluations. @@ -56,52 +58,4 @@ python examples/experiment_eval.py \ "--gin.ENV_MODEL='${ENV_MODEL}'" \ "--gin.AGENT2_MODEL='${MODEL_A}'" \ "--gin.AGENT1_MODEL='${MODEL_B}'" \ - "--gin.TAG='${TAG}'" - - - -# GPUs and ports -export SFT_GPU=7 -export ORI_GPU=8 -export SFT_PORT=8060 -export ORI_PORT=8070 - -# Model folders and checkpoints -export SFT_MODEL_FOLDER_NAME="sft_0510_epoch_500" -export ORI_MODEL_FOLDER_NAME="Qwen2.5-7B-Instruct" -export SFT_MODEL_CKPT_STEP=200 - -# Paths -export REPO_FOLDER_NAME="/data/haofeiy2/sotopia-rl" -export SFT_MODEL_PATH="${REPO_FOLDER_NAME}/${SFT_MODEL_FOLDER_NAME}/checkpoint-${SFT_MODEL_CKPT_STEP}/" -export ORI_MODEL_PATH="/mnt/data_from_server1/models/${ORI_MODEL_FOLDER_NAME}" - -# Experiment metadata -export ENV_MODEL="gpt-4o" -export TAG="${SFT_MODEL_FOLDER_NAME}_step_${SFT_MODEL_CKPT_STEP}_vs_${ORI_MODEL_FOLDER_NAME}-0509_with_fixed_action_order" - -# Names for served endpoints -export SFT_MODEL_NAME="${SFT_MODEL_FOLDER_NAME}-gpu${SFT_GPU}" -export ORI_MODEL_NAME="${ORI_MODEL_FOLDER_NAME}-gpu${ORI_GPU}" - -export MODEL_A="custom/${SFT_MODEL_NAME}@http://localhost:${SFT_PORT}/v1" -export MODEL_B="custom/${ORI_MODEL_NAME}@http://localhost:${ORI_PORT}/v1" - -export REDIS_OM_URL="redis://:QzmCUD3C3RdsR@35.232.108.130:6379" - -# Command 1: Launch the SFT (fine-tuned) server -CUDA_VISIBLE_DEVICES=$SFT_GPU python -m vllm.entrypoints.openai.api_server \ - --model /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ - --port "$SFT_PORT" \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ - --served-model-name "qwen25-7b-instruct" \ - --enable-lora \ - --max-lora-rank 64 \ - --lora-modules "$SFT_MODEL_NAME=$SFT_MODEL_PATH" - -# Command 2: Launch the ORI (base/instruct) server -CUDA_VISIBLE_DEVICES=$ORI_GPU python -m vllm.entrypoints.openai.api_server \ - --model "$ORI_MODEL_PATH" \ - --port "$ORI_PORT" \ - --chat-template /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ - --served-model-name "$ORI_MODEL_NAME" + "--gin.TAG='${TAG}'" \ No newline at end of file diff --git a/scripts/inference_ppo.py b/scripts/inference_grpo.py similarity index 100% rename from scripts/inference_ppo.py rename to scripts/inference_grpo.py diff --git a/scripts/inference_grpo.sh b/scripts/inference_grpo.sh new file mode 100644 index 0000000..122d2f3 --- /dev/null +++ b/scripts/inference_grpo.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +# Run the single evaluation script with your model checkpoint +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" +CUDA_VISIBLE_DEVICES=0 python inference_grpo.py \ + --model_path $MODEL_PATH \ + --adapter_path "../grpo_checkpoint/grpo_checkpoints_qwen2.5-7b/best-checkpoint" \ + --template_path "../evals/qwen2.5-7b.jinja" \ + --example_path "../data/sotopia_pi_gpt4_ppo_overfit.json" \ + --max_length 4096 \ + --use_qlora diff --git a/scripts/inference_ppo.sh b/scripts/inference_ppo.sh deleted file mode 100644 index 94c4985..0000000 --- a/scripts/inference_ppo.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash - -# Run the single evaluation script with your model checkpoint -CUDA_VISIBLE_DEVICES=5 python inference_ppo.py \ - --model_path "/mnt/data_from_server1/models/Qwen2.5-7B-Instruct" \ - --adapter_path "/data/haofeiy2/sotopia-rl/ppo_qwen25_7b/checkpoint-500" \ - --template_path "/data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja" \ - --example_path "/data/haofeiy2/sotopia-rl/data/sotopia_pi_gpt4_ppo_overfit.json" \ - --max_length 4096 \ - --use_qlora diff --git a/scripts/inference_rm.sh b/scripts/inference_rm.sh index f000421..28615db 100644 --- a/scripts/inference_rm.sh +++ b/scripts/inference_rm.sh @@ -1,15 +1,9 @@ #!/bin/bash # Run the single evaluation script with your model checkpoint -CUDA_VISIBLE_DEVICES=9 python inference_rm.py \ - --model_path "/mnt/data_from_server1/models/Qwen2.5-7B-Instruct" \ - --adapter_path "/data/haofeiy2/sotopia-rl/rm_reward_mixed/checkpoint-4000" \ - --template_path "/data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja" \ - --example_path "/data/haofeiy2/sotopia-rl/data/sotopia_pi_gpt4_rm_overfit.json" - - -CUDA_VISIBLE_DEVICES=8 python inference_rm.py \ - --model_path "/mnt/data_from_server1/models/Qwen2.5-7B-Instruct" \ - --adapter_path "/data/haofeiy2/sotopia-rl/rm_overfit_test/checkpoint-100" \ - --template_path "/data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja" \ - --example_path "/data/haofeiy2/sotopia-rl/data/sotopia_pi_gpt4_rm_overfit.json" +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" +CUDA_VISIBLE_DEVICES=0 python inference_rm.py \ + --model_path $MODEL_PATH \ + --adapter_path "../rm_checkpoints_qwen2.5-7b/best-checkpoint" \ + --template_path "../evals/qwen2.5-7b.jinja" \ + --example_path "../data/sotopia_pi_gpt4_rm_overfit.json" diff --git a/scripts/inference_sft.sh b/scripts/inference_sft.sh index 70fe7d7..100e733 100644 --- a/scripts/inference_sft.sh +++ b/scripts/inference_sft.sh @@ -1,10 +1,11 @@ #!/bin/bash # Run the single evaluation script with your model checkpoint -CUDA_VISIBLE_DEVICES=9 python inference_sft.py \ - --model_path "/mnt/data_from_server1/models/Qwen2.5-7B-Instruct" \ - --adapter_path "/data/haofeiy2/sotopia-rl/sft_qwen25_7b/checkpoint-2500" \ - --template_path "/data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja" \ - --example_path "/data/haofeiy2/sotopia-rl/data/sotopia_pi_gpt4_sft_overfit.json" \ +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" +CUDA_VISIBLE_DEVICES=0 python inference_sft.py \ + --model_path $MODEL_PATH \ + --adapter_path "../sft_checkpoints_qwen2.5-7b/best-checkpoint" \ + --template_path "../evals/qwen2.5-7b.jinja" \ + --example_path "../data/sotopia_pi_gpt4_sft_overfit.json" \ --max_length 4096 \ --use_qlora diff --git a/scripts/train_grpo.sh b/scripts/train_grpo.sh index d6370d6..ddbb273 100644 --- a/scripts/train_grpo.sh +++ b/scripts/train_grpo.sh @@ -1,17 +1,18 @@ +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 accelerate launch \ - --config_file /data/disk0/sotopia-rl/scripts/accelerate_config_grpo.yaml \ + --config_file ./accelerate_config_grpo.yaml \ --main_process_port 29511 \ - /data/disk0/sotopia-rl/scripts/train_grpo.py \ - --model_name /data/disk0/models/Qwen2.5-7B-Instruct \ - --policy_adapter_path /data/disk0/sotopia-rl/sft_qwen25_7b_sft_round_1_bc_data_top_2/checkpoint-1500 \ - --reward_adapter_path /data/disk0/sotopia-rl/rm_reward_direct_default_without_that_n_error_as_the_end/checkpoint-4480 \ + ./train_grpo.py \ + --model_name $MODEL_PATH \ + --policy_adapter_path ../sft_checkpoints_qwen2.5-7b/best-checkpoint \ + --reward_adapter_path ../rm_checkpoints_qwen2.5-7b/best-checkpoint \ --learning_rate 5e-6 \ --per_device_train_batch_size 4 \ --per_device_eval_batch_size 4 \ --gradient_accumulation_steps 8 \ - --grpo_data_path /data/disk0/sotopia-rl/data/sotopia_pi_round1_qwen_sft_all_with_instruct_string.json \ - --template_path /data/disk0/sotopia-rl/evals/qwen2.5-7b.jinja \ + --grpo_data_path ../data/sotopia_grpo_data.json \ + --template_path ../evals/qwen2.5-7b.jinja \ --num_grpo_epochs 2 \ --use_lora_train_grpo \ --num_generations 16 \ - --output_dir /data/disk0/sotopia-rl/grpo_rm_reward_direct_default + --output_dir ../grpo_checkpoints_qwen2.5-7b diff --git a/scripts/train_ppo.py b/scripts/train_ppo.py deleted file mode 100644 index 7af4b70..0000000 --- a/scripts/train_ppo.py +++ /dev/null @@ -1,74 +0,0 @@ -import argparse -import os -import argparse -from sotopia_rl import SotopiaPPOTrainer -from accelerate import Accelerator - -if __name__ == '__main__': - parser = argparse.ArgumentParser(description="Train a model with PPO using a reward model.") - - parser.add_argument("--model_name", type=str, default="/data/models/gemma-2-2b-it", - help="Path to the model") - - parser.add_argument("--per_device_train_batch_size", type=int, default=1, - help="Batch size per device for training") - parser.add_argument("--per_device_eval_batch_size", type=int, default=1, - help="Batch size per device for evaluation") - parser.add_argument("--num_train_epochs", type=int, default=3, - help="Number of training epochs") - parser.add_argument("--num_ppo_epochs", type=int, default=4, - help="Number of PPO epochs per update") - parser.add_argument("--learning_rate", type=float, default=5e-6, - help="Learning rate for optimizer") - parser.add_argument("--gamma", type=float, default=0.99, - help="Discount factor") - parser.add_argument("--lam", type=float, default=0.95, - help="GAE lambda for advantage estimation") - parser.add_argument("--max_length", type=int, default=4096, - help="Maximum length of input sequences") - parser.add_argument("--num_mini_batches", type=int, default=1, - help="Mini batch size for PPO updates") - parser.add_argument("--gradient_accumulation_steps", type=int, default=1, - help="Number of steps to accumulate gradients before performing an update") - parser.add_argument("--val_ratio", type=float, default=0.05, - help="Ratio of validation data") - parser.add_argument("--response_length", type=int, default=128, - help="Maximum length of generated responses") - parser.add_argument("--local_rollout_forward_batch_size", type=int, default=16, - help="Batch size for local rollout forward pass") - - # Adapter parameters - parser.add_argument("--policy_adapter_path", type=str, default=None, - help="Path to policy model adapter") - parser.add_argument("--reward_adapter_path", type=str, default=None, - help="Path to reward model adapter") - parser.add_argument("--value_adapter_path", type=str, default=None, - help="Path to value model adapter") - parser.add_argument("--ref_adapter_path", type=str, default=None, - help="Path to reference model adapter") - - # Data and checkpoint paths - parser.add_argument("--ppo_data_path", type=str, required=True, - help="Path to the reward data file") - parser.add_argument("--template_path", type=str, required=True, - help="Path to the Jinja template file") - parser.add_argument("--output_dir", type=str, default="checkpoints", - help="Directory to save the best LoRA checkpoint") - parser.add_argument("--save_steps", type=int, default=5, - help="Number of steps between saving checkpoints") - parser.add_argument("--missing_eos_penalty", type=float, default=1.0, - help="Penalty for missing EOS token in generated") - - # Logging parameters - parser.add_argument("--wandb_project", type=str, default="ppo-model-training", - help="Wandb project name") - parser.add_argument("--wandb_run_name", type=str, default=None, - help="Wandb run name") - - parser.add_argument("--use_lora_train_ppo", action="store_true", - help="Use LoRA for training PPO") - - args = parser.parse_args() - accelerator = Accelerator() - trainer = SotopiaPPOTrainer(args, accelerator) - trainer.train() diff --git a/scripts/train_ppo.sh b/scripts/train_ppo.sh deleted file mode 100644 index b9d8f5e..0000000 --- a/scripts/train_ppo.sh +++ /dev/null @@ -1,21 +0,0 @@ -CUDA_VISIBLE_DEVICES=1,7,8 accelerate launch \ - --config_file /data/haofeiy2/sotopia-rl/scripts/accelerate_config.yaml \ - --main_process_port 29511 \ - /data/haofeiy2/sotopia-rl/scripts/train_ppo.py \ - --model_name /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ - --policy_adapter_path /data/haofeiy2/sotopia-rl/sft_qwen25_7b_sft_round_1_bc_data_top_2/checkpoint-1500 \ - --ref_adapter_path /data/haofeiy2/sotopia-rl/sft_qwen25_7b_sft_round_1_bc_data_top_2/checkpoint-1500 \ - --reward_adapter_path /data/haofeiy2/sotopia-rl/rm_reward_direct_default_without_that_n_error_as_the_end/checkpoint-4480 \ - --value_adapter_path /data/haofeiy2/sotopia-rl/rm_reward_direct_default_without_that_n_error_as_the_end/checkpoint-4480 \ - --learning_rate 1e-5 \ - --per_device_train_batch_size 1 \ - --per_device_eval_batch_size 1 \ - --gradient_accumulation_steps 1 \ - --num_mini_batches 1 \ - --ppo_data_path /data/haofeiy2/sotopia-rl/data/sotopia_pi_round1_qwen_sft_all_with_instruct_string.json \ - --template_path /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ - --num_ppo_epochs 2 \ - --num_train_epochs 5 \ - --gamma 0.99 \ - --lam 0.95 \ - --output_dir /data/haofeiy2/sotopia-rl/ppo_origin_qwen25_7b_reward_direct_default_no_goal_gpt-4o_without_goal_leak_with_sft_self_play_data_use_sotopia_pi_full_data_0408 \ No newline at end of file diff --git a/scripts/train_rm.sh b/scripts/train_rm.sh index d934b76..a501b8e 100644 --- a/scripts/train_rm.sh +++ b/scripts/train_rm.sh @@ -1,8 +1,9 @@ +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" CUDA_VISIBLE_DEVICES=5,6,7,8,9 accelerate launch \ - --config_file /data/haofeiy2/sotopia-rl/scripts/accelerate_config_rm.yaml \ + --config_file ./accelerate_config_rm.yaml \ --main_process_port 29500 \ - /data/haofeiy2/sotopia-rl/scripts/train_rm.py \ - --model_name /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ + ./scripts/train_rm.py \ + --model_name $MODEL_PATH \ --learning_rate 1e-5 \ --max_length 4096 \ --train_batch_size 1 \ @@ -10,22 +11,6 @@ CUDA_VISIBLE_DEVICES=5,6,7,8,9 accelerate launch \ --accumulation_steps 8 \ --num_epochs 30 \ --evaluation_steps 50 \ - --reward_data_path /data/haofeiy2/sotopia-rl/data/sotopia_pi_bc_episodes_reward_token_length.json \ - --template_path /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ - --checkpoint_dir /data/haofeiy2/sotopia-rl/rm_token_length - -CUDA_VISIBLE_DEVICES=5,6,7,8,9 accelerate launch \ - --config_file /data/haofeiy2/sotopia-rl/scripts/accelerate_config_rm.yaml \ - --main_process_port 29500 \ - /data/haofeiy2/sotopia-rl/scripts/train_rm.py \ - --model_name /mnt/data_from_server1/models/Qwen2.5-7B-Instruct \ - --learning_rate 1e-5 \ - --max_length 4096 \ - --train_batch_size 1 \ - --val_batch_size 1 \ - --accumulation_steps 8 \ - --num_epochs 3000 \ - --evaluation_steps 50 \ - --reward_data_path /data/haofeiy2/sotopia-rl/data/sotopia_pi_gpt4_rm_overfit.json \ - --template_path /data/haofeiy2/sotopia-rl/evals/qwen2.5-7b.jinja \ - --checkpoint_dir /data/haofeiy2/sotopia-rl/rm_overfit_test + --reward_data_path ../data/sotopia_pi_bc_episodes_rm.json \ + --template_path ../evals/qwen2.5-7b.jinja \ + --checkpoint_dir ../rm_checkpoints_qwen2.5-7b diff --git a/scripts/train_sft.sh b/scripts/train_sft.sh index 275ad00..f68fcca 100644 --- a/scripts/train_sft.sh +++ b/scripts/train_sft.sh @@ -1,8 +1,9 @@ +export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct" CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 accelerate launch \ - --config_file /mnt/data/sotopia-rl/scripts/accelerate_config_sft.yaml \ + --config_file ./accelerate_config_sft.yaml \ --main_process_port 29512 \ - /mnt/data/sotopia-rl/scripts/train_sft.py \ - --model_name /mnt/data/models/Qwen2.5-7B-Instruct \ + ./train_sft.py \ + --model_name $MODEL_PATH \ --learning_rate 1e-4 \ --max_length 4096 \ --train_batch_size 2 \ @@ -11,6 +12,6 @@ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 accelerate launch \ --num_epochs 500 \ --use_lora \ --evaluation_steps 5 \ - --sft_data_path /mnt/data/sotopia-rl/data/sft_round_1_bc_data_top_2_with_aligned_format_instruction_prompt_0509.json \ - --template_path /mnt/data/sotopia-rl/evals/qwen2.5-7b.jinja \ - --checkpoint_dir /mnt/data/sotopia-rl/sft_round_1_bc_data_top_2_with_aligned_format_instruction_prompt_weight_decay_0_0510 + --sft_data_path ../data/sft_round_1_bc_data_top_2_with_aligned_format_instruction_prompt_0509.json \ + --template_path ../evals/qwen2.5-7b.jinja \ + --checkpoint_dir ../sft_checkpoints_qwen2.5-7b \ diff --git a/sotopia_rl/rm_trainer.py b/sotopia_rl/rm_trainer.py index b168ae5..47acbd5 100644 --- a/sotopia_rl/rm_trainer.py +++ b/sotopia_rl/rm_trainer.py @@ -40,7 +40,9 @@ def __init__(self, args, accelerator, **kwargs): project=args.wandb_project, name=args.wandb_run_name, config={ - k: v for k, v in vars(args).items() if isinstance(v, (int, float, str)) + k: v + for k, v in vars(args).items() + if isinstance(v, (int, float, str)) }, ) @@ -70,7 +72,6 @@ def __init__(self, args, accelerator, **kwargs): logging_steps=1, save_steps=args.evaluation_steps, save_strategy="steps", - eval_steps=args.evaluation_steps, logging_dir="./logs", gradient_accumulation_steps=args.accumulation_steps, learning_rate=args.learning_rate, @@ -80,9 +81,10 @@ def __init__(self, args, accelerator, **kwargs): dataloader_num_workers=4, report_to="wandb", ddp_find_unused_parameters=False, + eval_strategy="steps", + label_names=["labels"], ) - collate_fn = ( train_dataset.dataset.collate_fn if hasattr(train_dataset, "dataset")