Multimodal Reward Hacking in Reinforcement Learning

Code and data release for the paper "Multimodal Reward Hacking in Reinforcement Learning", a systematic empirical study of reward hacking in multimodal large language model (MLLM) reinforcement learning.

Overview

This repository contains the code and data needed to reproduce the Safety-VQA experiments in the paper:

• Training data — curated Safety VQA JSONL splits (clean / ambiguous / evidence-augmented).
• Reward implementations — four reward functions covering outcome-only, answer-aware, evidence-aware (keyword-verifier), and an extreme "golden-template" stress-test reward.
• Training pipelines — GRPO / RLOO / DAPO / SFT recipes at four model scales (2B / 4B / 8B / 32B), based on ms-swift.
• Evaluation pipeline — proxy reward computation, pairwise oracle judge, and metric aggregation (RHR / NRFR / ROG / WR).
• Model outputs & judge results — every outputs_*.jsonl (raw inference) and *_judged.jsonl (oracle-scored) file produced by the checkpoints reported in the paper.

Trained model checkpoints are not included in this release due to size. The scripts under train/{2B,4B,8B,32B}/run_*.sh reproduce them from the base Qwen3-VL checkpoints released on Hugging Face.

Repository Layout

.
├── final_data/                # All training / eval JSONL datasets used in the paper
│   ├── sft_final.jsonl                #   SFT (safety, both clean + ambiguous)
│   ├── grpo_clean.jsonl               #   GRPO clean split
│   ├── grpo_ambiguous.jsonl           #   GRPO ambiguous split
│   ├── grpo_clean_with_evidence.jsonl #   + per-sample image_evidence (used by R3)
│   ├── grpo_ambiguous_with_evidence.jsonl
│   ├── eval_clean_final.jsonl
│   ├── eval_ambiguous_final.jsonl
│   ├── algo_sft.json, algo_grpo.json, algo_eval.json   # Algorithm-comparison subsets
│   ├── add_assistant_data.py          # Data curation: assistant replies via a VLM API
│   ├── add_image_evidence_r3.py       #                image_evidence for R3
│   ├── add_missing_r3_data.py         #                fill in missing R3 samples
│   └── data_process_v2.py             #                general preprocessing
│
├── rewards/                   # Reward functions (plug into ms-swift via --external_plugins)
│   ├── Reward1.py             #   R1 — outcome-only, deliberately hackable
│   ├── Reward2.py             #   R2 — answer-aware
│   ├── Reward3.py             #   R3 — evidence-aware, keyword verifier
│   └── Reward_extreme.py      #   Golden-template stress-test reward
│
├── train/                     # Per-scale training and eval scripts + eval outputs
│   ├── 2B/, 4B/, 8B/, 32B/
│   │   ├── run_sft.sh                        # SFT recipe
│   │   ├── run_grpo_clean_{r1,r2,r3}.sh      # GRPO on clean split
│   │   ├── run_grpo_ambiguous_{r1,r2,r3}.sh  # GRPO on ambiguous split
│   │   ├── run_algo_{grpo,rloo,dapo,sft}.sh  # (2B/4B/8B) algorithm comparison
│   │   ├── run_oracle_score.sh               # (2B only) Reward_extreme oracle scoring
│   │   ├── judge_score_multi.py              # VLM oracle judge (2B / 8B / 32B)
│   │   ├── merge.sh                          # (8B / 32B) LoRA-merge utility
│   │   ├── eval/                             # Per-checkpoint inference + judge outputs
│   │   │   ├── infer.py, judge.py            # (2B / 4B)
│   │   │   ├── analysis.py                   # (2B only)
│   │   │   ├── run_{2b,4b,8b,32b}_infer*.sh, run_*_judge.sh, run_*_eval.sh
│   │   │   ├── outputs_*.jsonl               # Raw model outputs (every checkpoint)
│   │   │   └── outputs_*_judged.jsonl        # Oracle-judged outputs
│   │   ├── eval_algo/                        # (2B / 4B / 8B) algorithm-comparison outputs
│   │   ├── proxy_reward/                     # Per-checkpoint proxy-reward .jsonl files
│   │   └── oracle_score/                     # Per-checkpoint oracle-score .jsonl files
│   ├── merge.sh                              # Top-level LoRA-merge helper
│   ├── run_algo_judge_all.py                 # Batch judging across all algorithms
│   └── algo_metrics_summary.json             # Cached algorithm-comparison numbers
│
└── eval_pipeline/             # Metric computation
    ├── compute_all_metrics.py       # RHR / NRFR / ROG / WR aggregation
    ├── compute_proxy_reward.py      # Recompute reward scores on model outputs
    └── pairwise_oracle_judge.py     # Pairwise scoring for ROG / WR

Requirements

• Python ≥ 3.10
• CUDA-enabled PyTorch
• ms-swift with vLLM backend (for training + inference)
• openai (Python SDK) — used to call any OpenAI-compatible VLM endpoint for the oracle judge and data-curation scripts
• requests, numpy, pandas, scipy

Install ms-swift and its dependencies according to their upstream documentation. Base MLLMs (Qwen3-VL 2B/4B/8B/32B Instruct) can be downloaded from Hugging Face.

Environment Variables

The code depends on the following environment variables. Placeholder defaults are baked in so files remain importable, but any real run must set them:

Variable	Purpose
WORKSPACE	Absolute path to this repository root — used by shell scripts as ${WORKSPACE}/...
MODEL_HUB	Directory where Qwen3-VL base checkpoints live (e.g. /path/to/hf-cache) — shell scripts refer to ${MODEL_HUB}/Qwen/Qwen3-VL-8B-Instruct etc.
QWEN_API_KEY, QWEN_API_BASE, QWEN_API_URL	OpenAI-compatible endpoint used by the text oracle judge and data-curation scripts. QWEN_API_URL is the full /v1/chat/completions URL; QWEN_API_BASE is <host>/v1.
QWEN_VL_API_KEY, QWEN_VL_API_BASE, QWEN_VL_MODEL	OpenAI-compatible endpoint used by the vision VLM judge (data curation).
QWEN_MODEL	Model name to request from the API (e.g. qwen3-vl-235b).
QWEN_MODEL_PATH	Optional — set when the API expects a local model path rather than a model name.

Example:

export WORKSPACE=$(pwd)
export MODEL_HUB=/path/to/hf-cache
export QWEN_API_KEY=sk-...
export QWEN_API_BASE=https://api.openai.com/v1                   # or any OpenAI-compatible endpoint
export QWEN_API_URL=$QWEN_API_BASE/chat/completions
export QWEN_MODEL=qwen3-vl-235b
export QWEN_VL_API_KEY=$QWEN_API_KEY
export QWEN_VL_API_BASE=$QWEN_API_BASE
export QWEN_VL_MODEL=$QWEN_MODEL

Reproducing the Paper's Experiments

1. Prepare data

Safety-VQA data (final_data/*.jsonl) is already curated and included. To re-run the data-curation VLM pipeline, use final_data/add_assistant_data.py, add_image_evidence_r3.py, or add_missing_r3_data.py (all take --input_dir / --output_dir).

2. Train an SFT baseline

cd train/2B      # (or 4B, 8B, 32B)
bash run_sft.sh

The SFT checkpoint path is referenced by every downstream GRPO / RLOO / DAPO recipe.

3. Run GRPO with one of the reward variants

cd train/2B
bash run_grpo_clean_r1.sh        # R1 (outcome-only) on clean split
bash run_grpo_clean_r2.sh        # R2 (answer-aware)
bash run_grpo_clean_r3.sh        # R3 (keyword verifier)
bash run_grpo_ambiguous_r1.sh    # ...on ambiguous split

Each script writes checkpoints under ${WORKSPACE}/train/2B/checkpoints_grpo_* at every save step.

4. Algorithm comparison (2B / 4B / 8B)

cd train/2B
bash run_algo_grpo.sh    # GRPO with Reward_extreme
bash run_algo_rloo.sh    # RLOO with Reward_extreme
bash run_algo_dapo.sh    # DAPO with Reward_extreme

5. Inference on eval sets

Every scale has scripts to sweep all reward variants and checkpoints:

cd train/2B/eval
bash run_2b_infer.sh          # writes outputs_*.jsonl into eval/
bash run_2b_judge.sh          # runs oracle judge on outputs_*.jsonl → outputs_*_judged.jsonl

6. Compute reported metrics

cd eval_pipeline
python compute_proxy_reward.py    # populates proxy_reward/*.jsonl if missing
python pairwise_oracle_judge.py   # pairwise ROG / WR scoring
python compute_all_metrics.py     # aggregates RHR / NRFR / ROG / WR

Notes on Data Availability

• All safety-VQA queries and images are re-sourced from public benchmarks; the curated splits are provided as JSONL with image_path fields written as ./images/safety_vl/<source>/.... Place the downloaded source images under ${WORKSPACE}/images/safety_vl/{VLSBench, MSTS, MM-SafetyBench, Omni-SafetyBench}/ matching those paths.
• The outputs_*.jsonl / *_judged.jsonl files preserve every model completion and every oracle judgement reported in the paper, so all metrics can be recomputed without re-running any inference.
• Trained checkpoints are excluded to keep the release small. Base Qwen3-VL Instruct weights are on Hugging Face; every training recipe here starts from a base + SFT stage that finishes in one 8-GPU day for 2B.