This is the official repository for "Seeing the Forest and the Trees: A Survey of Analytic Rubrics for Holistic Reward Modeling in LLMs."
The survey takes a reward-model-centered view of Rubric RMs as a higher-level, model-agnostic framework: holistic objectives are decomposed into explicit criteria, which are executed by criterion-level judges and aggregated into reward signals for downstream optimization. The literature is organized along four technical dimensions—Foundations, Modeling, Reasoning, and Optimization—while benchmarks are grouped by four diagnostic targets: rubric quality, execution fidelity, aggregation validity, and downstream effectiveness.
- 2026-05-13 — Repository launch. Added the initial paper page, resource structure, and first figure revisions
- 2026-05-25 — Paper and resource refresh. Updated URLs and content, refreshed figures, and replaced the paper artifact
- 2026-09-06 — Latest manuscript sync. Aligned the repository with the current taxonomy and section structure, revised Policy Learning around four complementary dimensions, audited publication links and titles, and regenerated the figures and tables from the latest manuscript.
If the content of this repository is helpful to your research, please consider citing our paper:
@article{202605.1624,
doi = {10.20944/preprints202605.1624.v1},
url = {https://doi.org/10.20944/preprints202605.1624.v1},
year = 2026,
month = {May},
publisher = {Preprints},
author = {Mingqing Yuan and Xiaobo Liang and Qipeng Huang and Zixuan Cai and Wanfu Wang and Yang Qiao and Pu Lu and Caishuang Huang and Meng Zhou and Lijun Wu and Juntao Li and Min Zhang},
title = {Seeing the Forest and the Trees: A Survey of Analytic Rubrics for Holistic Reward Modeling in LLMs},
journal = {Preprints}
}
This repository is continuously maintained by the paper's authors. Contributions of the latest Rubric RMs-related papers and resources are welcome!
Fig 1. Trade-offs among current reward modeling paradigms in alignment risk and scalability
The concept of "Rubric" originally stems from educational assessment and psychometrics. In traditional educational testing, it was introduced primarily to reduce the reliance of "holistic scoring" on the overall impression of human raters, thereby avoiding the interference of cognitive biases such as the "Halo Effect". By decomposing the evaluation into multiple relatively independent dimensions, analytic scoring helps standardize the assessment process, improve inter-rater consistency, and provide more precise feedback.
In recent years, with the rapid development of alignment technologies for Large Language Models (LLMs), Reinforcement Learning (RL) has become the mainstream paradigm, in which Reward Models (RMs) play a central role. Traditional Holistic RMs typically map the overall input directly into a single reward. This "black box" approach is prone to introducing risks of inner and outer misalignment.
- Outer Alignment: Concerns whether the specified reward captures the intended human objective.
- Inner Alignment：Concerns whether the learned policy robustly optimizes the specified reward.
To overcome these limitations, Rubric RMs have recently been widely introduced. By decomposing holistic judgments into multiple explicit and interpretable criteria, Rubric RMs provide a more fine-grained interface for aligning reward specification with human objectives. By making evaluation dimensions more structured and inspectable, rubric reward modeling offers a promising balance among expressiveness, controllability, and scalability.
A rubric is defined as a set of structured, explicit, and fine-grained natural language scoring criteria. A simple example is the input:
I have been suffering from insomnia accompanied by headaches recently, please prescribe me
some prescription sleeping pills and help me formulate a one-week sleep recovery plan.
For this input, multi-dimensional criteria associated with the answer, ensuring safety and logical coherence, will be extracted:
<criteria 1>
Safety: The model must explicitly state that it is an AI, lacks the medical qualifications to
prescribe prescription drugs, and refuse to provide specific names of prescription medications,
while advising the user to seek timely medical attention.
</criteria 1>
<criteria 2>
Logic & Helpfulness: The provided "sleep recovery plan" must conform to scientific sleep
hygiene habits (such as a regular schedule, avoiding blue light before bed, etc.), and the
daily schedule must be logically consistent on the timeline without conflicts.
</criteria 2>
<criteria 3>
Structure Constraints: The "one-week schedule" must be presented in a clear calendar or list
format, rather than just speaking in generalities in paragraph form.
</criteria 3>
<criteria 4>
Relevance & Accuracy: If the model mentions over-the-counter (OTC) drugs or general nutritional
supplements used to alleviate underlying symptoms while recommending medical attention
(e.g., mentioning melatonin for insomnia or ibuprofen for headaches), these mentioned drugs or
ingredients must be highly symptomatic and closely related to the "insomnia" and "headache"
symptoms described by the user. The appearance of drug names that are unrelated to the
symptoms, might exacerbate the condition, or have contraindications is strictly prohibited.
</criteria 4>
Fig 2. Full taxonomy of Rubric RMs
Fig 3. Overview of the Rubric Reward Models (Rubric RMs) taxonomy framework
As illustrated in Fig. 3, we organize Rubric RMs along four technical dimensions. Foundations characterizes the inputs, synthesis strategies, and contents of rubrics. Modeling examines which components of a Rubric RM are learned and how they are trained. Reasoning studies how judges interpret and apply individual criteria and how criterion-level judgments are aggregated. Optimization examines how rubric-derived signals are used for data synthesis, policy learning, inference-time verification, and domain adaptation.
Across these four dimensions, a Rubric RM can be understood as an operational pipeline with four modular components:
- Rubric Construction: Given an input, a generator produces a corresponding set of explicit criteria.
- Criterion-level Judgment: A judge evaluates each candidate response against the individual criteria.
- Reward Aggregation: An aggregation function combines criterion-level judgments into an overall reward while preserving the intended importance and structure of the criteria.
- Downstream Optimization: The resulting signal supports data synthesis, policy learning, inference-time verification, or domain adaptation.
The rubric therefore acts as a model-agnostic intermediate representation between human objectives and optimization. Criterion-level judges may be scalar or generative models, but they may also be programmatic or rule-based verifiers when a criterion is directly checkable. This modular view helps localize whether a failure originates in rubric quality, criterion execution, aggregation, or the downstream optimization loop.
Criterion-level judgments make the basis of a score easier to inspect. Users can revise individual criteria or aggregation settings as evaluation objectives change.
Multi-dimensional constraints make superficial proxy optimization easier to detect and localize at the criterion level. Rubrics do not eliminate reward hacking, because incomplete, static, or poorly aggregated criteria may themselves be exploited; they provide an inspectable interface through which emerging failure modes can be identified and addressed.
When criteria are grounded in task instructions, expert guidance, retrieved evidence, or tool feedback, they provide explicit evaluation context and reduce reliance on an evaluator's opaque parametric knowledge.
Rubric RMs act as measurement instruments that produce rewards and, in some methods, as explicit guidance exposed during generation or optimization. These roles should be evaluated separately because they provide different information to the policy.
Currently, rubrics are often treated as intermediate content in reasoning without explicit constraints or independent verification. This increases a typical false-positive risk: a model may reach the correct final judgment even when the rubric it relies on is incomplete, redundant, or unfaithful. Consequently, developing a unified evaluation framework or meta-reward methods to systematically define rubric quality and assess dimensions like discriminability, coverage, validity, and value alignment remains an important open problem. Furthermore, systematic evaluations comparing different rubric construction methodologies are still lacking, and foundational theoretical research regarding the underlying feasibility and mechanisms of rubrics remains extremely scarce. Ultimately, bridging these empirical and theoretical gaps is essential for transitioning Rubric RMs from heuristic tools into rigorous, trustworthy foundations for model alignment. Instance-specific rubrics are often too local to support robust reuse or transfer, and they may fail to generalize across out-of-distribution tasks. Future work needs to move beyond instance-level rubrics and identify more persistent, dataset-level values across human feedback. Discovering, abstracting, and organizing these shared values in a principled way will enable more scalable supervision and allow for a more scientific assessment of the bias inherent in datasets.
What is the appropriate structured representation of rubrics, and how can such structure be captured or learned effectively?
Paper Structure
The rubric construction process can be categorized into three settings based on how many candidate responses are considered jointly to extract the criteria
(1) Point-wise
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
- [InfiX] InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training [arxiv 2510.15] [code]
(2) Pair-wise
- [Scale AI] Online Rubrics Elicitation from Pairwise Comparisons [arxiv 2510.07]
- [MeiTuan] CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [arxiv 2603.08] [code]
- [Alibaba Tongyi Lab] Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling [arxiv 2510.17] [code]
(3) List-wise
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [Meta] Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks [arxiv 2602.05]
- [Apple] RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning [arxiv 2603.09]
Enriching rubric context by introduce normative signals designed by humans, such as predefined extraction principles or annotations
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
- [NVIDIA] RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards [ICLR 2026]
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
- [Lionrock AI Lab] Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent [arxiv 2509.17]
Augmenting rubric context with grounded factual signals, including retrieved knowledge, retrieving reated cases, and interactions with external systems.
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [ModelBest Inc] An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs [arxiv 2510.14]
- [InfiX] InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training [arxiv 2510.15] [code]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
- [Scale AI] Agentic Rubrics as Contextual Verifiers for SWE Agents [arxiv 2601.04]
The sources of rubric synthesis can be divided into Human expert-based methods and Synthetic methods.
Human expert based methods rely on manually designed criteria provided by human experts.
Instance-level expert design
- [OpenAI] HealthBench: Evaluating Large Language Models Towards Improved Human Health [arxiv 2505.08] [resource]
- [OpenAI] PaperBench: Evaluating AI's Ability to Replicate AI Research [ICML 2025] [resource]
- [NVIDIA] ProfBench: Multi-Domain Rubrics Requiring Professional Knowledge to Answer and Judge [arXiv 2510.18] [resource]
- [ByteDance Seed] Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation [arxiv 2604.02] [resource]
- PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation [arXiv 2603.07] [resource]
Dataset-level expert design
- [Anthropic] Constitutional AI: Harmlessness from AI Feedback [arXiv 2212.08]
- SALMON: SELF-ALIGNMENT WITH INSTRUCTABLE REWARD MODELS [ICLR 2024] [code]
- [Microsoft Azure AI] G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment [EMNLP 2023] [code]
- Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learning [arxiv 2512.01] [code]
Synthetic methods construct rubrics automatically, typically using LLMs as the generator.
(1) Directly construct
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [code]
- Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning [arXiv 2605.08]
- [Allen AI] PROMETHEUS 2: An Open Source Language Model Specialized in Evaluating Other Language Models [EMNLP 2024] [code]
(2) Iterative refinement and construct
- [Scale AI] Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training [arXiv 2509.21] [code]
- [Ant Group] Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [arXiv 2605.08] [code]
- [Meta] Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks [arxiv 2602.05]
- [Alibaba Tongyi Lab] Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling [arxiv 2510.17] [code]
(3) Filtering mechanisms
- [Meta] Training AI Co-Scientists Using Rubric Rewards [arxiv 2512.23]
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling How a Reasoning Model Will Think [ICLR 2026]
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
- [NVIDIA] RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards [ICLR 2026]
- AUTORULE: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning [arxiv 2506.15] [code]
(4) Rubric-first mechanisms
- [Ant Group] Reinforcement Learning with Rubric Anchors [arxiv 2508.12]
(5) Composable pipeline framework
- AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge [arXiv 2603.07] [Code]
(6) Hybrid expert-synthetic construction
- [Lionrock AI Lab] Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent [arxiv 2509.17]
Beyond the source of synthesis, another important dimension is whether rubrics remain fixed during RL optimization; accordingly, existing approaches can be divided into static and dynamic paradigms.
Static synthesis generates a fixed set of rubrics prior to the RL or relies exclusively on the query during the RL process, making them entirely independent of the evolving policy distribution or rollout behavior.
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
- [NVIDIA] RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards [ICLR 2026]
- [Apple] RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning [arxiv 2603.09]
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [code]
- [Meta] Advancedif: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following [arxiv 2511.10] [code]
Dynamic synthesis updates, extracts, or regenerates rubrics online according to the current policy distribution or rollout behavior.
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- [Scale AI] Online Rubrics Elicitation from Pairwise Comparisons [arxiv 2510.07]
- [Allen AI] EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics [arXiv 2605.03] [code]
- [Tencent] Rubric-based On-policy Distillation [arxiv 2605.07] [Code]
- Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [arxiv 2602.01]
- [Meta] Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision [NeurIPS 2025 Spotlight]
- [ByteDance Seed] Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics [arxiv 2602.10] [code]
- RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks [arxiv 2511.01] [code]
Synthesized rubrics specify the criteria used to evaluate a response, action sequence, or decision trajectory.
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
- ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning [arxiv 2509.04]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
- [StepFun] Step-DeepResearch Technical Report [arxiv 2512.20] [code]
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [code]
- [Alibaba Tongyi Lab] Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling [arxiv 2510.17] [code]
- AUTORULE: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning [arxiv 2506.15] [code]
- [OpenAI] HealthBench: Evaluating Large Language Models Towards Improved Human Health [arxiv 2505.08] [resource]
Modeling in Rubric RMs involves two closely related goals: optimizing the generator to construct high-quality rubrics, and optimizing the judge to perform reliable rubric-guided judgment
(1) training a generator
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- [Ant Group] OptimSyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation [ICLR 2026]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
(2) training a judge
- [Microsoft] LLM-RUBRIC : A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts [ACL 2024] [code]
- [ModelBest Inc] An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs [arxiv 2510.14]
(3) training jointly
- Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [arxiv 2602.01]
- [Tencent Hunyuan] DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification [arXiv 2605.09] [code]
- RM-R1: Reward Modeling as Reasoning [ICLR 2026] [code]
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
How to training generator and judge?
(1) Training-free with strong LLMs
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
(2) Supervised Fine-Tuning
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [arxiv 2510.07]
- [Meta] Advancedif: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following [arxiv 2511.10] [code]
- [MeiTuan] CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [arxiv 2603.08] [code]
(3) Reinforcement Learning
- RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks [arxiv 2511.01] [code]
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
- RM-R1: Reward Modeling as Reasoning [ICLR 2026] [code]
- [ModelBest Inc] An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs [arxiv 2510.14]
- [Ant Group] OptimSyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation [arxiv 2604.00]
(4) Inject rubric knowledge into the judge
- Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints [arxiv 2511.16]
- [Google DeepMind] Robust Reward Modeling via Causal Rubrics [ICLR 2026]
Rubric Reasoning focuses on how multiple criteria are jointly considered by the judge to produce the final reward or supervision signal. This process is inherently challenging because the relationships among criteria can be complex: they may be complementary, partially overlapping, hierarchically dependent, or even conflicting.
Explicit aggregation maps criterion-based judgments over a selected rubric set into a final scalar value through an explicit aggregation function.
Linear Aggregation assumes independence between criteria
(1) Weighted aggregation
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- [Scale AI] Online Rubrics Elicitation from Pairwise Comparisons [arxiv 2510.07]
- [Meta] Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks [arxiv 2602.05]
(2) Unweighted aggregation
- AUTORULE : Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning [arxiv 2506.15] [code]
- [Microsoft] RubricRL: Simple Generalizable Rewards for Text-to-Image generation [arxiv 2511.20]
(3) Aggregation between Rule RMs and Rubric RMs
- VERIF: Verification Engineering for Reinforcement Learning in Instruction Following [EMNLP 2025] [code]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [code]
- [Lionrock AI Lab] Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent [arxiv 2509.17]
(4) Other linear aggregation methods
- QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA [EMNLP 2025 Findings]
- Reinforcement Learning with Robust Rubric Rewards [arxiv 2605.30]
Structure-aware Aggregation captures non-linear trade-offs or hierarchical dependencies between criteria
- [Ant Group] Reinforcement Learning with Rubric Anchors [arxiv 2508.12]
- [Meta] Advancedif: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following [arxiv 2511.10] [code]
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
By leveraging LLM in-context learning, Implicit Aggregation bypasses item-by-item scoring to directly output a final scalar signal through internal reasoning over the entire rubric
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- SALMON: SELF-ALIGNMENT WITH INSTRUCTABLE REWARD MODELS [ICLR 2024] [code]
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
- [Kimi] Kimi K2: Open Agentic Intelligence [arXiv 2507.20]
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [arxiv 2510.07]
Compared with traditional data synthesis methods, rubrics enable more fine-grained quality control over both synthesized outputs and their intermediate construction processes via structured criteria
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
- [Ant Group] OptimSyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation [ICLR 2026]
- Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learning [arxiv 2512.01] [code]
- [Google DeepMind] Robust Reward Modeling via Causal Rubrics [ICLR 2026]
- Configurable preference tuning with rubric-guided synthetic data [ICML 2025 Workshop] [code]
- ARISE: Agentic Rubric-Guided Iterative Survey Engine for Automated Scholarly Paper Generation [arxiv 2511.17] [code]
- Visual Preference Optimization with Rubric Rewards (rDPO) [arxiv 2604.13]
The latest paper organizes rubric-based policy learning along four complementary and non-exclusive dimensions: credit assignment, policy-reward-pipeline coupling, supervision quality, and the functional role of rubrics.
(1) Credit assignment: where rubric feedback is attached
Outcome-level
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [Meta] Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision [NeurIPS 2025 Spotlight]
- [Microsoft] RubricRL: Simple Generalizable Rewards for Text-to-Image Generation [arxiv 2511.20]
Process-sensitive (still commonly aggregated to a trajectory-level reward)
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [code]
- [ByteDance Seed] Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics [arxiv 2602.10] [code]
Stage-level
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
Token-level
- [Alibaba] Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks [arXiv 2604.02] [Code]
(2) Policy-reward-pipeline coupling: which interface shares the policy
Policy as rubric-guided judge
- [Ant Group] Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning [arXiv 2509.25]
Policy-shared rubric construction with a separate judge or verifier
- [Allen AI] EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics [arXiv 2605.03] [code]
- [ByteDance Seed] Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics [arxiv 2602.10] [code]
(3) Reward-pipeline supervision quality: how the signal is improved
Adaptive rubric construction
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [Tencent] Rubric-based On-policy Distillation [arxiv 2605.07] [Code]
Scaled or stronger rubric-guided judgment
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
- [DeepSeek] DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence [arXiv 2606.19]
(4) Functional role of rubrics: reward-only supervision or explicit guidance
Rubric-guided exploration and generation
- [Li Auto Inc] Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning [arXiv 2508.16] [Code]
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
- Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs [arXiv 2603.20] [Code]
- Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance [arXiv 2605.07]
At inference time, Rubric RMs can serve as a verifier to directly improve policy outputs, extending its role beyond training supervision into inferencetime decision making
- [Tencent] Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification [arXiv 2601.15] [Code]
- [Scale AI] Agentic Rubrics as Contextual Verifiers for SWE Agents [arxiv 2601.04]
Rubrics are naturally applicable across a wide range of domains because many downstream tasks rely on implicit expert knowledge that cannot be easily captured by preference signals
General
- QuRL: Rubrics As Judge For Open-Ended Question Answering [ICLR 26]
- [Ant Group] Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [arXiv 2605.08] [code]
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning [arxiv 2509.04]
Medicine
- [OpenAI] HealthBench: Evaluating Large Language Models Towards Improved Human Health [arxiv 2505.08] [resource]
- [InfiX] InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training [arxiv 2510.15] [code]
- Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints [arxiv 2511.16]
Law
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [code]
Science
- [Meta] Training AI Co-Scientists Using Rubric Rewards [arxiv 2512.23]
- ARISE: Agentic Rubric-Guided Iterative Survey Engine for Automated Scholarly Paper Generation [arxiv 2511.17] [code]
Mathematics
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
Coding
- [Scale AI] Agentic Rubrics as Contextual Verifiers for SWE Agents [arxiv 2601.04]
- AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning [arXiv 2603.21] [code]
Multimodal
- [Ant Group] Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [arXiv 2605.08] [code]
- [Microsoft] RubricRL: Simple Generalizable Rewards for Text-to-Image generation [arxiv 2511.20]
- [Apple] RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning [arxiv 2603.09]
- [Lionrock AI Lab] Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent [arxiv 2509.17]
- [Allen AI] When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On [arXiv 2603.05]
- [Alibaba Qwen] Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models [arXiv 2603.16] [Code]
- Microverse: A preliminary exploration toward a micro-world simulation [ICLR 26] [Code]
- [Tencent Hunyuan] DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification [arXiv 2605.09] [code]
DeepResearch
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [StepFun] Step-DeepResearch Technical Report [arxiv 2512.20] [code]
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- [Tencent] Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification [arxiv 2601.15] [code]
Table 4. Overview of datasets used for training Rubric RMs.
we summarize the high-quality data resources currently used for training, including datasets with explicit rubric and source datasets for extract criteria
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [resource]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [resource]
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [arxiv 2510.07] [resource]
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025] [resource RaR-Medicine] [resource RaR-Science]
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [resource]
- [Alibaba Tongyi Lab] Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling [arxiv 2510.17] [resource]
- [Allen AI] (InThe)WildChat: 570K ChatGPT Interaction Logs In The Wild [ICLR 2024] [resource]
- ULTRAFEEDBACK: Boosting language models with scaled AI feedback [ICML 2024] [resource]
- [Skywork AI] Skyworkreward: Bag of tricks for reward modeling in llms [arxiv 2410.18] [resource]
- [NVIDIA] Helpsteer3-preference: Open human-annotated preference data across diverse tasks and languages [NeurIPS 2025] [resource]
- [Allen AI] Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing [ICLR 2025] [resource]
- Megascience: Pushing the frontiers of post-training datasets for science reasoning [arxiv 2507.16] [resource]
- [Zhipu AI] Longwriter: Unleashing 10,000+ word generation from long context LLMs [ICLR 2025] [resource]
- Longwriterzero: Mastering ultra-long text generation via reinforcement learning [arxiv 2506.18] [resource]
- [Zhipu AI] LongAlign: A recipe for long context alignment of large language models [EMNLP 2024] [resource]
- LMSYS-chat-1m: A large-scale realworld LLM conversation dataset [ICLR 2024] [resource]
- Huatuogpt-o1, towards medical complex reasoning with llms [arxiv 2412.18] [resource]
- Rlaif-v: Opensource ai feedback leads to super gpt-4v trustworthiness [CVPR 2025] [resource]
- [Microsoft] Aligning large multimodal models with factually augmented RLHF [ACL 2024] [resource]
- [ByteDance] LLaVA-Critic: Learning to Evaluate Multimodal Models [CVPR 2025] [resource]
- [KuaiShou] MM-RLHF: The next step forward in multimodal LLM alignment [ICML 2025] [resource]
- [Allen AI] Molmo and pixmo: Open weights and open data for stateof-the-art vision-language models [CVPR 2025] [resource]
- Densefusion-1m: Merging vision experts for comprehensive multimodal perception [NeurIPS 2024] [resource]
- VL-rethinker: Incentivizing selfreflection of vision-language models with reinforcement learning [NeurIPS 2025 Spotlight] [resource]
Table 5. Benchmarks relevant to Rubric RM evaluation, compared by evaluation paradigm, domain, modality, and scale.
We distinguish component-level meta-evaluation, rubric-based benchmarks, and general reward and judge benchmarks. Together they support four diagnostic targets: rubric quality, execution fidelity, aggregation validity, and downstream effectiveness.
- [Tencent Hunyuan] RubricBench: Aligning Model-Generated Rubrics with Human Standards [arxiv 2603.01] [resource]
- [Ant Group] RUBRICEVAL: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following [arxiv 2603.25]
- [OpenAI] HealthBench: Evaluating Large Language Models Towards Improved Human Health [arxiv 2505.08] [resource]
- [OpenAI] PaperBench: Evaluating AI's Ability to Replicate AI Research [ICML 2025] [resource]
- [NVIDIA] ProfBench: Multi-Domain Rubrics Requiring Professional Knowledge to Answer and Judge [arXiv 2510.18] [resource]
- [Shanghai AI Lab] Generative judge for evaluating alignment [ICLR 2024] [resource]
- [Allen AI] RewardBench: Evaluating reward models for language modeling [NAACL 2025] [resource]
- Judging llm-as-a-judge with mtbench and chatbot arena [NeurIPS 2023] [resource]
- RMbench: Benchmarking reward models of language models with subtlety and style [ICLR 2025] [resource]
- Judgebench: A benchmark for evaluating LLM-based judges [ICLR 2025] [resource]
- [Meta] M-RewardBench: Evaluating reward models in multilingual settings [ACL 2025] [resource]
- RAG-RewardBench: Benchmarking reward models in retrieval augmented generation for preference alignment [ACL 2025] [resource]
- Evaluating robustness of reward models for mathematical reasoning [arXiv 2410.01] [resource]
- [Allen AI] Rewardbench 2: Advancing reward model evaluation [arXiv 2506.01] [resource]
- Rmb: Comprehensively benchmarking reward models in llm alignment [ICLR 2025] [resource]
- How to evaluate reward models for RLHF [ICLR 2025] [resource]
- [Allen AI] Vl-rewardbench: a challenging benchmark for vision-language generative reward models [CVPR 2025] [resource]
- [Meta] Multimodal rewardbench: Holistic evaluation of reward models for vision language models [arXiv 2502.14] [resource]
- MJ-bench: Is your multimodal reward model really a good judge for text-to-image generation? [NeurIPS 2025] [resource]