-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
219 lines (198 loc) · 24.4 KB
/
Copy pathindex.html
File metadata and controls
219 lines (198 loc) · 24.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
<!DOCTYPE html>
<html
xmlns:og="http://opengraphprotocol.org/schema/"
xmlns:fb="http://www.facebook.com/2008/fbml"
lang="en-US"
class="
wf-orpheuspro-n4-active wf-orpheuspro-n5-active wf-orpheuspro-n7-active wf-orpheuspro-i4-active
wf-adobegaramondpro-n4-active wf-adobegaramondpro-n7-active wf-adobegaramondpro-i4-active wf-adobegaramondpro-i7-active
wf-active yui3-js-enabled js flexbox
canvas canvastext webgl no-touch
hashchange history draganddrop rgba
hsla multiplebgs backgroundsize borderimage
borderradius boxshadow textshadow opacity
cssanimations csscolumns cssgradients cssreflections
csstransforms no-csstransforms3d csstransitions video
audio svg inlinesvg svgclippaths
"
style="scroll-behavior: smooth;"
>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width, initial-scale=1">
<base href=".">
<title>Position: RL Should Be Used to Adjust Foundation Models, NOT Abused</title>
<meta http-equiv="Accept-CH" content="Sec-CH-UA-Platform-Version, Sec-CH-UA-Model">
<link rel="icon" type="image/x-icon" href="https://assets.squarespace.com/universal/default-favicon.ico">
<meta property="og:type" content="article">
<meta itemprop="datePublished" content="2025-07-15T18:29:13-0700">
<meta itemprop="dateModified" content="2026-06-09T00:00:00+0800">
<meta itemprop="headline" content="Position: RL Should Be Used to Adjust Foundation Models, NOT Abused">
<meta name="twitter:card" content="summary">
<meta name="description" content="A blog summary of our ICML 2026 position paper arguing that reinforcement learning should usually adjust foundation models after pretraining and cold-start supervision, not replace them as the default path to capability creation.">
<link rel="preconnect" href="https://images.squarespace-cdn.com/">
<link rel="preconnect" href="https://use.typekit.net/" crossorigin="">
<link rel="preconnect" href="https://p.typekit.net/" crossorigin="">
<style>@keyframes fonts-loading { 0%, 99% { color: transparent; } } html.wf-loading * { animation: fonts-loading 3s; }</style>
<link rel="stylesheet" type="text/css" href="./assets/website.components.html.styles.css">
<link rel="stylesheet" type="text/css" href="./assets/site.css">
<link rel="stylesheet" type="text/css" href="./assets/custom.css">
<link rel="stylesheet" type="text/css" href="./assets/static.css">
<style type="text/css">@font-face{font-family:orpheus-pro;src:url(https://use.typekit.net/af/2baa0b/00000000000000007735ebd5/31/l?subset_id=2&fvd=n4&v=3) format("woff2"),url(https://use.typekit.net/af/2baa0b/00000000000000007735ebd5/31/d?subset_id=2&fvd=n4&v=3) format("woff"),url(https://use.typekit.net/af/2baa0b/00000000000000007735ebd5/31/a?subset_id=2&fvd=n4&v=3) format("opentype");font-weight:400;font-style:normal;font-stretch:normal;font-display:auto;}@font-face{font-family:orpheus-pro;src:url(https://use.typekit.net/af/ab0afa/00000000000000007735ebd8/31/l?subset_id=2&fvd=n5&v=3) format("woff2"),url(https://use.typekit.net/af/ab0afa/00000000000000007735ebd8/31/d?subset_id=2&fvd=n5&v=3) format("woff"),url(https://use.typekit.net/af/ab0afa/00000000000000007735ebd8/31/a?subset_id=2&fvd=n5&v=3) format("opentype");font-weight:500;font-style:normal;font-stretch:normal;font-display:auto;}@font-face{font-family:orpheus-pro;src:url(https://use.typekit.net/af/64ac83/00000000000000007735ebd7/31/l?subset_id=2&fvd=n7&v=3) format("woff2"),url(https://use.typekit.net/af/64ac83/00000000000000007735ebd7/31/d?subset_id=2&fvd=n7&v=3) format("woff"),url(https://use.typekit.net/af/64ac83/00000000000000007735ebd7/31/a?subset_id=2&fvd=n7&v=3) format("opentype");font-weight:700;font-style:normal;font-stretch:normal;font-display:auto;}@font-face{font-family:orpheus-pro;src:url(https://use.typekit.net/af/1b6cd4/00000000000000007735ebd6/31/l?subset_id=2&fvd=i4&v=3) format("woff2"),url(https://use.typekit.net/af/1b6cd4/00000000000000007735ebd6/31/d?subset_id=2&fvd=i4&v=3) format("woff"),url(https://use.typekit.net/af/1b6cd4/00000000000000007735ebd6/31/a?subset_id=2&fvd=i4&v=3) format("opentype");font-weight:400;font-style:italic;font-stretch:normal;font-display:auto;}@font-face{font-family:adobe-garamond-pro;src:url(https://use.typekit.net/af/1a756d/00000000000000007735d927/31/l?subset_id=2&fvd=n4&v=3) format("woff2"),url(https://use.typekit.net/af/1a756d/00000000000000007735d927/31/d?subset_id=2&fvd=n4&v=3) format("woff"),url(https://use.typekit.net/af/1a756d/00000000000000007735d927/31/a?subset_id=2&fvd=n4&v=3) format("opentype");font-weight:400;font-style:normal;font-stretch:normal;font-display:auto;}@font-face{font-family:adobe-garamond-pro;src:url(https://use.typekit.net/af/a4e080/00000000000000007735d91e/31/l?subset_id=2&fvd=n7&v=3) format("woff2"),url(https://use.typekit.net/af/a4e080/00000000000000007735d91e/31/d?subset_id=2&fvd=n7&v=3) format("woff"),url(https://use.typekit.net/af/a4e080/00000000000000007735d91e/31/a?subset_id=2&fvd=n7&v=3) format("opentype");font-weight:700;font-style:normal;font-stretch:normal;font-display:auto;}@font-face{font-family:adobe-garamond-pro;src:url(https://use.typekit.net/af/c45313/00000000000000007735d924/31/l?subset_id=2&fvd=i4&v=3) format("woff2"),url(https://use.typekit.net/af/c45313/00000000000000007735d924/31/d?subset_id=2&fvd=i4&v=3) format("woff"),url(https://use.typekit.net/af/c45313/00000000000000007735d924/31/a?subset_id=2&fvd=i4&v=3) format("opentype");font-weight:400;font-style:italic;font-stretch:normal;font-display:auto;}@font-face{font-family:adobe-garamond-pro;src:url(https://use.typekit.net/af/7b6e7b/00000000000000007735d921/31/l?subset_id=2&fvd=i7&v=3) format("woff2"),url(https://use.typekit.net/af/7b6e7b/00000000000000007735d921/31/d?subset_id=2&fvd=i7&v=3) format("woff"),url(https://use.typekit.net/af/7b6e7b/00000000000000007735d921/31/a?subset_id=2&fvd=i7&v=3) format("opentype");font-weight:700;font-style:italic;font-stretch:normal;font-display:auto;}</style>
<link rel="stylesheet" type="text/css" href="./assets/6d80fa01db20c41f-min.en-US.css">
<link rel="stylesheet" type="text/css" href="./assets/blog-custom.css">
</head>
<body
id="item-687700442691c71a7f502537"
class="
tweak-blog-alternating-side-by-side-width-full tweak-blog-alternating-side-by-side-image-aspect-ratio-11-square tweak-blog-alternating-side-by-side-text-alignment-left tweak-blog-alternating-side-by-side-read-more-style-show
tweak-blog-alternating-side-by-side-image-text-alignment-middle tweak-blog-alternating-side-by-side-delimiter-bullet tweak-blog-alternating-side-by-side-meta-position-top tweak-blog-alternating-side-by-side-primary-meta-categories
tweak-blog-alternating-side-by-side-secondary-meta-date tweak-blog-alternating-side-by-side-excerpt-show image-block-poster-text-alignment-left image-block-card-content-position-center
image-block-card-text-alignment-left image-block-overlap-content-position-center image-block-overlap-text-alignment-left image-block-collage-content-position-center
image-block-collage-text-alignment-left image-block-stack-text-alignment-left form-field-style-solid form-field-shape-square
form-field-border-all form-field-checkbox-type-icon form-field-checkbox-fill-solid form-field-checkbox-color-inverted
form-field-checkbox-shape-square form-field-checkbox-layout-stack form-field-radio-type-icon form-field-radio-fill-solid
form-field-radio-color-normal form-field-radio-shape-pill form-field-radio-layout-stack form-field-survey-fill-solid
form-field-survey-color-normal form-field-survey-shape-pill form-field-hover-focus-outline form-submit-button-style-label
tweak-global-animations-complexity-level-detailed tweak-global-animations-animation-style-fade tweak-global-animations-animation-type-none tweak-global-animations-animation-curve-ease
tweak-blog-masonry-width-full tweak-blog-masonry-text-alignment-left tweak-blog-masonry-primary-meta-categories tweak-blog-masonry-secondary-meta-date
tweak-blog-masonry-meta-position-top tweak-blog-masonry-read-more-style-show tweak-blog-masonry-delimiter-space tweak-blog-masonry-image-placement-above
tweak-blog-masonry-excerpt-show tweak-portfolio-grid-overlay-width-full tweak-portfolio-grid-overlay-height-large tweak-portfolio-grid-overlay-image-aspect-ratio-11-square
tweak-portfolio-grid-overlay-text-placement-center tweak-portfolio-grid-overlay-show-text-after-hover tweak-blog-single-column-width-inset tweak-blog-single-column-text-alignment-left
tweak-blog-single-column-image-placement-above tweak-blog-single-column-delimiter-dash tweak-blog-single-column-read-more-style-hide tweak-blog-single-column-primary-meta-author
tweak-blog-single-column-secondary-meta-date tweak-blog-single-column-meta-position-top tweak-blog-single-column-content-title-only header-width-inset
tweak-fixed-header-style-basic tweak-blog-item-width-narrow tweak-blog-item-text-alignment-left tweak-blog-item-meta-position-below-title
tweak-blog-item-show-categories tweak-blog-item-show-date tweak-blog-item-show-author-name tweak-blog-item-delimiter-bullet
tweak-blog-side-by-side-width-full tweak-blog-side-by-side-image-placement-left tweak-blog-side-by-side-image-aspect-ratio-11-square tweak-blog-side-by-side-primary-meta-categories
tweak-blog-side-by-side-secondary-meta-date tweak-blog-side-by-side-meta-position-top tweak-blog-side-by-side-text-alignment-left tweak-blog-side-by-side-image-text-alignment-middle
tweak-blog-side-by-side-read-more-style-show tweak-blog-side-by-side-delimiter-bullet tweak-blog-side-by-side-excerpt-show primary-button-style-solid
primary-button-shape-square secondary-button-style-solid secondary-button-shape-square tertiary-button-style-solid
tertiary-button-shape-square tweak-events-stacked-width-full tweak-events-stacked-height-large tweak-events-stacked-show-thumbnails
tweak-events-stacked-thumbnail-size-32-standard tweak-events-stacked-date-style-with-text tweak-events-stacked-show-time tweak-events-stacked-show-location
tweak-events-stacked-show-excerpt tweak-blog-basic-grid-width-full tweak-blog-basic-grid-image-aspect-ratio-32-standard tweak-blog-basic-grid-text-alignment-center
tweak-blog-basic-grid-delimiter-bullet tweak-blog-basic-grid-image-placement-above tweak-blog-basic-grid-read-more-style-show tweak-blog-basic-grid-primary-meta-categories
tweak-blog-basic-grid-secondary-meta-date tweak-blog-basic-grid-excerpt-show tweak-portfolio-grid-basic-width-full tweak-portfolio-grid-basic-height-large
tweak-portfolio-grid-basic-image-aspect-ratio-11-square tweak-portfolio-grid-basic-text-alignment-left tweak-portfolio-grid-basic-hover-effect-fade header-overlay-alignment-center
tweak-portfolio-index-background-link-format-stacked tweak-portfolio-index-background-width-full tweak-portfolio-index-background-height-large tweak-portfolio-index-background-vertical-alignment-middle
tweak-portfolio-index-background-horizontal-alignment-center tweak-portfolio-index-background-delimiter-none tweak-portfolio-index-background-animation-type-fade tweak-portfolio-index-background-animation-duration-medium
tweak-portfolio-hover-follow-layout-inline tweak-portfolio-hover-follow-delimiter-bullet tweak-portfolio-hover-follow-animation-type-fade tweak-portfolio-hover-follow-animation-duration-fast
tweak-portfolio-hover-static-layout-inline tweak-portfolio-hover-static-front tweak-portfolio-hover-static-delimiter-hyphen tweak-portfolio-hover-static-animation-type-fade
tweak-portfolio-hover-static-animation-duration-fast hide-opentable-icons opentable-style-dark tweak-product-quick-view-button-style-floating
tweak-product-quick-view-button-position-bottom tweak-product-quick-view-lightbox-excerpt-display-truncate tweak-product-quick-view-lightbox-show-arrows tweak-product-quick-view-lightbox-show-close-button
tweak-product-quick-view-lightbox-controls-weight-light native-currency-code-usd collection-type-blog-single-column view-item
collection-layout-default collection-633a6d155b5e672d036e0d6d mobile-style-available sqs-seven-one
seven-one-global-animations
"
tabindex="-1"
data-animation-state="booted"
>
<div id="siteWrapper" class="clearfix site-wrapper">
<main id="page" class="container" role="main">
<article class="sections" id="sections" data-page-sections="633a6d155b5e672d036e0d76">
<section
data-test="page-section"
data-section-theme="light-bold"
class="page-section content-collection full-bleed-section collection-type-blog-single-column background-width--full-bleed section-height--medium content-width--wide horizontal-alignment--center vertical-alignment--middle light-bold"
data-sqsp-section="blog-item"
style="padding-top: 135.203px;"
>
<div class="section-border">
<div class="section-background"></div>
</div>
<div class="content-wrapper">
<div class="content">
<div class="blog-item-wrapper blog-single-column-item-wrapper" data-content-field="main-content" data-item-id="">
<article id="article-" class="h-entry entry hentry post-type-">
<div class="blog-item-inner-wrapper">
<div class="blog-item-top-wrapper">
<div class="blog-item-title">
<h1 class="entry-title entry-title--large p-name" itemprop="headline" data-content-field="title">Position: <span class="title-highlight">RL</span> Should Be Used to <span class="title-highlight">Adjust Foundation Models</span>, <span class="title-highlight-strong">NOT Abused</span></h1>
<div class="paper-venue">ICML 2026 Position</div>
</div>
<div class="blog-item-meta-wrapper">
<div class="blog-item-author-date-wrapper" data-animation-role="date">
<div class="blog-meta-item blog-meta-item--author p-author author" data-content-field="author">
<span class="paper-authors">Ting Huang<sup>*</sup>, <a href="https://steve-zeyu-zhang.github.io/">Zeyu Zhang</a><sup>*†</sup>, Hao Tang<sup>‡</sup></span>
<span class="paper-affiliation">School of Computer Science, Peking University</span>
<span class="paper-notes"><sup>*</sup>Equal contribution. <sup>†</sup>Project lead. <sup>‡</sup>Corresponding author: bjdxtanghao@gmail.com.</span>
<a class="paper-link-button" href="https://openreview.net/pdf?id=Q83EEeC6ce">Paper</a>
</div>
</div>
</div>
</div>
<div class="blog-item-content-wrapper">
<div class="blog-item-content e-content">
<div class="sqs-layout sqs-grid-12 columns-12" data-layout-label="Post Body" data-type="item" id="item-687700442691c71a7f502537">
<div class="row sqs-row">
<div class="col sqs-col-12 span-12">
<div class="sqs-block website-component-block sqs-block-website-component sqs-block-html html-block" data-block-type="1337" data-definition-name="website.components.html" data-sqsp-block="text" data-website-component-id="6c2475d8f4de1b0f6b46" id="block-6c2475d8f4de1b0f6b46">
<div class="sqs-block-content">
<div class="sqs-text-block-container">
<div class="sqs-html-content blog-post-content" data-sqsp-text-block-content="">
<p class="blog-lede">Reinforcement learning has become the shiny final step in foundation-model post-training. That status is deserved, but also dangerous. Our position is simple: RL should adjust foundation models after pretraining and cold-start supervision, not be abused as the default recipe for creating capabilities from scratch.</p>
<figure class="blog-figure">
<img src="./assets/cake.png" alt="Yann LeCun's cake analogy: self-supervised learning as cake, supervised learning as icing, reinforcement learning as the cherry on top.">
<figcaption>Yann LeCun's cake analogy: self-supervised learning is the cake, supervised learning is the icing, and reinforcement learning is the cherry on top.</figcaption>
</figure>
<p>RL is powerful because it can push a model toward behaviors we can reward: correctness, constraint satisfaction, instruction following, tool-use discipline, and long-horizon consistency. But RL is also expensive and brittle. It depends on sampling, reward design, stabilization tricks, and careful evaluation. When meaningful supervision or scaffolding is available, using it first usually provides a cheaper and more stable behavior manifold for RL to refine.</p>
<p class="blog-callout">The key distinction is <strong>capability creation vs. capability adjustment</strong>. Pretraining and cold-start supervision establish usable structure; RL most reliably reallocates probability mass toward better trajectories the model can already express. This is a default-practice claim, not an impossibility claim about RL-first discovery.</p>
<h2>Five Pillars</h2>
<section class="pillar-block">
<h3>Pillar 1</h3>
<p class="pillar-statement">RL should be used to adjust foundation models after pretraining, particularly when a model remains unsatisfactory in certain aspects.</p>
<p>Across recent R1-style pipelines, RL is most useful after the model can already produce the desired behavior sometimes. It then acts as a selector and stabilizer, shifting probability mass toward more reliable, correct, and constraint-compliant trajectories.</p>
</section>
<section class="pillar-block">
<h3>Pillar 2</h3>
<p class="pillar-statement">When meaningful supervised or scaffolded signals are available, they should usually precede RL.</p>
<p>The first-order question is not simply whether RL is expensive, but whether a meaningful scaffold exists. If it does, use it to establish a stable behavior manifold and reserve RL for residual failures; RL-first should carry a higher burden of proof.</p>
</section>
<section class="pillar-block">
<h3>Pillar 3</h3>
<p class="pillar-statement">RL should not be assumed to create reasoning capability; in current foundation-model practice, it more often refines reasoning behaviors made elicitable by pretraining, SFT, or other scaffolds.</p>
<p>This is the core causal claim. SFT and cold-start traces can provide reasoning structure; RL improves selection, consistency, and constraint satisfaction. “RL improves reasoning” is plausible, but “RL creates reasoning from scratch” requires stronger evidence that controls for hidden scaffolds, verifier artifacts, and sampling budgets.</p>
</section>
<section class="pillar-block">
<h3>Pillar 4</h3>
<p class="pillar-statement">Reward design should prioritize auditability, verifiability, and minimal composition over opaque reward complexity.</p>
<p>Verifiable rewards are powerful because they are externally checkable. Simple checks, tests, format constraints, and deterministic validators are easier to debug than opaque reward stacks, but even simple verifiers should be stress-tested for noise, brittleness, and proxy gaming.</p>
</section>
<section class="pillar-block">
<h3>Pillar 5</h3>
<p class="pillar-statement">Self-Supervised Reinforcement Learning (SSRL) is a boundary case for structured interaction settings, not a generic replacement for pretraining or supervised scaffolding.</p>
<p>Self-evolution is promising when the loop is grounded in interaction, memory, and verifiable progress signals. Without those anchors, self-generated optimization can drift into self-confirming errors, unstable objectives, or reward hacking.</p>
</section>
<h2>Why “More RL” Is Not a Strategy</h2>
<p>The popular “RL creates reasoning” story is too coarse. RL can amplify reasoning-like behavior, and sometimes dramatically. But a stronger causal claim requires stronger evidence: compute-matched comparisons, cold-start ablations, verifier stress tests, and clear reporting of sampling budgets. Without that discipline, we risk crediting RL for structure that was actually supplied by pretraining, data curation, prompt formats, expert trajectories, or supervised traces.</p>
<p>Reward design is another reason to be cautious. More sophisticated rewards are not automatically safer. If the proxy is wrong, a stronger optimizer can simply exploit it more effectively. This is why we argue for reward minimalism: use checkable signals where possible, report verifier failures, and make the optimization target easy to audit.</p>
<h2>Practical Guidelines</h2>
<p>Before applying RL, ask four questions: <strong>Is meaningful supervision or scaffolding available?</strong> <strong>Can the model already do the behavior sometimes?</strong> <strong>Can we diagnose the failures?</strong> <strong>Can success be audited with a verifiable signal?</strong> If the answer is yes, RL is a strong adjustment tool. If not, the clearer path is often better supervision, better data, better prompting, better model specification, or a better verifier.</p>
<ul class="blog-points">
<li><mark class="blog-highlight">Use stage discipline by default:</mark> pretraining, cold-start supervision or scaffolding, then RL adjustment.</li>
<li><mark class="blog-highlight">State falsifiable RL claims:</mark> separate the effect of RL from scaffolds, data curation, prompt formats, and verifiers.</li>
<li><mark class="blog-highlight">Make cost visible:</mark> report sampled trajectories, reward definitions, stabilizers, verifier pass rates, verifier failures, and regressions.</li>
</ul>
<h2>Call to Action</h2>
<p>Researchers should stop treating RL as a magic capability dial. Companies and institutions should require transparency around sampling cost, reward definitions, verifier behavior, and stabilizers when RL is claimed to replace supervision. The ML community should build standards that distinguish capability creation, capability elicitation, and behavioral refinement.</p>
<p class="blog-ending">The field does not need less RL. It needs more disciplined RL: staged after cold-start alignment or scaffolding when available, grounded in auditable rewards, and reported with compute transparency. RL should be the cherry, not the whole cake.</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</article>
</div>
</div>
</div>
</section>
</article>
</main>
</div>
</body>
</html>