Skip to content

About

Awesome Hill Climbing: curated papers, code and courses on last-mile improvement of robot policies (RL fine-tuning, residual/steering, RECAP-style retraining, human corrections, test-time verifiers, reward & world models), plus a hands-on course on a simulated SO-101 arm (MuJoCo; real hardware planned). Updated weekly.

afinitus.github.io/awesome-hill-climbing/

Topics

awesomeawesome-listembodied-aifine-tuninghill-climbingimitation-learningreinforcement-learningrobot-learningroboticsvision-language-action

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

Awesome Hill Climbing Awesome

Climbing the Nines: a success-rate landscape drawn as contour lines, with a dashed hill-climbing path from a pretrained policy at 50% up past the 99% contour.

Papers, posts, code and courses on hill-climbing robot policies: improving task reliability after pretraining or imitation learning. RL fine-tuning, residual and steering policies, advantage-weighted retraining, human corrections, test-time verifiers, reward and world models, sim-to-real, and the classic black-box search underneath it all.

952 papers and posts · 716 since September 2025 · 122 must-know (⭐) · 212 tools, courses and benchmarks · updated 2026-10-05

Interactive explainer: Climbing the Nines: the hill-climbing loop, the RL ideas underneath, a timeline from September 2025 onward, the course results, and a searchable index of every paper.

A pretrained robot policy that works half the time needs more reliable behavior before deployment. Even 99% success needs a declared task scope, quality bar, speed and evaluation protocol. "Hill climbing" is what labs and companies call the loop in between: deploy, measure, find failures, improve, re-measure. This list maps the ways people run that loop, the textbook RL ideas each one comes from, and what changed since September 2025, through the update date above.

Contents

What's new

183 papers and posts from the last 14 days (2026-09-21 to 2026-10-05), newest first. Descriptions are on each family page.

Start here

Seven reads from the last year to start with:

  1. Sunday ACT-2 Preview (Sunday Robotics, 2026-07) — Sunday's framing of hill climbing: scale pretraining until in-house gains carry over to unseen homes, then climb reliability in-house. Proposes the "Solve" reporting standard.
  2. pi*0.6 / RECAP (Physical Intelligence, 2025-11) — Advantage-conditioned retraining of a VLA: a value model tags chunks positive or negative, the model is retrained with its supervised loss on that tag, and at test time it is conditioned on positive (optionally with classifier-free guidance).
  3. EXPO-FT (Stanford University, 2026-05) — Off-policy RL around a VLA with no RL gradient through it: propose chunks, apply a small bounded edit, execute the best under a Q-ensemble. The VLA itself keeps training on its own supervised loss.
  4. Golden Ticket (Robotics and AI Institute; Arizona State University; Northeastern University, 2026-03) — Black-box search (random search, CEM, zeroth-order) for one fixed input-noise vector for a frozen diffusion or flow policy, training no new networks. Simple enough to run before RL: it beat Gaussian noise sampling on 46 of 51 sim and real tasks.
  5. SCORE (University of Washington; Amazon FAR, 2026-06) — Noise-space steering constrained to what the base policy can already do, trained only in a scanned digital twin, transferred with zero real RL.
  6. UF-OPS (University of Toronto; Google DeepMind; University of Alberta; Harvard University; UT Austin, 2026-03) — Train a verifier on your own evaluation rollouts and rerank N samples at deploy time. No policy weight updates.
  7. Dissecting advantage-guided post-training (Shanghai Jiao Tong University; Xiaomi Robotics, 2026-09) — A controlled study of advantage-guided (RECAP-style) post-training that separates how the advantage is built, calibrated and used, screens each choice with offline diagnostics, then tests the recipe on four real bimanual tasks.

The hill-climbing loop

Every method here runs some version of one loop. The families differ in which step they spend effort on and which part of the system they change.

flowchart LR
  S([Start]) --> D[Deploy] --> M[Measure] --> F[Find failures] --> I[Improve] --> R[Re-measure]
  R --> D
Loading
Step What happens Families that live here
Start A pretrained policy that already succeeds sometimes: BC, ACT, a diffusion or flow policy, a VLA.
Deploy Collect episodes on the robot, a fleet, a simulator, a digital twin or a learned world model. World models, sim-to-real
Measure Turn each episode into a number: a success check, a progress model, a VLM judge. Reward & progress models, evaluation
Find failures Locate where and why it fails: progress dips, runtime monitors, human takeovers. Reward models, human corrections
Improve Apply an update. Each family is a different update operator. Black-box search, on-policy PG, off-policy critics, residual & steering, weighted BC, DAgger, test-time search
Re-measure Held-out start states, trial counts and confidence intervals. Then repeat. Evaluation, industry loops

Measuring the nines. Near the top, the hard part is knowing whether you climbed. 30/30 successes gives a [88.6%, 100%] Wilson 95% interval, not proof of perfect reliability. At a fixed evaluation size, 73/73 gives a lower bound of 95%, and 381/381 gives 99%, assuming independent trials with a constant success probability. Report the interval, keep search episodes separate from evaluation episodes, and compare methods on the same start states.

Three patterns run through the list: these methods mostly amplify behavior the base policy already has (at 0% success there is usually little to climb); many recipes that work keep the big model frozen or anchored and train something small next to it, while others retrain the whole model, sometimes with no anchor when the base already succeeds often (SimpleVLA-RL); and the bottleneck has moved from the optimizer to the success signal and the evaluation.

Foundations

Most 2025–26 methods recombine about a dozen textbook ideas. The standard text is Sutton & Barto, Reinforcement Learning: An Introduction (free online).

Idea Core equation Classic references Used most by
Perturb and score θ' = θ + σε; keep the best (hill climbing) or step along Σᵢ R(θ+σεᵢ)·εᵢ (ES, ARS) Kohl & Stone 2004 · CMA-ES tutorial · OpenAI ES · ARS Black-box search, test-time search
Reward-weighted averaging π_new(a) ∝ π_old(a) · exp(R(a)/λ); AWR puts the advantage A in place of R, PI² the negative path cost −S PoWER · PI² · AWR Advantage-weighted BC, black-box search
Likelihood-ratio policy gradient ∇J = E[∇log π(a|s) · A] REINFORCE · TRPO · GAE · PPO · GRPO On-policy PG, world-model RL, sim-to-real
Bellman backups on replay Q(s,a) ← r + γ Q(s', a'), a' ~ π(·|s') (DQN: max over a') DQN · DDPG · TD3 · SAC · RLPD Off-policy critics, residual RL, HITL RL
Conservative / in-sample critics don't trust Q on actions the data never shows CQL · IQL · Cal-QL Offline-to-online RL, verifiers
Residual learning on a frozen base a = a_base + f_θ(s); bounded variant: a_base + α · tanh(u) Residual RL (Johannink) · Residual Policy Learning (Silver) Residual & steering, sim-to-real
On-policy labels label the states the learner actually visits DAgger · HG-DAgger Human corrections, industry loops
Greedy selection & guidance a* = argmax over samples of Q(s,a); guidance: (1+w)∇log p(a|s,c) − w∇log p(a|s) Classifier-free guidance · IDQL · V-GPS Test-time search, EXPO-style selection, RECAP
Reward shaping & learned judges r' = r + γΦ(s') − Φ(s) Potential-based shaping · Human preferences · VICE-RAQ Reward & progress models
Model-based RL / Dyna learn p(s'|s,a); practice or plan in imagination Dyna · MBPO · Dreamer World models & evaluation
Domain randomization & twins make reality one more sample of the simulator Domain randomization · ADR (Rubik's cube) Sim-to-real

Papers and posts by family

Each family answers one question. Below are the must-know works (⭐) per family; the link under each heading opens the full list, newest first.

Black-box search

How do I improve a policy when all I can do is run it and get a score, for example a success flag, and I cannot or do not want to backprop through it?

Treat the policy, or a few knobs around it, as a vector. Try perturbed copies, score each with one number per episode, and move toward the copies that scored well. No backprop through the policy, no critic, no action log-probs.

Loop step: Improve · Built on: Perturb, evaluate, step (random search / finite differences / ES); Reward-weighted averaging (PoWER, PI2, CEM, MPPI); Adapt the search distribution (CMA-ES); Bayesian optimization (BO)

All 40 entries → (26 from September 2025 onward)

  • ⭐ Decade of BO for control (RWTH Aachen, University of Technology Nuremberg, TU Munich, 2026-09-08) — Survey framing controller tuning as episodic policy search where a GP surrogate picks each trial; reviews 110 hardware papers and releases TuneControl. 57 of 110 use vanilla BO; sim 4-parameter cart-pole: LogEI 4310±48 vs Sobol 4647±190 (median±IQR cost, lower better). · code
  • ⭐ ENPIRE (NVIDIA, Carnegie Mellon University, UC Berkeley, 2026-06-18) — Coding agents edit training recipes, train, run real rollouts scored automatically, and keep what raises success. Pin insertion: 8 agent/robot pairs reached near-perfect success in about 40 min vs over 1.5 h with one (success counts up to 8 retries). · code
  • ⭐ Golden Ticket (Robotics and AI Institute; Arizona State University; Northeastern University, 2026-03-16) — Replaces per-chunk Gaussian noise in a frozen diffusion or flow policy with one fixed noise vector found by black-box search. Real Franka block/cube pick: 80 → 98% success on 50 held-out episodes after 150 search episodes (~30 min). · code
  • ⭐ EGGROLL (University of Oxford (FLAIR, WhiRL); Mila; NVIDIA AI Technology Center; NormaCore.dev, 2025-11-20) — Shares matrix multiplication across low-rank evolution-strategies perturbations. With pre-generated noise, one GH200 benchmark (8192-dimensional bf16 layer, batch 1024) reaches 91% of maximum batch-inference throughput, versus PPO 34% and OpenES 0.41% (chart reading). · code
  • ⭐ TD-ES (The University of Sydney; NVIDIA, 2025-11-13) — Continues from a PPO checkpoint with antithetic evolution strategies whose bounded triangular per-coordinate noise keeps candidates in a small box around the weights. Sim IQM success, 3 Isaac Lab Franka tasks: PPO 67.2%, Gaussian ES 73.7%, TD-ES 85.0%.
  • ⭐ ES at Scale (LLM) (Cognizant AI Lab, MIT, UCLA, UT Austin, 2025-09-29) — Perturbs all weights of 30 model copies with tiny Gaussian noise, scores each with an outcome-only reward, and steps toward high-scoring noise, with no backprop or log-probs. Countdown, Qwen-2.5-3B (language-only): ES 60.5 vs Dr.GRPO-v 43.8 (base 10.0). · code
  • ⭐ EvolSAC (University of Padova, 2025-07-14) — Pretrains SAC on a dense surrogate reward, then fine-tunes the policy with SNES using the true task score as fitness. Sim cartpole swing-up time (10-agent mean): 2.085 s → 1.135 s, vs 1.565 s after 1000 more SAC episodes.
  • ⭐ Zero-order optimization primer (Machines in Motion Laboratory, New York University, 2025-06-27) — Tutorial framing derivative-free robotics methods as one loop: sample Gaussian perturbations, score them, move toward better samples; the sample weighting recovers MPPI, CMA-ES or CEM-like updates. Plots only: on 4 Hydrax sim tasks, MPPI-CMA consistently outperforms MPPI. · code

On-policy policy gradients

How do I take a BC or SFT policy (diffusion, flow, ACT or a 7B VLA) that succeeds 50-90% of the time and push it toward 95-100% using only a success signal, a resettable simulator, and fresh on-policy rollouts?

Roll out the pretrained policy as a stochastic policy, raise the log-probability of rollouts that beat a baseline, and clip each step so the pretrained skill survives.

Loop step: Improve · Built on: Likelihood-ratio policy gradient; Trust region by clipping (PPO); Where the baseline comes from: critic vs group; Getting a log-prob out of an expressive policy

All 70 entries → (58 from September 2025 onward)

  • ⭐ ADEPT (NVIDIA; University of Michigan, 2026-08-19) — Post-trains a reposing-pretrained PPO arm-hand policy: BC-distill the actor into new observations, warm up a fresh critic with the actor frozen, then PPO at actor LR 1e-5. Sim peg insertion: 5/5 seeds reach ADR 50 vs 0/5 at LR 1e-3.
  • ⭐ StructRL (Fudan University; Singapore Management University; Shanghai Artificial Intelligence Laboratory, 2026-08-15) — Keeps the flow decoder deterministic and adds exploration noise only to the output action, AR(1)-correlated within a chunk with learned, group-bounded scales for position, rotation and gripper. ManiSkill pi0.5 OOD average: SFT 25.9, piRL 49.3, action-space Gaussian 65.9, StructRL 70.3.
  • ⭐ Temporal GRPO (Institute of Software, CAS; UCAS, 2026-08-13) — A frozen VLM proposes checkable stages compiled into predicates; group advantages are computed per stage over rollouts that reached it, so good early actions escape blame for later failures. Sim RoboTwin 2.0, OpenVLA-OFT, macro-average: 75.8 vs SimpleVLA-RL 68.8 (SFT 38.3).
  • ⭐ PAC-ACT (SUSTech, 2026-07-10) — Chunk-level PPO on a pretrained ACT (CVAE removed, learnable Gaussian noise), with a KL term between consecutive updates and a reward penalty for leaving the frozen BC policy. Sim Contour and Square Assembly: ACT 60.0%/51.2% → 100.0%/98.2%.
  • ⭐ PDE (Prompt-Driven Exploration) (MIT Improbable AI Lab; MIT-IBM Computing Research Lab, 2026-07-09) — A VLM rewrites task prompts to discover successful rollouts; PPO trains with a mixture including the deployment prompt. Sim LIBERO-PRO, pi0.5 zero-start Regime 3 task, step 120: 81.8% vs action-noise PPO 0%.
  • ⭐ Simple Recipe Works (continual VLA RL) (UT Austin, UCLA, NTU, Sony AI, 2026-03-12) — Fine-tunes a VLA on each new task in sequence with GRPO on sparse 0/1 success and a rank-32 LoRA adapter, with no replay or regularizer. Sim LIBERO-Spatial: AVG 81.2, forgetting 0.3, vs 7.3 and 40.9 with full fine-tuning. · code
  • ⭐ FPO++ (Amazon FAR, UC Berkeley, Stanford, HKU, CMU, 2026-02-02) — Flow policy gradient using exp(old CFM loss - new CFM loss) as the PPO ratio, clipped per (tau, eps) sample, with zero-noise test-time sampling. Zero-sampling alone lifts the sim Robomimic Can base policy from 10.00% to 71.35% before RL. · code
  • ⭐ piRL (Tsinghua University; Peking University; CASIA; Carnegie Mellon University; Infinigence AI; Zhongguancun Academy, 2025-10-29) — Makes pi0/pi0.5 flow chunks PPO-trainable by adding a learned per-step noise net (Flow-Noise) or an equal-marginal SDE (Flow-SDE) for exact log-probs, dropped at deployment. Few-shot LIBERO (sim): pi0 57.6% → 97.6%, pi0.5 77.1% → 98.3%. · code
  • ⭐ RL-100 (Shanghai Qi Zhi Institute; Shanghai Jiao Tong University; HKU; IIIS, Tsinghua University; UNC Chapel Hill; Carnegie Mellon University; CASIA, 2025-10-16) — BC-trains a diffusion policy on 29-400 teleop demos per task, then applies PPO-clip at every denoising step, offline with IQL advantages and an OPE gate, then online with GAE. Real, 8 tasks: 100 mean vs DP3 67.8; 1000/1000 trials. · code
  • ⭐ RLinf-VLA (Tsinghua University; Zhongguancun Academy; Infinigence AI; Peking University; UC Berkeley; Harbin Institute of Technology; CASIA, 2025-10-08) — Infrastructure plus recipe for VLA RL: simulators, rollout and trainer share GPUs, split them or run pipelined, with chunk-aware PPO and GRPO tricks. ManiSkill 25 tasks, ID/OOD: OpenVLA 53.91/39.10 → GRPO 84.38/75.15 → PPO 96.09/81.93. · code
  • ⭐ SimpleVLA-RL (Tsinghua University, Shanghai AI Lab, SJTU, PKU, HKU, 2025-09-11) — GRPO on a tokenized-action OpenVLA-OFT: 8 rollouts per scene, binary success reward on every action token, group-normalized advantages, no critic, and all-agree groups dropped. LIBERO average 91.0 → 99.1; with one demo per task, 48.9 → 96.9. · code

Off-policy critics & offline-to-online RL

How do I take a pretrained robot policy that works 40-80% of the time to 95-100% with minutes-to-hours of real robot data, reusing every demo and rollout I already have?

Learn a Q-function from every transition you have (demos, human corrections, old rollouts, new rollouts), then use it to pick, edit or push the policy's actions toward higher value.

Loop step: Improve · Built on: Bellman backups and off-policy replay; Max-entropy actor-critic (SAC) and its offline-data variant RLPD (the engine of SERL and HIL-SERL); Conservative and in-sample critics; The offline-to-online hand-off

All 123 entries → (70 from September 2025 onward)

  • ⭐ RAPolicy (Fudan University; Shanghai Key Laboratory of Multimodal Embodied AI; Singapore Management University, 2026-09-19) — A rollout copy of pi0.5, re-synced after each small update group, collects data while the learner trains an IQL-style chunk critic and one-step flow actor only on stored actions. Real, 4 tasks, 20 trials each: 86.3% average vs EXPO-FT 50%. · code
  • ⭐ VLA-Precision (University of Science and Technology of China; Beihang University; Zhongguancun Academy; Hefei SpinX Technology, 2026-09-03) — Post-trains pi0.5's action-expert LoRA with flow-matching BC on demos, successes and human corrections, plus ensemble-critic advantages relative to a frozen reference instead of absolute-Q maximization. 9 real chemistry-lab tasks: 98.3% vs demo-only pi0.5 67.8%, ConRFT (Octo) 7.8%. · code
  • ⭐ IPE (Stanford University, 2026-07-29) — Shows demo-pretrained critics barely beat random ones, then rolls out an ensemble of diverse policies trained on the same data so the critic sees contrasting actions. Six Robomimic/OGBench sim tasks: reported 1.26x average fine-tuning improvement over naive Q pretraining.
  • ⭐ AutoSERL (Institute of Automation, CAS; PKU-PsiBot Joint Lab; Peking University, 2026-07-02) — Replaces HIL-SERL's human interventions with one-demo rules: a planner corrects drift beyond 2 cm; stalls trigger partial demo replay. Real, no start randomization: 50/50 on all 6 tasks in 8-45 min; SERL (20 demos, same time): 0-20/50. · code
  • ⭐ FlowDPG (Carnegie Mellon University, University of Pennsylvania, Skild AI, 2026-06-21) — Predicts the clean action from a noisy flow point, nudges it along the critic's gradient, and regresses velocity toward it alongside BC, without ODE backprop. Real dual-arm AirPods assembly: 88% offline vs 80% for RLT/QAM (BC 64%).
  • ⭐ EXPO-FT (Stanford University, 2026-05-25) — Samples 8 pi0.5 chunks, adds a tanh-bounded learned edit to each, runs the argmax-Q of 16; the VLA keeps training on its supervised loss, with human interventions. Real, 8 tasks: SFT 20.5/30 → 30/30 each, 19.1 min average (HG-DAgger 22.1/30). · code
  • ⭐ LWD (Learning While Deploying) (AGIBOT Finch; Shanghai Innovation Institute; Columbia University, 2026-05-01) — A 16-robot fleet logs rollouts and takeovers with sparse rewards; off-policy RL bootstraps the critic from an uncertainty-adaptive quantile of a distributional value model, with QAM policy extraction. 8 real tasks, mixed success/stage score: 0.95 vs SFT 0.76, RECAP 0.85.
  • ⭐ RL Token (RLT) (Physical Intelligence, 2026-04-24) — Compresses frozen pi0.6 embeddings into one 'RL token'; a tiny TD3-style actor-critic then edits the VLA's own sampled action chunk under an L2 penalty, only in each task's critical phase. Screw critical-phase success: 20% → 65%.
  • ⭐ ALOE (AgiBot, HKUST, Fudan University, Nanjing University, 2026-02-13) — Learns a chunk-level Q of the current pi0.5 policy by TD bootstrapping with a pessimistic ensemble, instead of Monte Carlo state values, to weight policy updates. Phone Assembly, 60 trials, 10.0% BC start: ALOE 95.0, AWR 80.0.
  • ⭐ QAM (UC Berkeley, 2026-01-20) — Trains a second flow toward pi_beta(a|s)·exp(tau·Q(s,a)) via adjoint matching, using the critic's action gradient without backpropagating through the sampler. Offline RL over 50 OGBench tasks (sim): QAM 44 vs ReBRAC 40, FQL 36, FBRAC 11. · code
  • ⭐ Three Regimes of Offline-to-Online RL (Mila - Quebec AI Institute; Universite de Montreal, 2025-10-01) — Compares the pretrained policy's return with the dataset's average: if the policy is better, protect it with warm-up rollouts; if the data is better, replay offline data. Regime predictions matched outcomes in 45/63 D4RL settings (71%).
  • ⭐ Q-chunking (QC, QC-FQL) (UC Berkeley, 2025-07-10) — Critic and policy act on h-step action chunks with an unbiased h-step TD target, while the policy stays near a flow-matching BC model of the data. OGBench, 25 tasks, offline → online: QC 52→86 vs FQL 37→58. · code

Residual, edit & steering policies

How do I push a big imitation, diffusion or VLA policy from 50% to 95%+ when I can't cheaply fine-tune or backprop through it, without breaking what it already does well?

Freeze the big pretrained policy and train a small add-on with RL: one that corrects its actions (residual/edit) or one that picks the noise it is fed (steering).

Loop step: Improve · Built on: Residual policy: the base is part of the environment; Start at base performance: zero-init, bound and scale the correction; Off-policy actor-critic with prior data is what makes this fit on a real robot; Latent-noise steering: RL picks the noise, not the action

All 81 entries → (72 from September 2025 onward)

  • ⭐ PSS (Principal Steering Subspaces) (XPENG Robotics; University of Michigan, 2026-09-27) — Finds noise directions that most move a frozen generative policy's actions using finite differences at 64 states; SAC steers those directions. Sim RoboMimic Diffusion-Square, final-five-evaluation mean, one seed: 0.878 vs 0.716 for full-latent steering.
  • ⭐ Real-Time EXPO-FT (Stanford University, 2026-09-16) — Slow pi0.5 proposes 32 chunks from a stale observation; a fast policy adds bounded edits from the fresh one; a Q-ensemble picks one. Real, 4 dynamic tasks, 10 min online data: 12.5/30 → 29/30 (EXPO-FT w/ RTC 25/30). · code
  • ⭐ MiDAS (Carnegie Mellon University, 2026-08-11) — LoRA-tunes pi0.5 on one demo, then freezes it and trains a sparse-reward off-policy edit actor-critic that outputs the full executed chunk, so it can leave the base's action support. LIBERO-Long, 1 demo: 91.2 vs DICE-RL 89.3, DSRL 33.5.
  • ⭐ Foresight Residual RL (Rutgers University, 2026-07-17) — Trains a PPO residual on a frozen per-phase pi0 for each subtask, multiplying each phase's reward by a learned predictor of next-phase success. Sim wrench-on-nut full task: 85.6±3.9% vs 54.5±3.6% with plain per-phase residuals.
  • ⭐ SCORE (University of Washington; Amazon FAR, 2026-06-25) — Freezes a real-demo flow policy; in a scanned sim twin, PPO learns to steer its input noise plus a small residual, staying near the base's action support. Real Franka/LEAP, 8 tasks, no real-world RL: 37.8% → 89.9% vs Residual-RL 59.5%.
  • ⭐ BMD (Behavioral Mode Discovery) (Naver Labs Europe; Stanford, 2026-05-12) — A noise-steering policy on a frozen diffusion policy, conditioned on a per-episode code, learns distinct modes via a DIAYN-style classifier reward, kept as an RL fine-tuning bonus. ManiSkill sim Avoid, success ~1.00: DSRL keeps 2/24 modes, DPPO 6.33/24, DSRL[BMD] 10/24.
  • ⭐ UniSteer (Microsoft Research; Tsinghua University; University of Technology Sydney, 2026-05-11) — Trains a SAC-style noise actor for frozen pi0, inverting the flow decoder by fixed-point iteration so human teleop corrections become noise-space targets. Real AgileX Piper, 4 tasks: average success 20% → 90% (~66 min training) vs DSRL 55%, DAgger 60%.
  • ⭐ DICE-RL (Stanford University, 2026-03-10) — Freezes a diffusion/flow BC prior, trains a residual on the same noise with TD3+BC, then executes the best-Q sample. Real UR5 BeltAssembly, 30 trials/checkpoint: 56.67% → 93.33% after ~420 training episodes (chart reading). · code
  • ⭐ DAWN (Nanyang Technological University; Mila / Universite de Montreal; National University of Singapore, 2026-02-11) — Speeds up residual SAC with 20K base-policy replay transitions and LayerNorm in the critic. ManiSkill sim PegInsertionSide reaches 90% success around 0.3M environment steps versus roughly 2.3M for Policy Decorator (chart reading). · code
  • ⭐ RFS (University of Washington; Amazon FAR, 2026-02-02) — One PPO policy outputs both the starting noise for a frozen flow-matching base and an additive residual, so it can switch modes and leave the demo manifold. Sim, 6 IsaacLab dexterous tasks: 0.861 vs base 0.250, DSRL-only 0.483.
  • ⭐ PLD (Probe, Learn, Distill) (NVIDIA GEAR; CMU; UC Berkeley; UT Austin, 2025-10-30) — Trains a bounded Gaussian residual with SAC on a frozen VLA, lets the VLA drift before the residual recovers, and distills successes into it. LIBERO Spatial/Object/Goal avg: OpenVLA-OFT 91.8 → 99.2; real Franka cube pick-up: 30/30 vs 10/30 (human data).
  • ⭐ ResFiT (Amazon FAR; Stanford; CMU; UC Berkeley, 2025-09-23) — Keeps an ACT chunked BC policy frozen and trains a small per-step residual with a tuned TD3-style off-policy recipe (demos in each batch, LayerNorm critic, n-step returns). Real 29-DoF humanoid: WoollyBallPnP 14% → 64% after 134 rollouts. · code

Advantage-weighted & filtered BC

How do I push a big imitation policy (a diffusion policy or a flow VLA) up on one task using its own rollouts, human corrections and success labels, without running policy-gradient RL through it?

Keep the supervised imitation loss, but decide which actions (or which demos) the policy copies and how much, using a success label, a value function or a data-influence score.

Loop step: Improve · Built on: Reward- and advantage-weighted regression (RWR/AWR); Filtered BC and rejection sampling; Advantage- or return-conditioned policies; Classifier-free guidance as a policy-improvement knob

All 63 entries → (46 from September 2025 onward)

  • ⭐ Dissecting advantage-guided post-training (Shanghai Jiao Tong University; Xiaomi Robotics, 2026-09-23) — Ablates separately how critic-derived advantages are built, calibrated and used; n-step TD, per-value-bin normalization and soft exponential weights on the pi0.5 loss worked best. Real bimanual, 4-task mean success: SFT 0.11, source-weighted DAgger 0.45, continuous weighting 0.74.
  • ⭐ RoboDrop (Tsinghua University; Striding AI, 2026-09-09) — In one warm-up epoch, compares each sample's compressed gradient with one from the most similar clean validation frames, and drops conflicting episodes before re-post-training pi0.5. Real, 4 tasks, 20 rollouts each: 35.0% → 67.5% with automatic filtering.
  • ⭐ CLIFT (UC Berkeley, Google DeepMind, NVIDIA Research, 2026-07-31) — Tunes closed-weight Gemini Robotics On-Device via an SFT API only, tagging the top 30% of chunks by dense-reward advantage over similar-start chunks as 'advantage positive'. Real Unitree G1, 2 cycles of 100 rollouts: Plate Handover 53% → 96%.
  • ⭐ RedFlow (HKUST, 2026-07-30) — Clusters action chunks by estimated progress and joint state; within a cluster, positive-scored chunks (progress gain plus episode outcome) form correction targets for negative chunks, which are also suppressed. Real pi0 clothes folding: 36 → 67/100 (AWR 48). · code
  • ⭐ DEED (HIVE Robots; Technical University of Denmark, 2026-07-22) — Data-efficient GR00T N1.6 post-training plus a RECAP-adapted refinement (text advantage prefix, VLM value function) for chip restocking on a Unitree G1, ~108 min of data. 50 real episodes each: SFT 32%, iteration 1 42% (CIs overlap), iteration 2 22%.
  • ⭐ STEAM (Institute of Automation, CAS; Tsinghua University; UCAS, 2026-06-29) — An ensemble of small VLMs predicts time offsets between expert frame pairs (forward and reversed); the ensemble minimum gives per-frame advantages for CFGRL training of pi0. Real towel folding: 92.3% vs one-iteration RECAP-style baseline 55.6%, BC 33.3% (trials unstated).
  • ⭐ ROVE (XPENG Robotics; Fudan; CUHK; SJTU, 2026-06-15) — Splits each human intervention into autonomous rollout, human adaptation and human recovery, placing the failure penalty after adaptation; an optimistic expectile critic drives RECAP-style conditioning. Iteration 1, same robot data, Erase/Bread: 60.0%/73.3% vs RECAP 50.0%/66.7%.
  • ⭐ IG-RFT (Zhejiang University; Torch Kernel, 2026-02-24) — Trains pi0.5 with 60 demos per task, offline advantage-weighted flow matching from a learned critic, then human-in-the-loop correction rounds with the same weighting. Four real long-horizon tasks, 20 trials each: 18.8% → 40.0% → 85.0%.
  • ⭐ pi*0.6 / RECAP (Physical Intelligence, 2025-11-18) — Fits a distributional value, binarizes n-step advantages into text conditioning, and supervises the whole flow VLA; CFG is optional. Same-data real laundry: about 2x AWR throughput; success ~96% versus AWR ~91%, PPO ~75% (chart reading).
  • ⭐ Hi-ORS (Tsinghua SIGS; Tencent Robotics X, 2025-10-30) — Keeps episodes above a rising return threshold, including human-corrected successes, and trains pi0 with reward-weighted flow matching. Three real tasks, two robots: 23.3 percentage points over offline BC (chart reading); about 1.5 hours of real training. · code
  • ⭐ CUPID (Stanford; Toyota Research Institute; Cambridge; NVIDIA, 2025-06-23) — Scores each demo's effect on closed-loop success by chaining influence functions with a REINFORCE estimate from labeled eval rollouts, then drops negative-influence demos and retrains. Real TuckBox mass shift, 66% filtered: base 0%, CUPID 84%, oracle 88% (bar readings). · code
  • ⭐ CFGRL (UC Berkeley, 2025-05-29) — Trains one flow policy by BC with an optimality token; classifier-free guidance samples reference policy times a non-decreasing advantage function. Sim ExORL walker-stand, same IQL critic: 782±8 vs AWR 603±8. OGBench: mixed vs AWR, beats goal-conditioned BC on most tasks. · code

Human corrections & DAgger

My imitation policy works most of the time but fails in states the demos never covered. How do I spend a small amount of human time so the next version fixes exactly those failures?

Run the policy, let a person (or now a machine) step in where it goes wrong, and train on those on-policy fixes so the policy learns to handle the states it actually reaches.

Loop step: Find failures → Improve · Built on: Compounding error (covariate shift); DAgger: label the learner's own states; Human-gated vs robot-gated takeovers; Reweighting correction data

All 80 entries → (47 from September 2025 onward)

  • ⭐ BlenDAgger (Carnegie Mellon University, 2026-09-29) — Keeps the policy running during corrections, blending human and policy actions per DoF group and giving the human more control when they push hard, disagree, or the policy is uncertain. Final round, real Almond Scooping: 50% vs HG-DAgger 15%.
  • ⭐ Skill-Space Shooting (Tsinghua University; UC Berkeley; Shanghai Qi Zhi Institute, 2026-09-29) — A learned value model halts a pi0.5-class policy when its score drops below zero; Gemini 2.5 Pro picks and checks a repair skill; successful repairs become training data. Real Coffee mean progress: 71.25% vs 46.25% with extra human demos.
  • ⭐ TimelyDAgger (Tsinghua University; SEEN·E Robotic, 2026-09-27) — Robot-gated DAgger: asks for takeover when a PCA residual on VLA features exceeds a threshold that the expert's first post-takeover actions raise or lower. pi0.5 StackCube-Green (ManiSkill sim, oracle expert): 81 vs HG-DAgger 29, offline BC 49.
  • ⭐ PAKT (NVIDIA, 2026-09-22) — Operators correct HIL-SERL-style real-world RL by hand-guiding the arm via admittance control sharing the policy's action space, limits and impedance tracker. Vs HIL-SERL, four real insertion tasks: 62%-86% fewer interventions; 23%-48% lower cycle time, mostly from the new controller stack.
  • ⭐ GAINS (Beijing Institute of Technology; Beijing Innovation Center of Humanoid Robotics; City University of Hong Kong; Nankai University, 2026-08-16) — Learns the return distribution with a truncated-quantile critic and acts on its lower tail, because takeover timing is inconsistent (over half of repeated marks differ by 10+ frames). Four sim and two real tasks: 22% higher success than RLIF. · code
  • ⭐ FlowDAgger (Microsoft Research; Microsoft; ETH Zurich; University of Washington, 2026-07-09) — Inverts the flow sampler to find the noise that would have produced the operator's correction and trains a small noise policy on it, VLA frozen. MetaWorld, pi0.5: 0.78 vs SFT 0.71; held-out skills 0.88 (base 0.96) vs SFT 0.02. · code
  • ⭐ HELP (Shanghai AI Laboratory, 2026-07-08) — A progress critic (VLAC-CUT) cuts autonomous pi0.5 rollouts into progress, recovery, idle and failure-inducing segments; only progress and recovery segments join HITL recovery data for retraining. Real round 2, Microplate: 42 tasks/hr at 90% vs HITL-only 25 at 60%. · code
  • ⭐ UniIntervene (Nanyang Technological University; Beijing University of Posts and Telecommunications, 2026-06-10) — Adds an intervention agent to HIL-SERL: a VLM-based proxy value model and a temporal value-risk critic detect stagnation, then steer back to a remembered high-value state. 5 real UR7e tasks: HIL-SERL 81% success at 34.3% intervention → 88% at 14.6%. · code
  • ⭐ FlowPRO (Tencent Robotics X; Futian Laboratory; Tsinghua University, 2026-06-03) — Operators roll back and teleoperate fixes, giving paired failed/corrected segments for a critic-free DPO-style flow-matching loss with proximal and SFT terms. Real Pack, pi0 base, 3 rounds: 99% vs 94% for a RECAP-style advantage-conditioning baseline, DAgger 88%.
  • ⭐ Hi-WM (Current Robotics; Tsinghua University; Peking University; University of Toronto, 2026-04-23) — A human corrects failure-prone policy rollouts inside a learned action-conditioned world model, with rollback and branching from cached states; the corrections post-train the policy. 3 real tasks, DP and pi0: +37.9 points over base, +19.0 over world-model self-training without humans.
  • ⭐ SOP (AgiBot Research; Shanghai Innovation Institute, 2026-01-06) — Ten real AgiBot G1 robots run pi0.5 with human takeovers; a cloud learner mixes on-policy data with old demos, publishing weights every 25 steps. 3 h on-policy interaction: 0.571 → 0.800; 80 more demo hours: 0.576 → 0.612.
  • ⭐ RaC (Carnegie Mellon University, 2025-09-09) — Changes HG-DAgger takeovers: the human first recovers to a familiar state, then finishes the sub-task, and the episode ends with the takeover. Shirt-hanging: 78.3% with 5 hours of data vs ALOHA Unleashed's 75.0% with ~89 hours (RaC's estimate; cross-paper).

Test-time search & verifiers

My policy sometimes does the right thing but not reliably. Can I raise its success rate without retraining it, just by choosing better among its own samples at run time?

Leave the policy's weights alone and spend extra compute at deployment: sample several candidate actions, score them with a critic, verifier, world model or monitor, and run the best one (or nudge the sampler toward it).

Loop step: Improve at deploy time (no policy weight updates) · Built on: Best-of-N with a verifier (sample, score, pick); KL-regularized optimum = guidance; Offline critics that are safe to argmax over; Lookahead with a model (shooting, MPPI, MCTS)

All 143 entries → (122 from September 2025 onward)

  • ⭐ PreferenceFlow (Westlake University, 2026-09-29) — Pairs each human takeover window with a frozen pi0.5 chunk from the same state, trains a small Bradley-Terry scorer on the pairs, and uses it to guide sampling. Four real Franka insertion tasks: 90.5% vs frozen 69.0%, intervention BC 84.5%.
  • ⭐ SeeQ (Carnegie Mellon University, 2026-09-18) — Trains a critic that only values finishing the active subtask (reward at subtask end, no bootstrapping across boundaries) and reranks 8 samples from a frozen pi0.5 with it. Real bimanual, 24 trials per task: 35.4% → 66.7% average. · code
  • ⭐ Q-Planning (Georgia Institute of Technology, 2026-08-21) — Keeps the policy frozen as a proposal, scores 32-64 candidate chunks with a separate ~1B Q-function, runs their softmax-Q-weighted average, and retrains only the critic. Real robot, 5 iterations: stack-cups 40% → 90% vs 55% for SFT on successful rollouts. · code
  • ⭐ CoRe (Case Western Reserve University, 2026-08-14) — Flags out-of-distribution states via a Mahalanobis test on the action-expert latent, rewinds to a recovery anchor, imagines the frozen policy's moves, and plans minimal object restorations; training-free. Sim LangSwitch (ManiSkill) switch average: pi0 18.3 → 87.5, pi0.5 35.5 → 92.7. · entry corrected after source review
  • ⭐ QGF (Q-Guided Flow) (UC Berkeley; Physical Intelligence, 2026-06-09) — Freezes a BC flow policy and IQL critic; each flow step adds grad Q, evaluated at a one-step estimate of the finished action, to the velocity. OGBench sim, charted 20-task mean: beats tested test-time methods, slightly edges EDP. · code
  • ⭐ VLA-ATTC (University of Sydney; Central South University; USTC; SenseTime Research; HKUST, 2026-05-02) — Samples two chunks; when their DTW distance exceeds a threshold, samples 16 and picks one by knockout tournament under a relative action critic trained on auto-generated preference pairs. LIBERO-LONG, 50 trials/task: pi0 82.8% → 90.6% gated (92.2% ungated); RoboMonkey 56.5%.
  • ⭐ FASTER (Stanford University, 2026-04-21) — A critic predicts action Q-values from raw noise, so only the best seed gets denoised. pi0.5 on five libero_90 tasks versus EXPO: inference FLOPs 37.55 TF → 4.70 TF, comparable success (chart reading). · code
  • ⭐ UF-OPS (University of Toronto; Google DeepMind; University of Alberta; Harvard University; UT Austin, 2026-03-10) — Trains a small verifier on labeled evaluation rollouts of the deployed policy, then reranks 10 samples from the frozen diffusion or flow policy. Real ALOHA, 5 tasks, 20 trials each: mean success 38% → 87% (chart reading).
  • ⭐ CoVer (Stanford University; NVIDIA Research, 2026-02-12) — Keeps pi0 frozen, samples K instruction rephrasings times M action chunks, and picks one with a CLIP-style contrastive verifier trained on BridgeV2. SIMPLER red-teaming, ID/OOD: pi0 41.5/29.7, pi0 fine-tuned on augmented data 44.0/48.7, pi0 (rephrase) + CoVer 65.5/62.0. · code
  • ⭐ VLS (University of Washington; AI2; University of Oxford; NUS, 2026-02-03) — Training-free: a VLM names task objects, SAM/DINOv2 features plus depth give 3D keypoints, and VLM-generated reward gradients steer frozen diffusion/flow denoising. CALVIN (sim) movable objects: 0.94 versus DynaGuide 0.26 and base 0.13 (chart reading). · code
  • ⭐ TACO (TeleAI (China Telecom); USTC; Tsinghua University; HKUST, 2025-12-02) — Samples M chunks from a flow VLA and runs the one with the highest pseudo-count in the fine-tuning data, from a small Coin Flipping Network on internal features. RoboTwin 2.0 (pi0.5, 41 tasks, M=50): 59.3 → 64.0. · code
  • ⭐ JITI (Jilin University; Microsoft Research Asia, 2025-11-27) — Trains an offline Cal-QL critic on frozen VLA features; reranks chunks only when Q deviates from its moving average. LIBERO-Elegant, SmolVLA: ESR 49.8 → 67.2, versus 53.8 with guidance every step (chart reading).

Reward, value & progress models

My robot only gets a success bit at the end of an episode, or a human has to watch every rollout. How do I get a dense, trustworthy score so RL, filtered BC or advantage conditioning can push the policy from about 60-80% toward 95%+?

These methods leave the policy alone and build the score it climbs against: a learned or zero-shot judge that says how far along a rollout is, whether it succeeded, and where it went wrong.

Loop step: Measure → Find failures · Built on: Success classifier as reward; Progress from time (time-as-label); Preference and ranking rewards (Bradley-Terry); Potential-based shaping

All 88 entries → (61 from September 2025 onward)

  • ⭐ Dyna: reward model as failure detector (Dyna Robotics, 2026-09-24) — A learned video progress model monitors deployment: progress dips flag errors and steer targeted recovery-data collection and retraining. Dyna-1 napkin folding: 99.4% success over a 24-hour run; Dyna says conventional pipelines stall near 80% (talk, no paper).
  • ⭐ RynnValue (DAMO Academy, Hupan Lab, 2026-08-10) — Predicts seconds until task completion from an instruction and a few frames, trained on over 7,000 hours of video with no preference labels. As reward for real online DSRL on frozen pi0.5: 72.5% vs Robometer 52.5%, sparse 48.8%. · code
  • ⭐ SVM (Success Visitation Matching) (UC Berkeley, 2026-06-22) — Trains a classifier on the policy's own successful vs failed rollouts and adds its clipped log-odds to the sparse success reward. Real WidowX with DSRL, Pick and Place: about 0.93 vs about 0.2 for outcome-only RL (plot reading).
  • ⭐ RARM (National University of Singapore, Booking.com, University of Cambridge, Nanjing University, 2026-06-20) — A comparator trained on non-robot videos (DAVIS, MOSE) matches rollout clips to one successful demo, rewarding only confident forward progress. Real pi0 with DSRL, cloth folding after 90 rollouts: 25/30 vs binary reward 7/30, Robometer 9/30.
  • ⭐ SARM2 + SPIRAL (Stanford, UC Berkeley, SJTU, xdof.ai, 2026-06-09) — Classifies the active action primitive to gate a mixture-of-experts value head regressing normalized remaining steps, used as reward for a residual TD3+BC policy on frozen pi0.5. Real, Round 3 Whiteboard: 18/20 vs fine-tuned Robometer 13/20, sparse 11/20. · code
  • ⭐ ViVa (GigaAI, Sichuan University, Tsinghua University, 2026-04-09) — Builds the RECAP value model from the Wan2.2 video generator, jointly predicting future proprioception and a return separating successes [0,1) from failures [1,2). Real, three tasks, two rounds: 80.0% average success vs 63.3% with a VLM value. · code
  • ⭐ SOLE-R1 (MIT, RAI Institute, 2026-03-30) — An 8B video-language model writes chain-of-thought at each timestep, then outputs progress in [-100, 100] as the only reward for off-policy RL from a random policy. 40 sim and real tasks: >=50% success on 24 vs 7 for GPT-5. · code
  • ⭐ Robometer (USC, UT Dallas, MIT, University of Washington, Ai2, NVIDIA, 2026-03-02) — Qwen3-VL-4B reward model with progress, success and preference heads, trained on RBM-1M including failures. Real DSRL on frozen pi0: 20% → 85% in ≤45 min (10k steps) vs 55% with RoboReward; 0 false-positive successes vs 45. · code
  • ⭐ TOPReward (University of Washington, Allen Institute for AI, Amazon, UNC Chapel Hill, 2026-02-22) — Reads a frozen Qwen3-VL-8B's log-probability of 'True' that a video prefix completes the task as a training-free progress curve. Real ManiRewardBench progress subset (113 tasks, 497 episodes): VOC 0.942 vs GVL 0.316 on the same model, trained Robometer-4B 0.858. · code
  • ⭐ RoboReward (Stanford University, UC Berkeley, 2026-01-02) — Fine-tunes Qwen3-VL to score rollouts 1-5 on a rubric, adding counterfactual-relabeled and clipped negatives; in sim, reward accuracy tracks RL success (r=0.83). Real WidowX DSRL, open drawer, 20 trials: base 10%, RoboReward 8B 80%, human-reward oracle 90%.
  • ⭐ Robo-Dopamine (Peking University, BAAI, University of Sydney, CASIA, 2025-12-29) — A VLM reward model predicts signed progress 'hops' from multi-view images, fused into a potential used as policy-invariant shaping, not raw reward. 8 real tasks: 95.2% final success vs 68.0% with sparse reward (ConRFT) and 9.8% for BC. · code
  • ⭐ VLAC (Shanghai AI Laboratory, 2025-09-19) — Fine-tunes InternVL into a pairwise critic outputting signed progress between two frames plus a done signal, labeled from video temporal order, and uses it for real-arm PPO. Four real tasks: about 30% → about 90% within 200 episodes. · code

World models & evaluation

How do I tell whether checkpoint k+1 really beats checkpoint k, and how do I improve a policy, when every real robot trial is slow, noisy and needs a reset?

Build a stand-in for the real robot, either a learned video or latent world model or a scanned real-to-sim twin, then use it to score checkpoints cheaply and to run the try-score-improve loop without spending robot time.

Loop step: Deploy → Measure · Built on: Dyna: learn a model, then practice inside it; Latent imagination (PlaNet/Dreamer); Monte Carlo policy evaluation in a stand-in world; Model exploitation and compounding error

All 104 entries → (93 from September 2025 onward)

  • ⭐ WorldSample (Nanyang Technological University; Tsinghua University; Central South University; Beijing University of Posts and Telecommunications, 2026-07-02) — Adds a Dyna-like stream to real-robot SAC/RLPD: jittered actions are rendered by a post-trained Cosmos-Predict2.5 model, reward-labeled, and gated into the critic loss. Real robot, 5 tasks: 82% in 64 min vs HIL-SERL 56% in 83 min.
  • ⭐ RoboWorld (KAIST; Config, 2026-07-01) — Converts Wan2.1-T2V-1.3B into a causal, KV-cached, 4-step autoregressive video model on DROID, trained with Step Forcing to limit drift, and scores rollouts with GPT-4o. Across 8 VLAs (4,186 rollouts): Pearson r = 0.989 with the real RoboArena leaderboard.
  • ⭐ VLA-MBPO (Nanjing University; Mila / Universite de Montreal, 2026-03-21) — Fine-tunes Bagel on ~50 collected rollouts per task to predict multi-view frames 10 steps ahead plus task success, then runs PPO on short imagined branches from dataset states. Sim LIBERO average 76.8 → 85.9, vs 82.6 for equal-budget online RL.
  • ⭐ PlayWorld (Princeton University, 2026-03-09) — Trains a video world model on 30 hours of autonomous robot play, then improves a diffusion policy inside it with DSRL. Real towel-folding success peaks at 75% from 10% (2000 steps); human-demo-model peak: 40% (chart reading). · code
  • ⭐ WoVR (CASIA; Zhongguancun Academy; Tsinghua University, 2026-02-15) — Runs GRPO entirely inside an action-conditioned Wan2.2-5B world model, starting imagined episodes near failure keyframes and refreshing the model with policy-aligned data to curb hallucination. Real Franka: 51.1 → 80.0 average (WMPO 64.5), no online robot RL. · code
  • ⭐ VLAW (Stanford University; Tsinghua University, 2026-02-12) — Grounds a world model on 50 real pi0.5 rollouts per task each iteration, labels synthetic rollouts with a strict Qwen3-VL-4B, and fine-tunes on real plus synthetic successes. 5 real DROID tasks: 0.460 → 0.868 vs filtered BC 0.752. · code
  • ⭐ RISE (CUHK; Kinetix AI; HKU; Shanghai Innovation Institute; Horizon Robotics; Tsinghua University, 2026-02-11) — Replaces online physical rollouts with a learned action-conditioned video model and a pi0.5-based value model that scores imagined chunk outcomes, driving advantage-conditioned training of pi0.5. Real dual-arm, 20 trials per task: box closing 35% → 95% vs RECAP 60%. · code
  • ⭐ PolaRiS (University of Washington; Princeton University; UC Berkeley; Stanford University; Toyota Research Institute; USC; Cornell University; Physical Intelligence, 2025-12-18) — Turns a 2-5 minute monocular video into a splat-rendered IsaacSim twin for policy evaluation, with brief co-training to align visuals. Average Pearson r = 0.9 across 6 unseen scenes; in their tests the Ctrl-World video model mis-ranked policies (MMRV 0.22). · code
  • ⭐ Veo world-sim eval (Google DeepMind, 2025-12-11) — Runs Gemini Robotics On-Device inside an action-conditioned, multi-view Veo 2 world model; humans score the videos. Eight nominal checkpoints: Pearson 0.88, MMRV 0.03 against real success, with under-predicted absolute rates (chart reading).
  • ⭐ Prophet / ProphRL (Fudan University; Shanghai Innovation Institute; Logos Robotics, 2025-11-25) — Trains flow VLAs with FA-GRPO and FlowScale inside a 2B Cosmos-initialized action-to-video world model, rewarded by a VLM majority vote. Real UR30e, 100 in-model RL updates, 3 runs x 20 trials: pi0.5-3B 52.1 → 82.1 (+30.0). · code
  • ⭐ WMPO (HKUST; ByteDance Seed, 2025-11-12) — Runs critic-free GRPO for OpenVLA-OFT inside a pixel-space video world model fine-tuned on the policy's own rollouts, failures included, with a VideoMAE success classifier. MimicGen (sim), 4 tasks, 128-rollout budget: 33.6 → 47.1 vs online GRPO 33.2, offline DPO 37.3. · code
  • ⭐ Ctrl-World (Stanford University; Tsinghua University, 2025-10-11) — Fine-tunes a 1.5B multi-view video diffusion model on DROID so a VLA runs closed-loop inside it, then fine-tunes the policy on human-labeled successful imagined rollouts. pi0.5-DROID: 38.7% → 83.4% on novel instructions and objects, imagined rollouts only. · code

Sim-to-real & real-to-sim

How do I get the trial-and-error gains of RL for a robot policy without paying for thousands of risky, hand-reset real-world episodes?

Uses a simulator, often a scanned or generated copy of the real scene, as a safe practice gym where the policy can hill-climb with RL before or alongside a short real-robot stage. Real demos or real replay data keep it tied to reality.

Loop step: Deploy → Improve · Built on: Domain randomization; Real-to-sim: digital twins and cousins; BC-anchored sim RL (sim-real co-training); Sim-to-online RL on the real robot

All 53 entries → (42 from September 2025 onward)

  • ⭐ F4R (Nanyang Technological University; Xi'an Jiaotong University; Dexmal, 2026-09-28) — A VLM diagnoses real pi0.5 failures, the scene is rebuilt in Isaac Sim, and the policy is co-trained on failure-conditioned sim data, then PPO-refined and redeployed. Real OOD success: 26.25% → 90.0%, vs 71.25% for budget-matched Targeted BC.
  • ⭐ DLS (Where Success Breaks) (Show Lab, National University of Singapore, 2026-09-05) — Finds pi0/pi0.5 failures in parallel digital twins, turns hand-defined task phases into a dense potential, and applies a critic-free contrastive velocity loss plus a real-demo SFT anchor. pi0.5, 3 real tasks, ID/Unseen: 84.4/72.2 vs PPO 81.1/67.8, 100-demo SFT 78.9/54.4.
  • ⭐ SILO (NVIDIA; UC San Diego, 2026-07-06) — Trains PPO from scratch in sim on a capsule-chain cable for single-clip routing, then deploys through a 10 Hz twin whose sim joint angles become real targets. Real 3-clip routing, zero demos: 18/24 vs 12/24 copied from the hierarchical-IL paper.
  • ⭐ TacCoRL (UCLA, UC San Diego, Peking University, University of Utah, UESTC, 2026-06-10) — Adds contact-gated tactile tokens to a pi0.5-style VLA, co-trains on real plus sim demos, then runs sim PPO with a real-data BC anchor. Real, 4 bimanual tasks after sim RL: 72.5% visuo-tactile vs 50.0% vision-only.
  • ⭐ Video2Sim2Real (Georgia Institute of Technology, University of Pennsylvania, Toyota Research Institute, Carnegie Mellon University, 2026-06-07) — Builds a sim twin from one RGB-D human video, re-optimizes robot poses at object-motion keyframes, distills them into a point-cloud policy, adds a PPO finger residual. Real, 7 tasks, no robot demos: 67/70 (95.7%) vs 11/70 with FoundationPose poses. · code
  • ⭐ Sim-to-Real VLA Generalization Study (The Chinese University of Hong Kong, Shenzhen; Shenzhen Loop Area Institute, 2026-03-24) — Controlled study of zero-shot sim-to-real for a VLA trained only in RoboTwin 2.0, isolating which randomization factors matter and adding binary-success GRPO in sim. Real average success: SFT 5.6% → SFT+RL 33.4% → SFT+RL+DR 42.8%.
  • ⭐ Sim-to-Real VLA RL with Generative 3D Worlds (Horizon Robotics, 2026-03-19) — Generates 100 physical ManiSkill 3 scenes with a GPT-4o designer and EmbodiedGen, runs PPOFlow on a BridgeV2-pretrained pi0 across them, then transfers zero-shot. Real WidowX, 12 scenes, 240 trials: 21.7% → 75% success, compared only with the imitation starting policy.
  • ⭐ HandelBot (Stanford University; Amazon FAR, 2026-03-12) — Freezes a sim piano rollout, corrects lateral finger joints from MIDI errors, then trains bounded residual RL with MIDI rewards. Five real songs: mean best-validation key-press F1 77.6 versus open-loop simulation trajectory 41.8 (chart reading). · code
  • ⭐ What Matters for Sim-to-Online RL (ETH Zurich; Google DeepMind, 2026-02-23) — Empirical study of continuing sim-pretrained SAC on real robots: keep or warm-start replay, update the actor once per 20 critic steps at LR 1e-5, restore optimizer state. Franka vision grasping reaches near-perfect success in about ten real minutes. · code
  • ⭐ RL-Co (RLinf-Co / Beyond Imitation) (Tsinghua University, Harbin Institute of Technology, Peking University, Carnegie Mellon University, Shanghai AI Laboratory, Zhongguancun Academy, 2026-02-13) — Co-trains the VLA on real plus MimicGen sim demos, then runs PPO in a rough ManiSkill3 task copy with a weighted real-demo BC loss; no real-robot RL. Real average success, pi0.5: SFT co-training 45.9 → 66.2. · code
  • ⭐ TwinRL (Peking University; HKUST; Simplexity Robotics; Tsinghua University, 2026-02-09) — SFTs Octo-Small on 30 real demos plus twin trajectories, runs RL in a phone-scanned twin to seed the real buffer, then real HIL-SERL-style RL. 4 real Franka tasks: near-100% success, at least 30% less real interaction than HIL-SERL. · code
  • ⭐ VT-Refine (Columbia University, NVIDIA, UC San Diego, 2025-10-16) — BC-trains a visuo-tactile diffusion policy on 30 real demos per task, fine-tunes it with DPPO and sparse success reward in an Isaac Gym twin simulating the same tactile pad, and redeploys. Real tabletop, 5 parts, visuo-tactile mean: 0.50 → 0.85. · code

Industry recipes & deployment reports

How do robot companies say they climb from ~60-80% to 99% on a paying task, and what do they actually disclose?

Companies take a big pretrained robot model and push one real task from "works in a demo" to "works all shift" with deploy, grade, curate and retrain loops, sometimes with RL folded in.

Loop step: The whole loop · Built on: On-policy data aggregation (DAgger and its human-gated variant); Automatic success and progress detection; Policy improvement through supervised learning (advantage-weighted or advantage-conditioned BC); Broad pretraining as the reason local hill-climbing transfers (a company claim, e.g. ACT-2)

All 28 entries → (20 from September 2025 onward)

  • ⭐ Skild Physical Self-Play (Skild AI, 2026-09-23) — Previews RL post-training via self-play: in simulated robot soccer the policy's one stated objective is to score, playing recent versions of itself, then is transferred to a real humanoid. No metrics; 140 simulated years of play, real match on video.
  • ⭐ Dyna: deployment is the eval (Dyna Robotics, 2026-08) — Fleet workflow: an auto-labeler splits each deployed episode into SOP steps with outcome and failure mode, humans review uncertain cases, and engineers query the fleet to retrain. Napkin folding: 35/hour, 75% passing (Dyna-1) → 95/hour, 93% (Dyna-2).
  • ⭐ Sunday ACT-2 Preview (Sunday Robotics, 2026-07-17) — Pretrains ACT-2 on sensorized human data from Sunday's own capture hardware, then post-trains on repeated in-house Memo runs and recoveries (update rule undisclosed). Laundry: 99.1% over 785 autonomous attempts in unseen homes, one fixed checkpoint, zero per-home adaptation.
  • ⭐ Siemens factory VLA case study (Siemens, 2026-05-25) — Runs a pure-SFT loop on pi0.5 in a real Siemens factory: teleop demos, manual review, re-fine-tuning, and targeted recovery demos, relaxing constraints each round. 2,535 factory episodes; no final success rate given; authors say it fell short of expectations.
  • ⭐ Generalist GEN-1 (Generalist AI, 2026-04-02) — Pretrained on 500K+ hours of human wearable-device data, adapted per task with about 1 hour of robot data; RL is one of five listed ingredients, not detailed. Three charted tasks: 99% average vs 64% for fine-tuned GEN-0 (whole-system comparison).
  • ⭐ DYNA-1i open-world dexterity (Dyna Robotics, 2025-12) — Post-trains a VLA on 'only tens of hours' of office data to stay reliable in unseen venues; no objective or update rule is given. 30-minute shirt-folding trials (~40/hour): 22 consecutive failure-free folds in the office, 20 in unseen venues.
  • ⭐ Figure Helix logistics (Figure AI, 2025-06-07) — Improves one deployed parcel-sorting job with more in-domain demos, richer System-1 inputs (stereo, vision memory, state history, force) and a 50% larger decoder head; the post describes no RL. Over three months, barcode-orientation success rose ~70% → ~95%.

Broader RL advances that transfer

Which general RL results from the last year change how you should hill-climb a robot policy?

Not robot-manipulation papers: scaling laws for value functions and RL compute, critic and actor-critic design studies, flow/diffusion policy theory, RL for language models (group baselines, off-policy and stale-data training, process rewards, test-time compute), and last-mile RL for driving policies. Robot work keeps borrowing from all of them.

Loop step: Background for every step · Built on: Everything in Foundations

All 79 entries → (59 from September 2025 onward)

No ⭐ picks in this family yet; the full list is linked above.

Code, tools and learning resources

Code: robot learning and last-mile RL

  • ACT: Action Chunking with Transformers (Stanford) — Original Action Chunking with Transformers implementation; companion code to the ALOHA paper arXiv 2304.13705.
  • AgileX PiPER SDK (AgileX Robotics) — Official AgileX code for controlling the PiPER robotic arm.
  • ALOHA: A Low-cost Open-source Hardware System for Bimanual Teleoperation (Stanford) — ROS 1 teleoperation and data-collection code plus hardware assembly guide for the ALOHA bimanual system (four arms, four cameras); pairs with the ACT repo for imitation learning.
  • COST-HIL: Cost-Aware Human-in-the-Loop Reinforcement Learning for Real-World Dexterous Manipulation — Cost-aware HIL-SERL variant with weighted replay and gripper-switch penalties. Repository manuscript reports real Zhiyuan Elf G2 grasp success 99.9% versus 80.7% for HIL-SERL; human-intervened training steps fall from 30% to 6%.
  • Dexbotic: Open-Source Vision-Language-Action Toolbox (Dexmal) — PyTorch VLA toolbox that runs multiple mainstream VLA policies under one environment setup via experiment scripts, plus Dexbotic pretrained models that the authors report improve those policies.
  • EXPO: Expressive Policy Optimization (Stanford) — Official JAX code for "EXPO: Stable Reinforcement Learning with Expressive Policies" (arXiv 2507.07986), built on jaxrl and RLPD; offline-to-online scripts for D4RL Antmaze, Adroit, Robomimic and MimicGen.
  • FluxVLA Engine: A One-Stop VLA Engineering Platform for Embodied Intelligence (LimX Dynamics et al.) — Unified VLA engineering platform standardizing datasets, VLM/world models, action heads, reward/advantage-weighted learning, sim evaluation, human-in-the-loop correction, and Real-Time Chunking robot deployment.
  • GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (UC Berkeley) — MIT-licensed software for GELLO kinematic-replica teleoperation leaders built from 3D-printed parts and cheap motors; supports Franka, UR, xArm and YAM, with bimanual and demonstration data collection.
  • Interbotix ALOHA (Trossen Robotics) — Trossen Robotics' fork of the Mobile ALOHA repo: ROS 2 teleoperation, dataset collection and evaluation tools for Stationary and Mobile ALOHA kits; pairs with Interbotix ACT/ACT++ forks.
  • Isaac GR00T (NVIDIA) — Code (Apache-2.0) and weights (NVIDIA Open Model License) for GR00T N1.7, an open cross-embodiment humanoid VLA (Cosmos-Reason2 VLM + diffusion transformer head), with fine-tuning, inference and evaluation scripts; N1.5/N1.6 on branches.
  • Koch v1.1 low-cost robot arm (Jess Moss) — Version 1.1 of Alexander Koch's low-cost robot arm with small design changes; open hardware repo.
  • LeRobot (Hugging Face LeRobot) — Hugging Face robot learning library: datasets, ACT/diffusion/pi0/SmolVLA policies, SO-101 and other low-cost hardware, a modular RL stack with HIL-SERL, reward models (TOPReward, Robometer, SARM) and DAgger-style data collection.
  • Low-cost robot arm (Alexander Koch) — Open-source hardware design for Alexander Koch's low-cost leader/follower robot arm, the original that the Koch v1.1 arm builds on.
  • Model-Based RSL RL (rsl_rl_rwm): Robotic World Model and Uncertainty-Aware Robotic World Model code (ETH Zurich) — Fork of RSL RL adding model-based RL algorithms for Robotic World Model (arXiv 2501.10100) and Uncertainty-Aware RWM for offline MBRL on real robots (arXiv 2504.16680).
  • Nautilus: From One Prompt to Plug-and-Play Robot Learning — Agent harness (a Claude Code plugin) that turns a natural-language prompt into reproduction, evaluation, fine-tuning and deployment workflows via typed contracts, generated adapters and containers. Main code not yet public (project page: "Coming Soon").
  • Octo: An Open-Source Generalist Robot Policy (UC Berkeley and collaborators) — Open-source transformer diffusion generalist policy (27M/93M params) pretrained on 800k robot trajectories, with head-only or full fine-tuning to new robots on small datasets.
  • openpi (Physical Intelligence) — Official Apache-2.0 repo with pi0, pi0-FAST and pi0.5 base checkpoints (10k+ hours robot data) plus JAX/PyTorch fine-tuning and inference code; ALOHA and DROID examples.
  • pyAgxArm (AgileX Robotics) — Official AgileX Python SDK for Piper and Nero arms plus AgxGripper and Revo2 end effectors: CAN communication, status reading, motion and end-effector control. LGPL-3.0.
  • real-time-chunking-kinetix: Simulated experiments for "Real-Time Execution of Action Chunking Flow Policies" (Physical Intelligence) — Official Kinetix sim code for Real-Time Chunking and training-time RTC: trains experts, generates data, trains flow policies, and evaluates across inference delays and execution horizons on 12 levels.
  • resfit-pytorch (unofficial ResFiT implementation, WIP) (lucidrains) — Unofficial work-in-progress PyTorch implementation of ResFiT (arXiv 2509.19301), residual off-policy RL for behavior-cloned policies. An official implementation is also available from the authors.
  • SO-ARM100 (Standard Open Arm 100 / SO-101) (TheRobotStudio; Hugging Face) — Open-source 3D-printable low-cost leader/follower robot arm hardware ("Standard Open Arm 100"), Apache-2.0, widely used with LeRobot.
  • StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing (StarVLA Community) — Modular VLA codebase with swappable VLM/world-model backbones and action heads, co-training recipes, and unified eval on LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, BEHAVIOR-1K (arXiv 2604.05014).
  • Universal Manipulation Interface (Stanford; Columbia; Toyota Research Institute) — Official UMI code: robot-free handheld-gripper demo collection, SLAM video processing, dataset generation, and diffusion-policy training with UR5 deployment and a cup-arrangement checkpoint.
  • vla-rlt: RL Tokens for open-source VLAs (SmolVLA on SO-101) (akashspacesky) — Unofficial RLT-inspired implementation: frozen SmolVLA, bottleneck RL-token encoder, SAC actor adding regularized delta actions; offline pretrain then online SAC on SO-101, runs on Apple Silicon. No own results.
  • VLAb: Your Laboratory for Pretraining VLAs (Hugging Face) — LeRobot-derived library for pretraining VLAs; official SmolVLA reproduction kit with multi-dataset sampling and multi-GPU/SLURM training. Fine-tuning and inference are deferred to LeRobot.
  • X-VLA: Soft-Prompted Transformer as a Scalable Cross-Embodiment Vision-Language-Action Model (official code) (Tsinghua AIR) — Official X-VLA-0.9B code (ICLR 2026): embodiment-specific soft prompts on a unified Transformer for cross-embodiment VLA; natively in LeRobot; won AgiBot World Challenge at IROS 2025.
  • XLeRobot: Practical Dual-Arm Mobile Home Robot for $660 (Vector-Wangel) — Open-source (Apache-2.0) low-cost dual-arm mobile home robot hardware and software built on LeRobot and SO-101; README lists ~$660 parts cost for the basic config, excluding 3D printing, tools, shipping and taxes.

Code: general RL and black-box optimization

  • Ax: Adaptive Experimentation Platform (Meta) — Meta's open-source adaptive experimentation platform; general black-box optimization tooling.
  • BoTorch: Bayesian Optimization in PyTorch (Meta) — Meta's PyTorch library for Bayesian optimization.
  • Brax (Google) — Differentiable JAX physics engine with PPO, SAC, ARS, ES and APG trainers on accelerators; since 0.13.0 only brax/training is maintained, with environments via MuJoCo Playground and physics via MJX or MuJoCo Warp.
  • CleanRL (CleanRL community) — High-quality single-file implementations of deep RL algorithms with research-friendly features (PPO, DQN, C51, DDPG, TD3, SAC, PPG).
  • cmaes (CyberAgent AI Lab) — Python library for the CMA Evolution Strategy, a black-box optimizer.
  • CORL: Clean Offline Reinforcement Learning (Tinkoff AI) — Single-file implementations of offline and offline-to-online RL algorithms: AWAC, BC, CQL, DT, EDAC, IQL, SAC-N, TD3+BC, LB-SAC, SPOT, Cal-QL, ReBRAC (repo now archived).
  • d3rlpy (Takuma Seno) — An offline deep reinforcement learning library.
  • EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management — Asynchronous RLinf-based scheduler overlaps rollout and training with CPU/GPU resource allocation. Simulation: 1.30-3.47x rollout throughput over prior systems; LIBERO-Object/GR00T PPO completes 60 epochs ~2.5x faster than a synchronous baseline.
  • EnvPool (Sea AI Lab) — C++-based high-performance parallel (vectorized) environment execution engine for general RL environments.
  • EvoJAX: Hardware-Accelerated Neuroevolution (Google) — JAX neuroevolution toolkit running evolution algorithms, policy networks and vectorized tasks in parallel on GPUs/TPUs; README examples run in minutes on a single accelerator versus hours or days on CPUs. Repo now archived.
  • evosax: Evolution Strategies in JAX (Robert Tjarko Lange) — JAX library of 30+ evolution strategies (CMA-ES, OpenAI-ES, Differential Evolution, PSO, Diffusion Evolution) with an ask-eval-tell API compatible with jit, vmap and lax.scan on GPU/TPU.
  • EvoTorch (NNAISENSE) — Advanced evolutionary computation library built directly on PyTorch, created at NNAISENSE.
  • EvoX (EMI Group) — Distributed GPU-accelerated framework for evolutionary computation with a comprehensive library of evolutionary algorithms and benchmark problems.
  • Gymnasium (Farama Foundation) — Standard API for single-agent reinforcement learning environments, with popular reference environments and utilities (formerly OpenAI Gym).
  • jaxrl (Ilya Kostrikov) — Clean JAX/Flax implementations of continuous-action deep RL: SAC, AWAC, DrQ, DDPG with clipped double-Q, REDQ, and behavioral cloning; superseded by jaxrl2.
  • LeanRL (Meta) — CleanRL-derived single-file PyTorch RL scripts optimized for speed using torch.compile and CUDA graphs (repo now archived).
  • Nevergrad (Meta AI) — Python toolbox for performing gradient-free optimization.
  • openpi-RLT: openpi-based real-robot RL system with RL-token-guided action refinement — Community open-source reproduction of PI's RL Token on openpi pi0.5: RL-token modules, policy serving, online actor-critic, real-robot tools; AgileX arm Ethernet-insertion demo vs frozen VLA.
  • Optuna (Optuna) — A hyperparameter optimization framework.
  • PureJaxRL (Oxford FLAIR) — End-to-end JAX PPO with environment and training compiled together. README reports 10x over single-run CleanRL PyTorch PPO on CartPole/MinAtar-Breakout, and over 1000x over standard PyTorch RL when parallelizing many agents on GPUs.
  • pycma (CMA-ES) — Python implementation of CMA-ES (Covariance Matrix Adaptation Evolution Strategy) for black-box optimization.
  • RL Baselines3 Zoo (DLR-RM) — Training framework for Stable Baselines3 RL agents, with hyperparameter optimization and pre-trained agents included.
  • rl_games (Denys Makoviichuk) — PyTorch RL library with PPO, asymmetric actor-critic and SAC implementations, supporting vectorized GPU environments including Isaac Lab and MJLab.
  • RSL-RL (ETH Zurich RSL) — Lightweight GPU-accelerated robot learning library with PPO and student-teacher distillation, multi-GPU training; used by Isaac Lab, Legged Gym, mjlab and MuJoCo Playground (paper arXiv:2509.10771).
  • Sample Factory (Aleksei Petrenko) — High-throughput synchronous and asynchronous reinforcement learning library.
  • skrl (Toni-SM) — Modular RL library in PyTorch, JAX and NVIDIA Warp with support for Gymnasium/Gym, NVIDIA Isaac Lab, MuJoCo Playground and other environments.
  • Stable-Baselines3 (DLR-RM) — PyTorch version of Stable Baselines: reliable implementations of reinforcement learning algorithms in one library.
  • Tianshou (Tsinghua University) — An elegant PyTorch deep reinforcement learning library from Tsinghua's thu-ml group.
  • TorchRL (Meta) — A modular, primitive-first, python-first PyTorch library for reinforcement learning.

Simulators and robot models

  • Genesis (Genesis Embodied AI) — Open-source (Apache-2.0) simulation platform for general-purpose robotics and embodied AI learning.
  • gym-hil (Hugging Face) — Hugging Face's human-in-the-loop reinforcement learning environment suite; the repo describes itself as a "Human in the loop Reinforcement Learning suite".
  • Isaac Lab (NVIDIA) — Unified open-source (BSD-3) robot-learning framework with multi-physics and multi-renderer support; v3.0.0-EA released 2026-09-16.
  • Isaac Sim (NVIDIA) — Open-source NVIDIA Omniverse application for developing, simulating, and testing AI-driven robots in realistic virtual environments.
  • legged_gym: Isaac Gym Environments for Legged Robots (ETH Zurich Robotic Systems Lab) — Isaac Gym environments for training ANYmal and other legged robots on rough terrain with PPO (rsl_rl), including actuator network and domain randomization for sim-to-real; now migrated to Isaac Lab.
  • mjlab: A Lightweight Framework for GPU-Accelerated Robot Learning (UC Berkeley; Sorbonne University) — Isaac Lab's manager-based API on GPU-accelerated MuJoCo Warp with minimal dependencies; reference velocity-tracking, motion-imitation and manipulation tasks, multi-GPU training; Apache-2.0.
  • MuJoCo (Google DeepMind) — Multi-Joint dynamics with Contact: open-source (Apache-2.0) general-purpose physics simulator widely used for robot RL and sim-to-real.
  • MuJoCo Menagerie (Google DeepMind) — Collection of high-quality robot models for the MuJoCo physics engine, curated by Google DeepMind.
  • MuJoCo Playground (Google DeepMind) — Open-source suite of GPU-accelerated MuJoCo (MJX/Warp) environments for classic control, locomotion, and non-prehensile/dexterous manipulation, with vision via batch rendering, PPO training and sim-to-real focus.
  • MuJoCo Warp (MJWarp) (Google DeepMind; NVIDIA) — GPU-accelerated MuJoCo built on NVIDIA Warp, jointly maintained by Google DeepMind and NVIDIA as part of Newton; usable via MJX, Isaac Lab and mjlab.
  • Newton (newton-physics) — Open-source GPU-accelerated physics simulation engine built on NVIDIA Warp, targeting roboticists and simulation researchers; Apache-2.0.
  • robosuite: A Modular Simulation Framework and Benchmark for Robot Learning (ARISE Initiative) — Modular simulation framework and benchmark for robot learning; project site robosuite.ai.
  • so101_sim: MuJoCo sim environment for SO-100/SO-101 arms (tuul-ai) — MuJoCo simulation environment and task suite for SO-100/SO-101 arms, with an RL training notebook and a LeRobot-format wrapper for imitation learning.

Benchmarks and datasets

Courses, books and tutorials

Surveys

Talks and podcasts

Related awesome lists

  • Awesome Deep RL (kengz) — Curated general deep-RL resource list: libraries, benchmark results, environments (incl. Meta-World, PyBullet), competitions, a 1947-2021 timeline, books, tutorials and blogs.
  • Awesome Learning for Manipulation (Noietch) — Curated paper list on learning-based manipulation: VLAs (including an RL fine-tuning subsection), video-action models, visuomotor policies, world models, benchmarks, simulators and industry tech reports.
  • Awesome Real World RL (ugurkanates) — Curated list of papers, books, talks, courses, datasets and libraries for making RL work in real-world settings: sim-to-real, offline RL, imitation learning, meta-RL.
  • Awesome Test-Time Robot Learning (Oliverbansk) — Curated list and taxonomy of deployment-stage robot policy improvement in six lines: RL post-training, policy steering, test-time adaptation/training, in-context learning, test-time scaling, policy self-improvement.
  • Awesome-Embodied-AI (wadeKeith) (wadeKeith) — Community-maintained embodied-AI list of 430+ resources: surveys, VLA models, embodied RL, code-as-policy, in-context learning, humanoids, safety, simulators, datasets and toolkits.
  • Awesome-Flow-RL-Papers (Tonghe Zhang) — Curated table of 20+ papers/projects training flow-matching models and policies with RL, tagged by domain (robotics/CV/NLP), online/offline, on/off-policy, and pretrain vs fine-tune.
  • awesome-model-based-RL (OpenDILab) (OpenDILab) — Curated, continually updated list of model-based RL papers by venue (classics through ICLR/ICML 2026), with key insights, environments, a taxonomy, tutorials and codebases.
  • awesome-physical-ai (keon) — Curated paper list on Physical AI: VLAs, world models, action representations, and learning paradigms including a section on RL for VLA policies (e.g. VLA-RL, FLaRe, SimpleVLA-RL).
  • Awesome-Real2Sim2Real (sun254667) — Curated list of 2024-2026 papers on sim-to-real, real-to-sim and Real2Sim2Real closed-loop transfer (RL sim2real, domain randomization, Gaussian-splatting pipelines, benchmarks) for robotics and embodied AI.
  • Awesome-RL-VLA: Awesome RL-VLA for Robotic Manipulation — Companion list to the TechRxiv survey on RL for VLA manipulation models; organizes papers into offline, online, offline+online and test-time RL-VLA, plus base models, benchmarks and tools.
  • Awesome-Robotics-Diffusion (NUS Show Lab) — Curated list of robot-learning papers using diffusion models (diffusion as policy, as synthesizer, benchmarks, applications) for manipulation, navigation and planning; companion to an IJCV 2026 survey.
  • awesome-vla-wam — Curated VLA and world-action-model research list with latent action models, human action pretraining, robotic agentics, surveys, datasets, benchmarks and hardware; RL papers appear within the VLA list.
  • Awesome-WAM (World Action Models) (OpenMOSS) — Curated World Action Model papers (cascaded/joint WAMs, world models for VLA imitation, RL and evaluation, data, benchmarks) with reading blogs and leaderboard; accompanies survey arXiv 2605.12090.
  • Awesome-World-Model-for-Robotics-Policy (NTU MARS Lab) — Curated list accompanying the survey "World Model for Robot Learning" (arXiv 2605.00080): world models as policies, as simulators for RL and policy evaluation, video backbones, benchmarks, datasets.
  • Embodied_AI_Paper_List (HCPLab-SYSU) (Sun Yat-sen University HCP Lab; Pengcheng Laboratory) — Paper list for the embodied-AI survey "Aligning Cyber Space with Physical World" (arXiv 2407.06886): simulators, perception, interaction, agents/VLAs, sim-to-real, datasets.

Hands-on: lastmile

This repo also hosts lastmile, an open-source course that implements these families on one task: a simulated low-cost SO-101 arm (or an AgileX PiPER) picks up a cube and drops it in a cup. One base policy works about half the time; each chapter tries to push it toward 95%+ and reports what that cost in robot-minutes and human-minutes. Everything runs in MuJoCo on a laptop (no NVIDIA GPU needed). Each chapter also describes a real-arm step for the SO-101; none has been run on hardware yet, and each carries a safety note to read first (clear workspace, power switch or e-stop in reach, low speed limits, supervise every run). Each hardware step in the chapter notes opens with a safety checklist; read it before you power the arm.

Benchmark limitation: CupDrop-v1 scores a drop into the cup at its current position; it can count successes after the cup was moved or tipped and recovered. These scores do not establish a no-shoving benchmark. The measurements are in docs/DESIGN.md (§2, known metric limitation).

SO-101 picking up the cube and dropping it in the cup AgileX PiPER doing the same task

What is in the repo today: the simulated task on both arms, the evaluation harness (fixed search and held-out start states, Wilson intervals, a cost ledger) and a scripted controller with ten tunable knobs. Tuned, it scores 255/256 = 99.6% [97.8%, 99.9%] on the held-out set; with the deliberately mis-tuned knobs that chapters 1 and 2 start from, it scores 123/256 = 48.0% [42.0%, 54.2%]. Chapters 0, 1, 2, 3, 5, 6, 7, 8 and 12 are available, with notes and results in docs/chapters/.

make setup      # uv venv + dependencies
make test       # fast test suite
make validate   # sanity-check the task with the scripted controller, on both arms
Ch Method Base idea Mirrors Status
0 The base policy Behavior-clone a flow policy from mixed-quality demos Rectified flow, ACT, Diffusion Policy available
1 Hill climbing & ARS Finite differences on a smoothed objective Kohl & Stone 2004, ARS, TD-ES available
2 CEM, CMA-ES, PI², BO Reward-weighted averaging; BO: a surrogate model Zero-order primer, BO review available
3 The golden ticket Black-box search in noise space Golden Ticket available
4 PPO/GRPO for a flow policy Likelihood-ratio PG + anchors SimpleVLA-RL, πRL, PAC-ACT planned for v0.2
5 SAC → RLPD → Q-chunking Bellman backups on replay Q-chunking, Three Regimes, IPE available
6 Residual RL Frozen base + bounded add-on ResFiT, DAWN, Res-HIL available
7 DSRL noise steering RL in latent-noise space DSRL, SCORE, PSS, RFS available
8 Propose, edit, select Edit policy + argmax-Q EXPO-FT, Real-Time EXPO-FT available
9 Filtered BC, AWR, RECAP-lite Improvement as supervised learning π*0.6/RECAP, CFGRL planned for v0.2
10 DAgger, HG-DAgger, RaC On-policy labels RaC, SOP, FlowDAgger planned for v0.2
11 Rewards & progress models Shaping and learned judges Robometer, TOPReward planned for v0.2
12 Test-time best-of-N Greedy selection with a verifier UF-OPS, Q-Planning, SeeQ available

Design contract: docs/DESIGN.md · course plan: docs/research/curriculum.md · chapter notes: docs/chapters/

Leaderboard

Chapter Method Robot Setting Base SR [CI] Final SR [CI] Δ Improvement robot-min Human-min n eval
ch00 bc_flow so101 sim 99.6% [97.8, 99.9] 50.4% [44.3, 56.5] — (builds base_v1) 9,906.7 52.0 256
ch01 ars_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 1,207.8 0 256
ch01 ars_s3 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 1,194.4 0 256
ch01 fd_bigstep_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 3,731.8 0 256
ch01 fd_bigstep_s3 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 3,658.0 0 256
ch01 fd_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 3,586.3 0 256
ch01 fd_s3 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 3,591.2 0 256
ch01 fd_s4 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 3,582.2 0 256
ch01 greedy_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 1,309.7 0 256
ch01 greedy_s4 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 1,414.7 0 256
ch01 greedy_s3 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 1,449.0 0 256
ch01 ars_s4 so101 sim 48.0% [42.0, 54.2] 99.2% [97.2, 99.8] +51.2 pp 1,237.4 0 256
ch01 fd_bigstep_s4 so101 sim 48.0% [42.0, 54.2] 97.7% [95.0, 98.9] +49.6 pp 3,702.7 0 256
ch01 random_s2 so101 sim 48.0% [42.0, 54.2] 67.2% [61.2, 72.6] +19.1 pp 0 0 256
ch01 random_s3 so101 sim 48.0% [42.0, 54.2] 0.0% [0.0, 1.5] -48.0 pp 0 0 256
ch01 random_s4 so101 sim 48.0% [42.0, 54.2] 0.0% [0.0, 1.5] -48.0 pp 0 0 256
ch02 bo_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 110.9 0 256
ch02 cem-4k_s0 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 132.6 0 256
ch02 cem-4k_s1 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 123.0 0 256
ch02 cem-4k_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 138.1 0 256
ch02 cem-greedy_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 125.7 0 256
ch02 cem_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 146.9 0 256
ch02 cmaes-4k_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 162.9 0 256
ch02 cmaes_s1 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 170.8 0 256
ch02 cmaes_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 166.1 0 256
ch02 pi2-4k_s0 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 132.7 0 256
ch02 pi2-4k_s1 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 130.3 0 256
ch02 pi2-4k_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 149.0 0 256
ch02 pi2_s0 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 156.7 0 256
ch02 pi2_s1 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 148.3 0 256
ch02 random-4k_s1 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 236.2 0 256
ch02 random-4k_s2 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 243.1 0 256
ch02 sobol_s1 so101 sim 48.0% [42.0, 54.2] 100.0% [98.5, 100.0] +52.0 pp 160.7 0 256
ch02 bo-hw_s0 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 10.5 0 256
ch02 bo_s0 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 113.0 0 256
ch02 bo_s1 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 121.0 0 256
ch02 cem-greedy_s0 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 135.6 0 256
ch02 cem_s0 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 200.0 0 256
ch02 cem_s1 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 165.3 0 256
ch02 cmaes-4k_s1 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 140.1 0 256
ch02 pi2_s2 so101 sim 48.0% [42.0, 54.2] 99.6% [97.8, 99.9] +51.6 pp 137.8 0 256
ch02 cmaes_s0 so101 sim 48.0% [42.0, 54.2] 99.2% [97.2, 99.8] +51.2 pp 181.9 0 256
ch02 random_s2 so101 sim 48.0% [42.0, 54.2] 99.2% [97.2, 99.8] +51.2 pp 267.9 0 256
ch02 sobol_s2 so101 sim 48.0% [42.0, 54.2] 99.2% [97.2, 99.8] +51.2 pp 155.3 0 256
ch02 cmaes-4k_s0 so101 sim 48.0% [42.0, 54.2] 98.8% [96.6, 99.6] +50.8 pp 192.0 0 256
ch02 random_s0 so101 sim 48.0% [42.0, 54.2] 98.8% [96.6, 99.6] +50.8 pp 264.2 0 256
ch02 random_s1 so101 sim 48.0% [42.0, 54.2] 98.8% [96.6, 99.6] +50.8 pp 276.9 0 256
ch02 random-4k_s0 so101 sim 48.0% [42.0, 54.2] 98.0% [95.5, 99.2] +50.0 pp 242.5 0 256
ch02 pi2-hot_s1 so101 sim 48.0% [42.0, 54.2] 97.7% [95.0, 98.9] +49.6 pp 292.1 0 256
ch02 pi2-hot_s2 so101 sim 48.0% [42.0, 54.2] 97.7% [95.0, 98.9] +49.6 pp 226.3 0 256
ch02 pi2-hot_s0 so101 sim 48.0% [42.0, 54.2] 93.4% [89.6, 95.8] +45.3 pp 252.7 0 256
ch02 sobol_s0 so101 sim 48.0% [42.0, 54.2] 91.0% [86.9, 93.9] +43.0 pp 150.3 0 256
ch02 cem-greedy_s1 so101 sim 48.0% [42.0, 54.2] 79.7% [74.3, 84.2] +31.6 pp 188.4 0 256
ch02 bo-hw_s1 so101 sim 48.0% [42.0, 54.2] 48.0% [42.0, 54.2] +0.0 pp 18.0 0 256
ch02 bo-hw_s2 so101 sim 48.0% [42.0, 54.2] 48.0% [42.0, 54.2] +0.0 pp 15.5 0 256
ch03 cem_near0_s1 so101 sim 50.4% [44.3, 56.5] 87.5% [82.9, 91.0] +37.1 pp 204.2 0 256
ch03 golden_ticket (exploratory: method added after held-out peek) so101 sim 50.4% [44.3, 56.5] 87.5% [82.9, 91.0] +37.1 pp 3,859.0 0 256
ch03 cem_near0_s2 so101 sim 50.4% [44.3, 56.5] 85.9% [81.1, 89.7] +35.5 pp 214.8 0 256
ch03 racing_s1 so101 sim 50.4% [44.3, 56.5] 85.5% [80.7, 89.3] +35.2 pp 333.7 0 256
ch03 cem_s0 so101 sim 50.4% [44.3, 56.5] 84.0% [79.0, 88.0] +33.6 pp 354.0 0 256
ch03 racing_s2 so101 sim 50.4% [44.3, 56.5] 81.6% [76.4, 85.9] +31.2 pp 360.4 0 256
ch03 cem_near0_s0 so101 sim 50.4% [44.3, 56.5] 80.1% [74.8, 84.5] +29.7 pp 240.6 0 256
ch03 random_s0 so101 sim 50.4% [44.3, 56.5] 78.5% [73.1, 83.1] +28.1 pp 380.0 0 256
ch03 zero_ticket so101 sim 50.4% [44.3, 56.5] 75.8% [70.2, 80.6] +25.4 pp 7.2 0 256
ch03 random_s1 so101 sim 50.4% [44.3, 56.5] 75.4% [69.8, 80.3] +25.0 pp 393.8 0 256
ch03 cem_s2 so101 sim 50.4% [44.3, 56.5] 75.0% [69.4, 79.9] +24.6 pp 329.3 0 256
ch03 racing_s0 so101 sim 50.4% [44.3, 56.5] 73.4% [67.7, 78.5] +23.0 pp 301.4 0 256
ch03 random_s2 so101 sim 50.4% [44.3, 56.5] 71.1% [65.3, 76.3] +20.7 pp 406.5 0 256
ch03 cem_s1 so101 sim 50.4% [44.3, 56.5] 66.0% [60.0, 71.5] +15.6 pp 330.0 0 256
ch03 random_ticket_s1 so101 sim 50.4% [44.3, 56.5] 42.2% [36.3, 48.3] -8.2 pp 0 0 256
ch03 random_ticket_s2 so101 sim 50.4% [44.3, 56.5] 32.0% [26.6, 38.0] -18.4 pp 0 0 256
ch03 random_ticket_s0 so101 sim 50.4% [44.3, 56.5] 16.8% [12.7, 21.9] -33.6 pp 0 0 256
ch05 qc_s1 so101 sim 50.4% [44.3, 56.5] 95.7% [92.5, 97.6] +45.3 pp 1,448.5 11.1 256
ch05 qc_s0 so101 sim 50.4% [44.3, 56.5] 93.0% [89.2, 95.5] +42.6 pp 1,443.3 11.0 256
ch05 qc_s2 so101 sim 50.4% [44.3, 56.5] 93.0% [89.2, 95.5] +42.6 pp 1,444.2 11.2 256
ch05 qc-cql_s0 so101 sim 50.4% [44.3, 56.5] 60.2% [54.1, 66.0] +9.8 pp 1,395.4 11.0 256
ch05 qc-cql_s2 so101 sim 50.4% [44.3, 56.5] 55.1% [49.0, 61.1] +4.7 pp 1,395.2 11.2 256
ch05 qc-cql_s1 so101 sim 50.4% [44.3, 56.5] 54.7% [48.6, 60.7] +4.3 pp 1,394.3 11.1 256
ch05 qc-random so101 sim 50.4% [44.3, 56.5] 48.8% [42.8, 54.9] -1.6 pp 1,347.4 8.3 256
ch05 rlpd_s2 so101 sim 50.4% [44.3, 56.5] 45.3% [39.3, 51.4] -5.1 pp 1,470.4 11.2 256
ch05 sac_s1 so101 sim 50.4% [44.3, 56.5] 6.2% [3.9, 9.9] -44.1 pp 1,474.8 11.1 256
ch05 rlpd_s1 so101 sim 50.4% [44.3, 56.5] 3.5% [1.9, 6.5] -46.9 pp 1,474.8 11.1 256
ch05 sac_s0 so101 sim 50.4% [44.3, 56.5] 2.3% [1.1, 5.0] -48.0 pp 1,473.7 11.0 256
ch05 sac_s2 so101 sim 50.4% [44.3, 56.5] 0.4% [0.1, 2.2] -50.0 pp 1,472.8 11.2 256
ch05 rlpd_s0 so101 sim 50.4% [44.3, 56.5] 0.0% [0.0, 1.5] -50.4 pp 1,471.2 11.0 256
ch06 residual-td3_v1_s1 so101 sim 50.4% [44.3, 56.5] 90.2% [86.0, 93.3] +39.8 pp 6,842.8 0 256
ch06 residual-td3_v1_s2 so101 sim 50.4% [44.3, 56.5] 90.2% [86.0, 93.3] +39.8 pp 6,840.8 0 256
ch06 residual-td3_v1_s0 so101 sim 50.4% [44.3, 56.5] 88.7% [84.2, 92.0] +38.3 pp 6,843.2 0 256
ch06 residual-td3_bias_s0 so101 (bias) sim 18.8% [14.4, 24.0] 79.7% [74.3, 84.2] +60.9 pp 6,853.2 0 256
ch06 residual-td3_bias_s2 so101 (bias) sim 18.8% [14.4, 24.0] 75.4% [69.8, 80.3] +56.6 pp 6,854.8 0 256
ch06 residual-td3_bias_s1 so101 (bias) sim 18.8% [14.4, 24.0] 58.2% [52.1, 64.1] +39.5 pp 6,872.0 0 256
ch07 residual_s2 so101 sim 50.4% [44.3, 56.5] 98.8% [96.6, 99.6] +48.4 pp 2,269.7 0 256
ch07 residual_s1 so101 sim 50.4% [44.3, 56.5] 94.1% [90.6, 96.4] +43.8 pp 2,271.7 0 256
ch07 residual_s0 so101 sim 50.4% [44.3, 56.5] 90.2% [86.0, 93.3] +39.8 pp 2,309.9 0 256
ch07 dsrl-pss8-res_s0 so101 sim 50.4% [44.3, 56.5] 89.5% [85.1, 92.6] +39.1 pp 2,304.1 0 256
ch07 dsrl-pss8_s1 so101 sim 50.4% [44.3, 56.5] 87.9% [83.3, 91.3] +37.5 pp 2,264.5 0 256
ch07 noise-steering-best so101 sim 50.4% [44.3, 56.5] 87.9% [83.3, 91.3] +37.5 pp 3,218.2 0 256
ch07 dsrl-pss8-res_s1 so101 sim 50.4% [44.3, 56.5] 86.7% [82.0, 90.3] +36.3 pp 2,286.2 0 256
ch07 dsrl-pss8_s0 so101 sim 50.4% [44.3, 56.5] 80.9% [75.6, 85.2] +30.5 pp 2,262.3 0 256
ch07 dsrl_s0 so101 sim 50.4% [44.3, 56.5] 75.8% [70.2, 80.6] +25.4 pp 2,308.6 0 256
ch07 dsrl_s1 so101 sim 50.4% [44.3, 56.5] 75.8% [70.2, 80.6] +25.4 pp 2,295.0 0 256
ch07 dsrl_s2 so101 sim 50.4% [44.3, 56.5] 75.8% [70.2, 80.6] +25.4 pp 2,314.6 0 256
ch07 dsrl-pss8_s2 so101 sim 50.4% [44.3, 56.5] 74.2% [68.5, 79.2] +23.8 pp 2,271.3 0 256
ch07 random_actor_s2 so101 sim 50.4% [44.3, 56.5] 73.4% [67.7, 78.5] +23.0 pp 0 0 256
ch07 random_actor_s0 so101 sim 50.4% [44.3, 56.5] 70.7% [64.9, 75.9] +20.3 pp 0 0 256
ch07 random_actor_s1 so101 sim 50.4% [44.3, 56.5] 62.5% [56.4, 68.2] +12.1 pp 0 0 256
ch07 dsrl-pss8-res_s2 so101 sim 50.4% [44.3, 56.5] 59.8% [53.7, 65.6] +9.4 pp 2,316.9 0 256
ch07 residual-hard_s1 so101 (hard) sim 29.3% [24.1, 35.1] 73.0% [67.3, 78.1] +43.8 pp 2,314.8 0 256 (stress)
ch07 dsrl-pss8-hard_s1 so101 (hard) sim 29.3% [24.1, 35.1] 67.6% [61.6, 73.0] +38.3 pp 2,296.7 0 256 (stress)
ch07 residual-hard_s0 so101 (hard) sim 29.3% [24.1, 35.1] 66.4% [60.4, 71.9] +37.1 pp 2,312.8 0 256 (stress)
ch07 residual-hard_s2 so101 (hard) sim 29.3% [24.1, 35.1] 64.5% [58.4, 70.1] +35.2 pp 2,308.2 0 256 (stress)
ch07 dsrl-pss8-hard_s2 so101 (hard) sim 29.3% [24.1, 35.1] 60.2% [54.1, 66.0] +30.9 pp 2,305.0 0 256 (stress)
ch07 dsrl-pss8-hard_s0 so101 (hard) sim 29.3% [24.1, 35.1] 53.9% [47.8, 59.9] +24.6 pp 2,303.8 0 256 (stress)
ch07 dsrl-pss8-res-hard_s0 so101 (hard) sim 29.3% [24.1, 35.1] 51.6% [45.5, 57.6] +22.3 pp 2,339.3 0 256 (stress)
ch07 dsrl-pss8-res-hard_s1 so101 (hard) sim 29.3% [24.1, 35.1] 51.6% [45.5, 57.6] +22.3 pp 2,331.1 0 256 (stress)
ch07 dsrl-pss8-res-hard_s2 so101 (hard) sim 29.3% [24.1, 35.1] 51.6% [45.5, 57.6] +22.3 pp 2,335.8 0 256 (stress)
ch07 random_actor-hard_s2 so101 (hard) sim 29.3% [24.1, 35.1] 48.4% [42.4, 54.5] +19.1 pp 0 0 256 (stress)
ch07 random_actor-hard_s0 so101 (hard) sim 29.3% [24.1, 35.1] 47.3% [41.2, 53.4] +18.0 pp 0 0 256 (stress)
ch07 random_actor-hard_s1 so101 (hard) sim 29.3% [24.1, 35.1] 45.7% [39.7, 51.8] +16.4 pp 0 0 256 (stress)
ch08 expo-b0.2_s1 (ablation) so101 sim 50.4% [44.3, 56.5] 98.4% [96.1, 99.4] +48.0 pp 160.3 0 256
ch08 expo-b0.2_s2 (ablation) so101 sim 50.4% [44.3, 56.5] 98.4% [96.1, 99.4] +48.0 pp 161.1 0 256
ch08 expo-b0.1-cotrain_s1 (cotrained-ablation) so101 sim 50.4% [44.3, 56.5] 98.0% [95.5, 99.2] +47.7 pp 2,529.4 0 256
ch08 expo-b0.2_s0 (ablation) so101 sim 50.4% [44.3, 56.5] 98.0% [95.5, 99.2] +47.7 pp 160.9 0 256
ch08 expo-b0.05_s0 (ablation) so101 sim 50.4% [44.3, 56.5] 96.9% [94.0, 98.4] +46.5 pp 163.2 0 256
ch08 expo-b0.1-cotrain_s0 (cotrained-ablation) so101 sim 50.4% [44.3, 56.5] 96.5% [93.5, 98.1] +46.1 pp 2,531.2 0 256
ch08 expo-b0.1-cotrain_s2 (cotrained-ablation) so101 sim 50.4% [44.3, 56.5] 96.5% [93.5, 98.1] +46.1 pp 2,529.3 0 256
ch08 expo-b0.1_s1 so101 sim 50.4% [44.3, 56.5] 96.1% [93.0, 97.9] +45.7 pp 2,384.7 0 256
ch08 expo-b0.1_s2 so101 sim 50.4% [44.3, 56.5] 94.1% [90.6, 96.4] +43.8 pp 2,384.7 0 256
ch08 bon-q_s0 (ablation) so101 sim 50.4% [44.3, 56.5] 93.4% [89.6, 95.8] +43.0 pp 163.7 0 256
ch08 expo-b0.05_s2 (ablation) so101 sim 50.4% [44.3, 56.5] 92.2% [88.2, 94.9] +41.8 pp 169.4 0 256
ch08 bon-q-n16_s2 (control) so101 sim 50.4% [44.3, 56.5] 91.8% [87.8, 94.6] +41.4 pp 167.6 0 256
ch08 expo-b0.1_s0 so101 sim 50.4% [44.3, 56.5] 91.0% [86.9, 93.9] +40.6 pp 2,384.7 0 256
ch08 bon-q-n16_s0 (control) so101 sim 50.4% [44.3, 56.5] 90.2% [86.0, 93.3] +39.8 pp 168.3 0 256
ch08 bon-q_s1 (ablation) so101 sim 50.4% [44.3, 56.5] 90.2% [86.0, 93.3] +39.8 pp 171.4 0 256
ch08 expo-b0.05_s1 (ablation) so101 sim 50.4% [44.3, 56.5] 90.2% [86.0, 93.3] +39.8 pp 166.5 0 256
ch08 bon-q_s2 (ablation) so101 sim 50.4% [44.3, 56.5] 89.8% [85.5, 93.0] +39.5 pp 165.8 0 256
ch08 bon-q-n16_s1 (control) so101 sim 50.4% [44.3, 56.5] 88.7% [84.2, 92.0] +38.3 pp 171.3 0 256
ch08 random-pick-8 (control) so101 sim 50.4% [44.3, 56.5] 45.3% [39.3, 51.4] -5.1 pp 0 0 256
ch08 expo-b2_s1 (ablation) so101 sim 50.4% [44.3, 56.5] 1.2% [0.4, 3.4] -49.2 pp 230.7 0 256
ch08 expo-b2_s0 (ablation) so101 sim 50.4% [44.3, 56.5] 0.0% [0.0, 1.5] -50.4 pp 237.6 0 256
ch08 expo-b2_s2 (ablation) so101 sim 50.4% [44.3, 56.5] 0.0% [0.0, 1.5] -50.4 pp 246.3 0 256
ch12 bon-q-n32_s2 so101 sim 50.4% [44.3, 56.5] 95.7% [92.5, 97.6] +45.3 pp 17,775.4 0 256
ch12 bon-q-n32_s0 so101 sim 50.4% [44.3, 56.5] 94.5% [91.0, 96.7] +44.1 pp 17,787.9 0 256
ch12 bon-q-n32_s1 so101 sim 50.4% [44.3, 56.5] 91.4% [87.3, 94.3] +41.0 pp 17,788.0 0 256
ch12 bon-random-n32 so101 sim 50.4% [44.3, 56.5] 48.8% [42.8, 54.9] -1.6 pp 1,128.8 0 256

Generated by tools/leaderboard.py from results/**/*.json; do not edit by hand. SR = success rate on the recorded init set with its Wilson 95% interval. Plots include all listed runs, including exploratory and ablation rows; chapter docs identify the headlines. Improvement robot-min = search + train rollouts (the final evaluation is not counted). A robot shown as so101 (hard) ran a task variant other than v1. Hardware rows show k/n, never a bare 100%.

success vs robot minutes

success vs human minutes

How this list is made

The first version (September 2026) came from an agent-assisted literature sweep: many search angles plus rounds of gap-finding (arXiv month by month, citation mining, company blogs and talks). Of the 952 papers and posts, 912 are marked checked or corrected in the catalog, and 40 are marked read-once; these are recorded review statuses, not independent replication. The 212 resources are also listed from source pages. tools/arxiv_meta.py checks every arXiv link in the paper list and its first-version date against the arXiv API and flags titles that do not match, and tools/check_links.py checks the non-arXiv links. The workflow for weekly updates is documented in docs/maintenance/weekly-sweep.md.

Numbers are as reported by the authors. Some are bar-chart readings or estimates from small trial counts, and industry numbers are self-reported. Check the paper before citing a number. Research and drafting were assisted by Claude (Anthropic), which also helped write the lastmile code and chapter notes. Codex (OpenAI) was also used for source checks and fixes.

Contributing

Add a row to data/papers.csv or data/resources.csv, run uv run python tools/build_awesome.py, and open a pull request. A good entry has a concrete one-line description: the mechanism, then the headline number with its baseline. See CONTRIBUTING.md.

Citation

@misc{awesome-hill-climbing,
  title = {Awesome Hill Climbing: last-mile improvement of robot policies},
  year  = {2026},
  url   = {https://github.com/afinitus/awesome-hill-climbing}
}

License

The list and the code are MIT-licensed (LICENSE). Robot models under lastmile/envs/assets/ keep their MuJoCo Menagerie licenses (SO-101: Apache-2.0, PiPER: MIT).

About

Awesome Hill Climbing: curated papers, code and courses on last-mile improvement of robot policies (RL fine-tuning, residual/steering, RECAP-style retraining, human corrections, test-time verifiers, reward & world models), plus a hands-on course on a simulated SO-101 arm (MuJoCo; real hardware planned). Updated weekly.

afinitus.github.io/awesome-hill-climbing/

Topics

awesomeawesome-listembodied-aifine-tuninghill-climbingimitation-learningreinforcement-learningrobot-learningroboticsvision-language-action

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages