World-Calibrated Proposal-to-Action Flow
for Vision-Language-Action Models

Jie He^1,*, Wei Li^1,*, Junwen Tong^2,*, Rui Shao^1,†, Wei-Shi Zheng³, Liqiang Nie¹

¹Harbin Institute of Technology, Shenzhen
²State Key Laboratory of Mobile Network and Mobile Multimedia Technology, ZTE, China
³Sun Yat-sen University
^*Equal contribution
^†Corresponding author

ProAct: propose, calibrate, and refine action generation with world-aware motion priors.

⭐ Give us a star if you like it! ⭐
✨ If you find this work useful for your research, please kindly cite our paper. ✨

🔥 Introduction

Flow-based Vision-Language-Action (VLA) policies usually generate an action chunk by transporting samples from a task-agnostic isotropic Gaussian source. Although this formulation is expressive, it does not explicitly use the motion that has already been executed, and therefore spends part of the denoising trajectory recovering a plausible direction of motion.

ProAct introduces a world-calibrated proposal-to-action framework that turns recent interaction history into a scene-aware, task-consistent flow source. The framework follows a three-stage process:

• Propose. A lightweight Proposal Expert converts recent executed actions into a momentum-based motion proposal and makes it scene-aware through a single endpoint flow step.
• Calibrate. A proposal-aware World Expert uses VLM features, proposal condition tokens, and learnable spatiotemporal queries to predict the desired future representation with a frozen V-JEPA 2 target. It also predicts the refinement extent and low-rank geometry of the action source.
• Refine. An Action Expert starts from the proposal-centered anisotropic source and performs the final flow-matching refinement for the current observation.

The resulting source distribution preserves the efficiency of proposal-guided generation while allowing the world model to correct proposals that are inconsistent with the current scene.

ProAct uses recent executed motion as a local proposal, calibrates it with prospective world reasoning, and then refines the calibrated source into an executable action chunk.

⚙️ Method

Given an action history, ProAct first computes an exponentially smoothed motion estimate:

$$ m_t = \beta m_{t-1} + (1-\beta)\Delta a_t. $$

The proposal extrapolates this motion over the action horizon while keeping gripper dimensions at their latest executed values. The Proposal Expert then uses VLM key-value features to obtain a scene-aware proposal.

The World Expert treats the proposal as a soft motion prior, rather than copying it directly. It predicts a future visual latent and calibration parameters for an anisotropic source distribution:

$$ C = \sigma^2 \alpha (I + BB^\top), $$

where $\sigma$ controls the refinement extent and $B$ captures a low-rank geometry of the action distribution. The final Action Expert transports samples from this calibrated source to the demonstrated action distribution.

This design separates coarse motion estimation, world-state calibration, and fine-grained action refinement. It also enables fewer denoising steps without requiring the proposal to be correct in every dimension.

The framework combines a Proposal Expert, a proposal-aware World Expert, and an Action Expert. The World Expert uses future-latent prediction and proposal compatibility to determine how the action source should be centered, scaled, and geometrically shaped before final flow refinement.

📊 Results

ProAct is evaluated on simulated manipulation benchmarks and two real-world robot platforms. The reported results are taken from the paper's final evaluation tables.

Benchmark	ProAct result
LIBERO average	98.4%
LIBERO-Plus average	86.8%
RoboTwin 2.0 Randomized (Hard) ten-task subset	60.3%

On RoboTwin 2.0, the reported task success rates are 69% on Put Bottles Dustbin, 39% on Place Dual Shoes, 57% on Pick Diverse Bottles, 84% on Place Burger Fries, 76% on Open Laptop, 46% on Blocks Ranking RGB, 63% on Pick Dual Bottles, 78% on Dump Bin Bigbin, 29% on Stamp Seal, and 62% on Place Bread Basket.

Platform	TSR	CSR
GALAXEA R1 Lite	78.7%	86.1%
AgileX Cobot Magic	74.7%	82.3%

Compared with π0.5, ProAct uses 50% fewer denoising steps, with up to 25.8% lower latency and up to 34.8% higher throughput in the reported efficiency comparisons. Exact latency depends on hardware, compilation mode, checkpoint, and the measured input pipeline.

💻 Usage

Use Linux, Python 3.11, and an NVIDIA CUDA GPU for training and inference. Simulator clients have separate dependencies; do not install the LIBERO simulator environment over the main training environment.

Installation

git submodule update --init --recursive
uv venv --python 3.11
GIT_LFS_SKIP_SMUDGE=1 UV_LINK_MODE=copy uv sync
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .
cp -r src/openpi/models_pytorch/transformers_replace/* \
    .venv/lib/python3.11/site-packages/transformers/

The patches require the pinned transformers==4.53.2. They implement adaRMS and non-mutating expert KV-cache reads. Reapply the patches if Transformers is reinstalled.

Data and weights

• LIBERO: use the physical-intelligence/libero LeRobot dataset, or follow the LIBERO preparation instructions.
• RoboTwin: prepare a LeRobot dataset with the schema described in src/openpi/training/config.py. The registered dataset ID is lerobot_v21; it is not a promised public Hub dataset.
• Set HF_LEROBOT_HOME to the local dataset cache. Set HF_HUB_OFFLINE=1 only after all required assets have been cached.
• Training requires the converted pi05_base PyTorch initialization checkpoint and the V-JEPA 2-AC ViT-g encoder checkpoint. The default V-JEPA path is checkpoints/vjepa2/vjepa2-ac-vitg.pt; the default PyTorch checkpoint path is checkpoints/pi05_base_pytorch; launch scripts accept explicit overrides.
• V-JEPA is a frozen, lazily loaded training target. Its weights and future frames are not required by sample_actions at inference.

Compute normalization statistics before training:

uv run scripts/compute_norm_stats.py --config-name proact_libero
# Or, with the RoboTwin dataset cache configured:
uv run scripts/compute_norm_stats.py --config-name proact_robotwin

Use examples/convert_jax_model_to_pytorch.py to convert π0.5 checkpoints. A trained ProAct checkpoint is different from its π0.5 initialization checkpoint. A serving checkpoint directory contains:

model.safetensors
metadata.pt
assets/<asset_id>/norm_stats.json

Keep metadata.pt: serving restores model overrides saved during training. Only load trusted checkpoints; the metadata and V-JEPA loaders use PyTorch pickle loading.

The optional download.sh helper requires explicit CHECKPOINT_REPO and CHECKPOINT_FILE values for a pi05_base archive. It stores the verified PyTorch initialization at checkpoints/pi05_base_pytorch and rejects archives that do not contain model.safetensors; it no longer selects a RoboTwin-specific checkpoint by default.

Training and evaluation

The environment-variable-driven launchers are in scripts-sh/:

Task	Entry point
ProAct on LIBERO	scripts-sh/train_libero.sh
π0.5 LIBERO baseline	scripts-sh/train_pi05_libero.sh
ProAct on RoboTwin	scripts-sh/train_robotwin.sh
Policy serving	scripts-sh/serve_libero.sh, scripts-sh/serve_robotwin.sh
LIBERO evaluation	scripts-sh/eval_libero.sh
LIBERO-Plus evaluation	scripts-sh/serve_libero_plus.sh, scripts-sh/eval_libero_plus.sh

For a new LIBERO training run:

VJEPA_CHECKPOINT=/path/to/vjepa2-ac-vitg.pt \
EXP_NAME=proact_public RESUME=0 NPROC_PER_NODE=4 \
bash scripts-sh/train_libero.sh

For RoboTwin, set HF_LEROBOT_HOME to the directory containing lerobot_v21 and use scripts-sh/train_robotwin.sh. The launcher uses the paper's num_world_queries=4 and the same default pi05_base_pytorch initialization.

To serve a trained policy directly:

uv run scripts/serve_policy.py policy:checkpoint \
    --policy.config proact_libero \
    --policy.dir /path/to/proact/checkpoint \
    --policy.num-steps 5

Use proact_robotwin for RoboTwin. Clients must provide executed action history, not the unexecuted remainder of a predicted chunk. Reset history at episode boundaries, and keep the action history and action horizon consistent with the checkpoint.

Efficiency benchmark

The demos are text-only: they print predicted/demonstrated actions and measure model-forward latency and peak allocated CUDA memory. They do not generate figures or call private visualization forwards.

uv run examples/libero/demo.py \
    --config proact_libero --checkpoint /path/to/proact/checkpoint \
    --num-steps 5 --warmup 5 --repeats 20 --compile-mode none

uv run examples/libero/demo.py \
    --config pi05_libero --checkpoint /path/to/pi05/checkpoint \
    --num-steps 10 --warmup 5 --repeats 20 --compile-mode none

uv run examples/robotwin/demo.py \
    --checkpoint /path/to/proact_robotwin/checkpoint --num-steps 5

The first run caches one transformed dataset sample under .cache/benchmarks/; subsequent runs reuse it. Use --frame for an explicit path and --rebuild-frame after changing the dataset, index, checkpoint normalization, or model configuration. Shape checks cannot detect a stale cache with the same shapes. Do not share a ProAct cache with a baseline.

The benchmark includes warmup and CUDA synchronization. It includes the VLM and all enabled experts, but excludes loading, network transport, action unnormalization, and simulation. Compilation/startup is excluded from the measured loop. This cached-frame benchmark is not an end-to-end robot control rate or a task success-rate evaluation.

Code map

• src/openpi/models_pytorch/proact_pytorch.py: momentum proposal, experts, KV fusion, calibration, losses, and flow sampling.
• src/openpi/models/proact_config.py: model settings and calibration options.
• src/openpi/models_pytorch/vjepa_target.py: frozen future-latent targets.
• src/openpi/training/config.py, data_loader.py, and src/openpi/transforms.py: benchmark datasets, history/future windows, and normalization.
• src/openpi/policies/ and examples/{libero,robotwin}/: serving and clients.
• src/openpi/shared/action_benchmark.py: shared efficiency-demo utilities.

The existing ProAct regression tests are in src/openpi/models_pytorch/proact_pytorch_test.py. Full model tests require CUDA and do not substitute for GPU training, checkpoint inference, or simulator evaluation.

📂 Code Release

The repository provides the core ProAct model, training configuration, serving entry points, benchmark clients, and text-only efficiency demos.

• ProAct model, proposal/world/action experts, data transforms, and frozen V-JEPA 2 future-latent targets
• LIBERO/RoboTwin training and serving configurations, together with text-only efficiency benchmarks
• Public pretrained ProAct checkpoints and a complete RoboTwin rollout adapter

For RoboTwin, this checkout includes the dataset configuration, training launcher, and policy server. A complete simulator-side rollout adapter is not included; the upstream evaluator uses a different protocol from the OpenPI policy server.

This repository contains implementation code used for the paper. Code availability alone is not a claim that the reported benchmark results have been independently reproduced from this checkout.

📝 Citation

@article{proact2026,
  title={World-Calibrated Proposal-to-Action Flow for Vision-Language-Action Models},
  author={He, Jie and Li, Wei and Tong, Junwen and Shao, Rui and Zheng, Wei-Shi and Nie, Liqiang},
  journal={arXiv preprint arXiv:2610.02323},
  year={2026}
}

📜 Attribution and license

Preserve the existing Apache-2.0 license, Gemma notice, and upstream copyright notices. V-JEPA 2, LIBERO, LIBERO-Plus, RoboTwin, model weights, and datasets retain their respective licenses and redistribution terms; the repository license does not relicense these artifacts.

Local datasets, checkpoints, manuscript sources, and generated plots are not release artifacts. Before publication, confirm licensing authority, checkpoint/dataset redistribution permissions, publication metadata, and the intended public repository URL.