ΔVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variation

Yijie Zhu^1,2, Jie He¹, Rui Shao*¹, Kaishen Yuan³, Xiaochen Yuan⁴, Tao Tan⁴, Zitong Yu*²,

¹School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen
²Great Bay University ³The Hong Kong University of Science and Technology, Guangzhou
⁴ Macao Polytechnic University

*Corresponding author

🛠️ We're still cooking — Stay tuned!🛠️
⭐ Give us a star if you like it! ⭐
✨If you find this work useful for your research, please kindly cite our paper.✨

🔥 Introduction

ΔVLA is a prior-guided vision‑language‑action model that reasons about world knowledge variation instead of directly predicting future states. Its core idea shifts the learning paradigm from forecasting absolute outcomes to modeling the process of change, thereby enhancing causal consistency and efficiency in robotic manipulation. Unlike previous methods that directly predict future images or world states, ΔVLA first constructs an explicit current‑world knowledge prior through the Prior‑Guided World Knowledge Extractor (PWKE), which extracts actionable regions, semantic cues, and geometric depth from the visual input. Building on this prior, the model encodes how knowledge evolves under actions via the Latent World Variation Quantization (LWVQ), representing changes in a discrete latent space rather than reconstructing full future modalities. Furthermore, to reduce cross‑modal interference during variation modeling, ΔVLA introduces the Conditional Variation Attention (CV‑Atten) mechanism, which isolates attention flows across different knowledge types to promote disentangled causal reasoning.

We also provide a comprehensive comparison of different paradigms for world‑knowledge modeling and robotic control (see below).

The overall framework of ΔVLA is illustrated below: given a visual observation and a task instruction, ΔVLA first extracts an explicit world knowledge prior via PWKE, then encodes world changes into discrete latent variables through LWVQ. These prior representations along with the variation tokens are fed into a large language model, which performs reasoning internally using a Conditional Variation Attention (CV‑Atten) mechanism to ensure disentanglement among semantic, geometric, and regional variations, ultimately generating precise and causally consistent action sequences.consistent action sequences.

Videos

Task1-Drawer.Manipulation.mp4

Task2-Align.Shoes.mp4

Task3-T-shirt.Folding.mp4

Task4-Plate.Lemon.Apple.mp4

Acknowledgement

• Lots of code are inherited from OpenVLA-oft. Thanks for this great work.