CVPR 2026
We present PhysInOne, the largest dataset addressing the critical scarcity of physically-grounded training data for AI systems.
- 2 million videos generated from 153,810 dynamic 3D scenes
- Covers 71 fundamental physical phenomena in everyday environments, spanning four major domains: Mechanics, Optics, Fluid Dynamics, Magnetism
- Includes 2,231 common objects tailored to daily physical interactions
- Enriched with 623 materials across five categories: plastic, metal, wood, stone, and fabric
- Features 528 diverse 3D backgrounds to ensure realism and environmental variety
- Each scene involves 1–3 physical phenomena, reflecting real-world activities
- Supports complex multi-object interactions, with increasing scene complexity
- Average number of objects per scene: 3.9 (single-physics), 6.3 (double-physics), 7.8 (triple-physics)
- Each scene is captured from 13 viewpoints: 12 static cameras and 1 moving camera
- 3D geometry
- Semantic labels
- Object motion and dynamics
- Physical properties
- Natural-language scene descriptions
- Physics-aware video generation
- Short- and long-term future frame prediction
- Physical property estimation
- Motion transfer
- And more...
| Component | Progress | Status | Notes |
|---|---|---|---|
| SubSet | ██████████100% |
Released | |
| Rendered Data - Train | ██████████ 100%(122988/122988) |
Released | Last updated: Aug 21 |
| Rendered Data - Test | ██████████ 100% |
Released | All Leaderboard user inputs released; GT excluded |
| Rendered Data - Val | ░░░░░░░░░░ 1%(103/15411) |
In progress | |
| 3D Assets | ██░░░░░░░░ 20% |
Partially released | Validation assets for 1,000 scenes |
| Leaderboard | ██████████ 100% |
Released | Public evaluation inputs for all four tasks; GT excluded |
| PMF | ██████████100% |
Released | |
| Baselines | ███░░░░░░░25% |
In progress | Last updated: Jul 23 |
| Data processing | ░░░░░░░░░░ 0% |
Not released | Expected around Aug |
| Resource | Link |
|---|---|
| 📄 Paper | arXiv |
| 🌐 Project Page | vlar-group.github.io/PhysInOne |
| 🤗 Dataset | Hugging Face |
| 🏆 Leaderboard Data | Public evaluation inputs |
| 🧊 3D Assets | PhysicBenchmark project assets |
All user-facing evaluation inputs required by the public Leaderboard have been released for the four benchmark tasks. Ground-truth outputs remain private and are not included in the public download.
| Task | Public package count |
|---|---|
| Video Generation | 75,865 files |
| Future Prediction | 103 scene ZIP archives |
| Physical Properties Estimation | 72 scene ZIP archives and 2 shared support files |
| Motion Transfer | 217 scene ZIP archives |
Use the Leaderboard download script with the task-specific download lists. See the English download guide for commands, filtering, resume behavior, and integrity checks.
The first validation release contains project resources for 1,000 scenes: 4,299 files plus 8 ZIP archives, totaling approximately 22.25 GiB. Install Unreal Engine 5.5.4; Windows is recommended for the simplest setup, while Linux is also supported with additional configuration.
Download the PhysicBenchmark project folder, then run the 3D asset download script. The script preserves the repository layout and extracts the packaged assets into the project tree. Consult the setup guide and validation file list, then launch PhysicBenchmark/PhysInOne.uproject.
Due to the large scale of PhysInOne, the rendered data and annotations are split across 16 Hugging Face repositories. Each entry shows the shard size, release status, live all-time downloads, live downloads in the last 30 days, and its repository link.
Combined snapshot (Sep 5, 2026): P01–P16 have 986,449 all-time downloads and 615,484 downloads in the last 30 days. Including the main repository, the per-repository sums are 999,927 and 616,472.
Download badges query the official Hugging Face API and update automatically. Counts are repository-level download events, not deduplicated users; accessing multiple shards can produce one event in each shard.
The PMF (Physical Motion Fidelity) evaluates video similarity in the frequency domain using 3D FFT-based energy distributions. It is designed for physics-aware video generation, future prediction, and motion transfer tasks in the PhysInOne benchmark.
# Step 1: Install PyTorch first (choose your variant)
# CPU only:
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cpu
# CUDA 12.6:
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
# Step 2: Install pmf from this repo
pip install git+https://github.com/vLAR-group/PhysInOne.git#subdirectory=pmf#!/usr/bin/env python
"""Test PMF metric with random tensors."""
import torch
from pmf import compute_pmf
def main():
torch.manual_seed(42)
B, T, C, H, W = 1, 81, 3, 128, 128
video_pred = torch.randn(B, T, C, H, W)
video_gt = torch.randn(B, T, C, H, W)
score = compute_pmf(video_pred, video_gt, device='cpu')
# If you want to use gpu, set device='cuda'
# score = compute_pmf(video_pred, video_gt, device='cuda')
if isinstance(score, torch.Tensor):
score = score.item()
print(f"PMF similarity score: {score:.4f}")
if __name__ == "__main__":
main()We provide baseline implementations under the ./baselines directory for your reference. We welcome your feedback, please feel free to contact us if you need anything..
📅 Update Schedule: This section is actively being updated throughout July and August.
Data processing code will be released soon. Stay tuned!
If you find this work useful, please cite:
@article{zhou2026physinone,
title={PhysInOne: Visual Physics Learning and Reasoning in One Suite},
author={Siyuan Zhou and Hejun Wang and Hu Cheng and Jinxi Li and Dongsheng Wang and Junwei Jiang and Yixiao Jin and Jiayue Huang and Shiwei Mao and Shangjia Liu and Yafei Yang and Hongkang Song and Shenxing Wei and Zihui Zhang and Peng Huang and Shijie Liu and Zhengli Hao and Hao Li and Yitian Li and Wenqi Zhou and Zhihan Zhao and Zongqi He and Hongtao Wen and Shouwang Huang and Peng Yun and Bowen Cheng and Pok Kazaf Fu and Wai Kit Lai and Jiahao Chen and Kaiyuan Wang and Zhixuan Sun and Ziqi Li and Haochen Hu and Di Zhang and Chun Ho Yuen and Bing Wang and Zhihua Wang and Chuhang Zou and Bo Yang},
year={2026},
journal={CVPR}
}This project is licensed under the CC BY-NC-SA 4.0 license.
We would like to express our sincere gratitude to all contributors who participated in human evaluations and data collection efforts.