English | 简体中文
LoongForge is an open-source training framework developed by the Baidu AI Cloud Baige team, built to deliver faster training for mainstream LLMs, VLMs, diffusion, and embodied models.
- High Performance — Built on Megatron-LM and torch-native backends, with deep optimizations for training speed that keep training loss aligned with the baseline.
- Easy to Use — Ready-to-run configs and launch scripts for 40+ open-source models, plus checkpoint conversion and data preprocessing tools. Supports pre-training, mid-training, SFT, and LoRA.
- Production-Proven — Open-sourced from AIAK-Training-LLM, which powers enterprise customers' proprietary models and several open-source projects. Production runs reach 5,000+ XPUs.
Example: training the embodied model DreamZero on LoongForge reaches 4.38× the open-source baseline's throughput, with loss curves staying aligned
⭐ Give us a Star to help more people discover LoongForge and grow the community.
Since optimal training strategies differ across model families and scales, LoongForge adopts a multi-backend architecture.
- Megatron Backend — For LLM, VLM, and diffusion models, extended with MoE parallelism, per-component heterogeneous parallelism, long-sequence optimizations, etc.
- torch-native Backend — For embodied models (VLA and WAM). Built on native PyTorch with DDP / ZeRO-1 / FSDP / HSDP, with deep optimizations for representative models across I/O, communication strategy, kernel efficiency, etc.
- [2026/10] 🤖 Introducing LoongForge-VLDA, a Vision-Language-Delta-Action model trained with LoongForge: 98.5% on LIBERO, 84.5% zero-shot on LIBERO-Plus, and +17–18 points over π0.5 on real robots. [project page] [paper]
- [2026/10] 🦾 Open-sourced ego2robot — a pipeline that converts first-person videos of human manipulation into dual-arm robot training data, outputting LeRobot v3.0 format across 16 robot morphologies.
- [2026/09] ✨ Added training support for GLM-5.3-flash.
- [2026/09] ✨ Added Kimi-K3 BF16 training support for both LLMs and VLMs.
- [2026/09] ⚡ Added an optimized DreamZero Wan2.2-5B FSDP recipe with cache-aware data loading, compiled attention blocks, frozen-module handling, and FSDP2 Delta-FP8 Param AllGather.
- [2026/08] 🤖 Added VLA training support for Wall-OSS-0.5, with custom fused operators for higher training throughput.
- [2026/08] 📄 Released the TAOT paper — topology-aware dynamic expert replica placement that tackles expert-parallel (EP) load imbalance in MoE training, cutting overhead by up to 74% over industry solutions, with 1.43× speedup measured on a real training case. [blog]
- [2026/08] ✨ Added training support for GLM-5.2, along with a GLM-5.2 + MoonViT custom-composition example for extending GLM with multimodal capabilities.
- [2026/08] ✨ Added training support for MiniCPM-V-4.6 and Qwen3.8-27B.
- [2026/08] 🧪 Introduced a unified evaluation module for the embodied stack, currently covering Pi0.5 / xVLA / GR00T, with more models on the way.
📅 More
- [2026/07] 🐳 Unified the prebuilt Docker images — all model families (LLM / VLM / Embodied / Diffusion) now share a single image.
- [2026/07] 🤖 Released LoongForge-Embodied, a torch-native DDP/FSDP training subsystem for embodied models (Pi0.5, GR00T-N1.6/N1.7, xVLA, LingBot-VA, FastWAM, DreamZero, and Cosmos3), with up to 4.38× speedup.
- [2026/07] ✨ Added training support for DeepSeek-V4-Flash / DeepSeek-V4-Pro.
- [2026/07] ✨ Added training support for Qwen-Image-Edit-2511.
- [2026/06] 🤖 Expanded VLA coverage with GR00T N1.6; 2.3× speedup on GR00T training. [blog]
- [2026/05] ⚡ Accelerated Wan 2.2 training by 116%, and added CP and data packing support.
- [2026/05] ✨ Added training support for Kimi K2.5 / K2.6, and introduced INT4 / NVFP4 PTQ.
- [2026/05] 🎉 v0.1.0 — first official tagged release of LoongForge.
- [2026/05] 🌟 Powered the training and public release of LLaVA-OneVision-2.0.
- [2026/04] 🧩 Added training support for MiniMax-M2.7 on both NVIDIA GPU and Kunlun XPU.
- [2026/04] 🚀 LoongForge source code publicly available on GitHub. [blog]
- [2025/10] 🌟 Powered the training and public release of LLaVA-OneVision-1.5 under AIAK-Training-LLM, the predecessor of LoongForge. [blog]
Data processing, training, and evaluation for Vision-Language-Action (VLA) and World-Action (WAM) models, with 1.79×–4.38× training throughput over open-source baselines across benchmarked models. [Docs] (Paper coming soon)
- Unified Data Loading — A unified data-loading interface supporting mainstream formats like LeRobot (v2.0 / v2.1 / v3.0); each model's data pipeline matches its original open-source implementation.
- Feature Cache Precomputation (optional) — Precompute frozen-encoder outputs for DreamZero (VAE latents, text embeddings) and FastWAM (text embeddings) ahead of training, read directly during training to avoid re-encoding every step. [DreamZero] [FastWAM]
- Ego2Robot — Converts first-person videos of human manipulation into LeRobot v3.0 training data across 16 robot morphologies, each adapted to its own joint layout and gripper. [README]
| Human Demonstration (Input) | Ego2Robot Output Examples (Multiple Robot Morphologies) |
|---|---|
 |
 |
- Per-Model Deep Optimization — Custom optimizations targeting each model's bottlenecks (I/O, communication strategy,
torch.compile, CUDA Graph, custom fused operators, etc.), reaching 1.79×–4.38× measured throughput over open-source baselines (see Performance). - Precision Alignment — Training loss aligns with open-source baselines; comparison curves are already available for several models and will keep expanding alongside new model support. [Precision Comparisons]
- Multiple Distributed Strategies — Supports DDP / DDP + ZeRO-1 / FSDP / HSDP, selectable by model scale and cluster topology. [Tutorial]
- FP8 Communication Optimization (optional) — On supported NVIDIA GPUs, compresses cross-rank traffic to FP8: blockwise FP8 delta compression for FSDP2 parameter AllGather, and FP8 compression for DDP gradient all-reduce. [Usage]
- Unified Evaluation — Closed-loop evaluation of trained policies in the LIBERO / CALVIN / SimplerEnv / RoboTwin / ManiSkill simulators; coverage is expanding continuously. [README]
Note
Coming soon: evaluation results for models trained with LoongForge.
Built on the Megatron backend, supporting large-scale pre-training and SFT for LLMs and VLMs, as well as diffusion model training. Key optimizations for each scenario:
- MoE EP Communication Optimization — Overlapped All2All / activation offload / compute, with further memory reduction beyond upstream Megatron-LM. [Usage]
- MoE Expert Load Balancing — Topology-aware dynamic replication of hot experts to balance EP workloads, with up to 74% lower overhead than industry solutions. [TAOT Paper]
- Adaptive FP8 Training — End-to-end FP8 for LLMs and VLMs with standard blockwise FP8; an optional adaptive mode picks per-operator precision by GEMM shape and efficiency. [Usage]
- Custom Fused Operators — Fused kernels like FusedDSA for DSA-style models — TileLang version open-sourced, high-performance CUDA version available on Baidu Baige platform.
- Long-Sequence Training — Scales LLM training to long sequences via Context Parallel (CP) and chunked-pipeline scheduling.
- Flexible Composition — Assemble VLMs from interchangeable ViT and LLM components (e.g. GLM-5.2 + MoonViT) straight from config — no custom model code. [Usage]
- Heterogeneous Parallelism — Independent TP / DP / recompute / freeze per model component (e.g. ViT vs. LLM) for optimal throughput and memory. [blog] [Usage]
- Decoupled Encoder-Decoder Training — Eliminates encoder-induced pipeline bubbles by separating ViT and LLM into independent tasks. [Usage]
- DP Load Balancing — Improves multi-node scaling efficiency with load-aware data redistribution that mitigates sequence-packing imbalance. [blog] [Usage]
- Flexible Data Pipeline — Energon WebDataset input for multimodal data, with both online and offline sequence packing. [Usage]
- Context Parallelism — Supports Ring, Ulysses, and hybrid context parallelism (CP) to split long video sequences across GPUs.
- Variable-Length Sample Packing — Packs multiple variable-length video samples into a single sequence to reduce padding waste, compatible with context parallelism. [Usage]
- Mcore Bridge — Supports both offline bidirectional Megatron ↔ HuggingFace conversion and online native HF load/save. [Usage]
- Heterogeneous Hardware — Native support for NVIDIA GPUs and Kunlun XPUs via a minimally-intrusive plugin design.
Training throughput speedups over mainstream open-source baselines — each model and its baseline were benchmarked on the same machine type with the same training hyperparameters:
DeepSeek-V3.2 Lite reflects DSA operator-level optimizations and was validated on a reduced-layer configuration due to test-bed scale limits.
Numbers were measured at a point in time and may evolve as implementations change on both sides.
Using NVIDIA GPU as an example, pull the prebuilt image and start a container (requires the NVIDIA Container Toolkit):
docker pull loongforge/loongforge:latest
mkdir -p workspace
docker run --gpus all --ipc=host -it --rm \
-v "$(pwd)/workspace:/workspace/data" \
-w /workspace/LoongForge \
loongforge/loongforge:latest bashThe example above mounts
./workspaceto/workspace/datainside the container, for storing weights, datasets, and training outputs — these persist after the container exits.
For more installation options including source builds, see: Full GPU installation guide · Kunlun XPU installation guide
- NVIDIA GPU: LLM · VLM · VLA & WAM (Embodied) · Diffusion
- Kunlun XPU: Kunlun XPU Tutorials
Launch scripts for every supported model are already included in this repo: NVIDIA GPU scripts live under examples/, Kunlun XPU scripts under examples_xpu/ — see Supported Models for the full list.
The example below walks through DDP fine-tuning of Pi0.5 on LIBERO (single node, 8 GPUs); run all commands inside the container above.
① Prepare weights, tokenizer, and dataset (hf is the Hugging Face CLI):
hf download lerobot/pi05_base --local-dir /workspace/data/pi05_base
hf download google/paligemma-3b-pt-224 --local-dir /workspace/data/paligemma-3b-pt-224
hf download lerobot/libero_10 --repo-type dataset --local-dir /workspace/data/libero_10② Launch training (paths are set via environment variables; defaults to 20 steps to validate the workflow):
export CHECKPOINT_PATH=/workspace/data/pi05_base
export TOKENIZER_PATH=/workspace/data/paligemma-3b-pt-224
export DATA_PATH=/workspace/data/libero_10
export OUTPUT_DIR=/workspace/data/outputs/pi05
bash examples/embodied/pi05/run_pi05_ddp_finetune.shLogs and TensorBoard data are saved under OUTPUT_DIR. For FSDP / ZeRO-1 launch options and LIBERO evaluation, see the Pi0.5 tutorial.
Click any model for its training examples; see configs/models/ for the matching parameter configs. See the model support matrix for all variants.
| Embodied | LLM | VLM | Diffusion |
|---|---|---|---|
Tip
For VLMs, also see the custom composition example, which assembles a new model from a ViT and an LLM purely through configuration.
See more on the Projects page, and if you've built something with LoongForge (or its predecessor, AIAK-Training-LLM), let us know — we'd love to feature it.
- LLaVA-OneVision-2.0 — Next-generation multimodal model, with new VideoCaption and Spatial datasets
- Innovator-VL — Scientific multimodal LLM for advanced reasoning
- LLaVA-OneVision-1.5 — Fully open framework for democratized multimodal training
- Qianfan-VL — Domain-enhanced vision-language models for enterprise, 3B–70B parameters
- LoongForge-VLDA — Vision-Language-Delta-Action model for robotic manipulation: 98.5% on LIBERO, 84.5% zero-shot on LIBERO-Plus, and 17–18 points ahead of π₀.₅ on real robots.
loongforge-vlda.mp4
- VLA 训练如何一步步提速：以 Pi0.5 为例的 3 倍加速实践
- 百度百舸 LoongForge 团队提出 TAOT：拓扑感知传输破解大规模 MoE 训练端到端加速 1.43 倍
- 加速 GR00T N1.6 具身 VLA 模型训练！LoongForge 开源框架实现三层全链路优化，吞吐提升 2.3 倍、训练周期缩短 56.6%
- 多模态模型终于不用「抽帧看世界」？LLaVA-OneVision-2.0 全开源全帧率技术解读
- LoongForge 多模态异构并行训练加速，从问题到方案的全面解析
- 百度百舸开源全模态训练框架 LoongForge：一套代码跑通 GPU 与昆仑芯，多模态训练提速 45%
- 刷新记录！百度百舸助力 LLaVA-OneVision-1.5 高效训练
📁 Directory tree
LoongForge/
├── loongforge/ # Core training framework
│ ├── train/ # Training entry points and trainers
│ │ ├── pretrain/ # Pretraining
│ │ ├── sft/ # Supervised fine-tuning
│ │ └── diffusion/ # Diffusion training
│ ├── models/ # Unified model abstractions
│ │ ├── foundation/ # LLM backbones
│ │ ├── encoder/ # Vision encoders
│ │ ├── omni_models/ # Multimodal composition (encoder + projector + decoder)
│ │ ├── diffusion/ # Diffusion models
│ │ └── common/ # Shared base configs, layers and utilities
│ ├── embodied/ # Embodied (VLA + world-action) training: models, data, trainers, distributed, eval
│ ├── data/ # Data loading and processing pipelines
│ ├── tokenizer/ # Tokenizers
│ └── utils/ # Config registry, constants and shared helpers
├── third_party/ # Third-party dependencies and patches
│ ├── Loong-Megatron/ # Patched Megatron-LM (git submodule)
│ └── patches/ # TransformerEngine patches
├── configs/ # Hydra YAML configs (models, data)
├── examples/ # Launch scripts for NVIDIA GPUs
├── examples_xpu/ # Launch scripts for Kunlun XPUs
├── cookbook/ # End-to-end project recipes built on LoongForge
├── tools/ # Standalone dev tools: checkpoint conversion, data preprocessing, etc. (see tools/README.md)
├── ops/ # Custom fused operators (incl. open-sourced TileLang)
├── docker/ # Dockerfiles (GPU & XPU)
├── tests/ # Test suites (E2E, regression, unit)
└── docs/ # Documentation
If you find LoongForge helpful, please consider citing this project:
@software{LoongForge2026,
title = {LoongForge: A high-performance framework for training LLMs, VLMs, diffusion, and embodied models},
author = {{The LoongForge Authors}},
year = {2026},
url = {https://github.com/baidu-baige/LoongForge}
}If you find TAOT helpful, please consider citing our paper:
@article{zhang2026taot,
title = {{TAOT}: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in {MoE} Training},
author = {Zhang, Lingyun and Zhang, Henghua and Gu, Shilei and Mo, Kai and Han, Shuai and Li, Shiyong and Wang, Yanpeng and Shen, Dou},
journal = {arXiv preprint arXiv:2608.03676},
year = {2026},
url = {https://arxiv.org/abs/2608.03676}
}If you find LoongForge-VLDA helpful, please consider citing our paper:
@article{zhang2026vlda,
title = {{Vision-Language-Delta-Action}: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation},
author = {Zhang, Lingsen and Li, Wei and He, Jie and Zhang, Henghua and Mo, Kai and Li, Shiyong and Wang, Yanpeng and Shao, Rui},
journal = {arXiv preprint arXiv:2610.25874},
year = {2026},
url = {https://arxiv.org/abs/2610.25874}
}We warmly welcome community contributions — bug reports, feature proposals, and PRs alike. Please read our Contributing Guidelines before submitting.
Thanks to all our contributors:
| Channel | What it's for |
|---|---|
| GitHub Issues | Bug reports, usage questions, and feature requests |
| Developer Communities | WeChat group, Xiaohongshu, and more |
| Enterprise adoption, large-scale deployment, partnership, or any other topic |
LoongForge stands on the shoulders of the open-source community. Its Megatron backend builds on NVIDIA's Megatron-LM, and the project also draws on HuggingFace Transformers, LLaMA-Factory, Megatron-Bridge, LeRobot, and the official implementations of the models we support (e.g. OpenPI, NVIDIA Isaac GR00T).
We also thank the LINUX DO community for providing a welcoming space for technical discussion, and AtomGit for hosting LoongForge's official mirror.
LoongForge is released under the Apache License 2.0; some files derive from third-party projects — see their file headers.