Skip to content

[vla, train, data, docs] feat: add LoongForge-VLDA training and LIBERO evaluation - #285

Merged
nullnonenilNULL merged 11 commits into
baidu-baige:masterfrom
Dana57740:VLDA
Oct 19, 2026
Merged

nullnonenilNULL merged 11 commits into
baidu-baige:masterfrom
Dana57740:VLDA

Conversation

@Dana57740

Copy link
Copy Markdown
Contributor

Change Summary

Add LoongForge-VLDA with two-stage LIBERO training and policy-server evaluation. Stage 1 initializes from OpenPI π₀.₅ base and V-JEPA 2 weights and learns the latent world model; Stage 2 initializes from Stage 1 and jointly learns world-conditioned action prediction.

  • Add the VLDA model, configuration, three-stream Gemma integration, V-JEPA 2 encoder, teacher EMA, and checkpoint/statistics integration.
  • Add a LeRobot v2 reader for embedded images, future frames, negative actions, and quantile normalization.
  • Integrate VLDA with the shared dataset, trainer, model configuration, evaluation factory, and payload registries.
  • Add an optional cosine-decay start step for Stage 2's warmup/hold/decay schedule. Its default of zero retains decay immediately after warmup.
  • Add the cookbook/loongforge-vlda/ guide, two 8-GPU training launchers, and LIBERO evaluation configuration. Keep internal and evaluation identifiers as vlda.

Validation

  • Completed the full two-stage training workflow: Stage 1 training and checkpoint saving, followed by Stage 2 initialization from the Stage 1 checkpoint, training, and checkpoint saving.
  • Completed checkpoint-based LIBERO evaluation, verifying that the trained Stage 2 checkpoint and normalization statistics load correctly through the policy server and simulator workflow.
  • Shell syntax and launcher/configuration checks: 25/25 checks passed, including invocation outside the repository, argument overrides, and invalid-input cases.
  • Stage 1: 8 GPUs, global batch 96, 3 optimizer updates; compilation, FlexAttention, and gradient checkpointing enabled.
  • Stage 2: 8 GPUs, global batch 128, 3 optimizer updates; compilation, FlexAttention, and gradient checkpointing enabled.
  • Both stages exited successfully with finite losses/gradient norms and zero NaN or skipped iterations.
  • LIBERO closed-loop smoke test: 1 task, 1 episode, 20 steps using OSMesa; 20 finite 7-dimensional actions, 2 inference calls, 18 action-cache hits, and no recorded incidents.
  • All 32 changed Python files passed the CI-pinned Ruff 0.9.10 check and AST syntax checks. Changed files passed the sensitive-information scan with zero findings.
  • Full upstream Static Checks suite and regression coverage for other models.

CI, GPU, and Image Impact

  • This change documents model and simulator environment requirements; it does not modify dependency manifests, Dockerfiles, or patches.
  • GPU validation is required. Suggested suite: embodied; the completed VLDA training, checkpoint-based evaluation, and additional smoke tests supplement the standard regression suite.
  • A local candidate image should be built during GPU validation.
  • No GPU or image validation is required.

Reviewer Notes

VLDA training currently supports DDP and DDP with ZeRO1; FSDP is not supported. Review the optional shared LR-scheduler argument and validation alongside the new model-specific paths. V-JEPA 2 code retains its upstream MIT license and attribution.

Before marking this draft ready: synchronize with the latest upstream master, resolve SPDX-header checks while retaining third-party notices, remove the README's trailing whitespace, and replace or remove its paper/framework placeholders. Local test artifacts are kept outside the committed source tree.

@nullnonenilNULL
nullnonenilNULL merged commit 8cbdcea into baidu-baige:master Oct 19, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

checkpoint ci docker documentation Improvements or additions to documentation model training

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants