Physical AI and robotics are moving from headlines and experimentation into real-world industrial deployment. The shift creates practical implications for how human-robot-AI collaboration becomes an operational capability in manufacturing, logistics, healthcare, and autonomous systems. Operationalizing physical intelligence at scale, across fleets and federations of intelligent systems, is a challenge no single OEM or software vendor can deliver alone.
Physical AI is the strategic inflection point for AI platforms, and robotics is the hero use case. It sits at the intersection of cloud, edge, data, and agentic AI.
Physical AI Toolchain is an open-source, production-ready framework that integrates Microsoft Azure cloud services with NVIDIA's physical AI stack, accelerating robotics and physical AI developers to automate and scale data curation, augmentation, and evaluation across perception, mobility, imitation learning, and reinforcement learning pipelines. It provides:
- Accelerate physical AI innovation. From edge data capture on NVIDIA Jetson devices through cloud-based training on GPU clusters to model deployment at the edge, every stage of the physical AI lifecycle is addressed with tested, repeatable automation.
- Operationalize physical intelligence. Built on Azure Machine Learning, Azure Kubernetes Service, Azure Arc, and Azure Storage with Entra ID authentication, managed identities, and Infrastructure as Code, so workloads meet the security, compliance, and governance requirements of production environments.
- Scale through ecosystem collaboration. Native support for NVIDIA Isaac Sim and Isaac Lab for simulation and reinforcement learning, NVIDIA OSMO for workflow orchestration, and the NVIDIA Jetson platform for edge inference provides a hardware-accelerated path from research to deployment, enabled by deep partnership across the ecosystem.
- Human-robot-AI agent collaboration. Agentic engineering lets teams move from isolated machines to coordinated, instruction-driven workflows. AI agents can turn high-level instructions into executed pipelines, but they are a convenience layer, not a requirement. Start with manual workflows, introduce agents when you are ready, and customize their behavior to match your team's trust boundaries.
- Broad physical AI applicability. While robotics is the hero use case, the architecture supports any physical AI workload that follows the simulate → train → evaluate → deploy pattern, including autonomous mobile robots, robotic manipulation, industrial inspection, and embodied AI research.
Whether you are evaluating Azure and NVIDIA as a platform for physical AI, planning a proof of concept, or scaling to production, this toolchain provides a tested solution and working code to accelerate your timeline.
- Robotics researchers moving from Isaac Sim prototypes to production-grade training and deployment pipelines
- Platform engineers standardizing physical AI pipelines across teams with Infrastructure as Code and repeatable workflows
- Enterprise teams piloting Jetson + Azure deployments and need security, compliance, and scalability from day one
Note
Who it's not for (yet): This toolchain targets production and pre-production workloads. It is not designed for hobbyist projects, ROS beginners learning the basics, or general-purpose desktop demos. T0 is a first-class single-robot entry point when the goal is a production-oriented capture -> train -> validate -> run workflow.
Tip
Start on a laptop, not in the cloud. The default starting path (T0 — Dev) closes the full capture -> train -> validate -> run loop on one laptop and one robot, with zero cloud and zero Kubernetes. Cloud, Kubernetes, Arc, and fleet components are tier-gated additions you adopt only when your scale demands them, not prerequisites. See Start on a Laptop (T0 — Dev) below.
Adoption is modeled as six graduated tiers (T0-T5). Each tier states the minimum edge and cloud infrastructure required to complete the full training lifecycle: capture demonstrations on a robot, train an imitation policy, validate it, and run that policy back on the robot. Each tier is a legitimate stopping point. You adopt only the infrastructure your scale actually demands; the heavy components are opt-in additions, not a baseline you must stand up first.
T0 — Dev is the default starting path. One laptop, one robot, and the full loop:
- Capture: ROS 2 bag recording to local disk. No Arc, no ACSA, no PVC.
- Move data:
cporrsyncfrom robot to laptop. - Curate: the dataviewer in
localmode on the laptop. - Train:
lerobot-trainon the laptop, on CPU or a local GPU. - Track: training writes checkpoints and logs to local disk; hosted tracking enters at T2.
- Validate:
run-local-lerobot-eval.py/play.pylocally. - Run on robot: the inference node as a plain process or container. No Flux, no gating, no GitOps.
Edge infra: ROS 2 and Docker only. Cloud infra: none.
| Tier | When to use it | Quick start |
|---|---|---|
| T0 — Dev ⭐ | Default. One laptop, one robot; zero cloud and zero Kubernetes. | Tier 0 — Dev recipe |
| T1 — Lab | One site, a few robots, a shared GPU box. First cloud: storage. | Tier 1 — Lab recipe |
| T2 — Pilot ✅ | Recommended production. One site at scale; cloud training default. | Tier 2 — Pilot recipe |
| T3 — Production | Advanced. Single-site declarative deployment (local k3s + Flux, no Arc). | Tier 3 — Production recipe |
| T4 — Scale | Advanced. Multi-site fleet delivery; Arc as reachability broker. | Tier 4 — Scale recipe |
| T5 — Operate | Roadmap. Fleet intelligence for drift detection and retraining. | Tier 5 — Operate recipe |
⭐ default · ✅ recommended production
Cloud training, model registries, Kubernetes (k3s/AKS), Azure Arc, and the fleet delivery/intelligence planes enter the picture only at the tier where they earn their keep. Pick your tier in Getting Started → Choose Your Tier, read the tier-by-tier infrastructure boundaries in the Architecture Overview, and consult the canonical Tier Model for the authoritative tier table and vocabulary.
Note
Roadmap honesty. T5 (Operate / fleet intelligence) is on the roadmap and not yet available. The fleet-intelligence domain is currently specified, with implementation planned. Today's shipping capability spans T0-T4.
| Capability | Description |
|---|---|
| Simulation & Synthetic Data | Isaac Sim and Isaac Lab environments for RL task training and synthetic data generation |
| Edge Data Capture | ROS 2 demonstration recording on Jetson with chunking, compression, and cloud upload |
| Cloud Data Pipeline | Automated ROS-to-LeRobot conversion, quality validation, and event-driven orchestration |
| Training Infrastructure | OSMO + Azure ML integration for scalable RL and IL training with experiment tracking |
| Model Evaluation | Offline replay evaluation, Isaac Sim evaluation, and evaluation dashboards |
| Model Deployment | ONNX/TensorRT conversion, container packaging, and GitOps-based edge deployment |
| Agentic Workflows | Instruction-driven agents that orchestrate data collection, training, evaluation, and deployment end-to-end |
| Hybrid Architecture | Azure Arc, air-gapped training support, and MQTT telemetry for connected and disconnected sites |
- Infrastructure as Code: Terraform modules for reproducible Azure deployments
- Containerized Workflows: Docker-based Isaac Lab training with NVIDIA GPU support
- MLflow Integration: Automatic experiment tracking and model versioning
- Scalable Compute: Auto-scaling GPU nodes with pay-per-use cost optimization
- Enterprise Security: Entra ID integration with managed identities
- CI/CD Integration: Automated deployment pipelines with GitHub Actions
- Edge-to-Cloud Data Pipeline: Automated capture, upload, conversion, and validation
- Multi-Modal Training: Support for reinforcement learning and imitation learning workflows
- Agentic Pipeline Orchestration: Describe a task; agents handle data collection through policy deployment
./setup-dev.shThe setup script installs Python 3.12 via uv, creates a virtual environment, and installs training dependencies. This is all the default path (T0 — Dev) needs: train, track, and validate locally on a laptop GPU with no Azure subscription.
Follow the Quickstart Guide for the default local-first walkthrough, then Choose Your Tier when you are ready to graduate to cloud training (T2 — Pilot) or beyond. The cloud, Kubernetes, and fleet steps are opt-in additions layered onto this working local baseline.
Full documentation is available in the docs/ directory.
| Guide | Description |
|---|---|
| Getting Started | Prerequisites, quickstart, and first training job |
| Deployment | Infrastructure provisioning and setup |
| Training | RL and IL training workflows, MLflow, and checkpointing |
| Security | Threat model, security guide, deployment responsibilities |
| Recipes | Guides that take you from a standing start to a working result |
| Contributing | Architecture, style guides, contribution workflow |
The architecture is organized as graduated tiers (T0-T5): each component enters at the tier where it earns its keep, rather than as a baseline prerequisite. The default path (T0 — Dev) needs only the local components; everything below is an opt-in addition.
- NVIDIA Isaac Sim & Isaac Lab: Physics simulation and RL task environments
- NVIDIA Jetson: Edge inference and demonstration data capture
- Azure Storage: Cloud data and checkpoint storage (opt-in from T1 — Lab)
- Azure Machine Learning: Cloud training, experiment tracking, and model registry (default from T2 — Pilot)
- NVIDIA OSMO: Workflow orchestration and job scheduling (T2 — Pilot)
- Local k3s + FluxCD: Single-site declarative GitOps deployment, no Arc required (T3 — Production)
- Azure Arc + AKS: Cross-site reachability and identity broker for multi-site fleet delivery (T4 — Scale)
- Azure IoT Operations & Fabric: Telemetry aggregation and fleet intelligence (T5 — Operate, roadmap)
See the Architecture Overview for the per-tier infrastructure boundaries and the canonical Tier Model for the authoritative tier table.
The toolchain includes agent-driven automation that collapses multi-stage physical AI pipelines into simple, instruction-level interactions.
How it works:
- Describe the objective. Provide a natural-language instruction such as "collect 50 demonstrations of an inspection and sorting task and train an IL policy."
- Agent plans and executes. The agent decomposes the objective into pipeline stages: data collection, conversion, training configuration, compute provisioning, and training launch. It then executes each stage using the toolchain's APIs and infrastructure.
- Evaluate and iterate. The agent runs evaluation (simulation replay, success-rate metrics) and presents results. If the policy does not meet acceptance criteria, the agent adjusts hyperparameters or collects additional data and re-trains.
- Deploy. Once a policy passes evaluation, the agent packages it (ONNX/TensorRT), builds a container image, and triggers GitOps deployment to target edge devices.
What agents can do today:
| Capability | Description |
|---|---|
| Sample data collection | Configure Isaac Sim scenes and collect synthetic demonstration datasets |
| RL pipeline execution | Set up Isaac Lab tasks, launch OSMO training jobs, and track experiments in MLflow |
| IL pipeline execution | Convert demonstration data to LeRobot format, run imitation learning training |
| Policy evaluation | Execute offline replay and simulation-based evaluation against success criteria |
| Deployment promotion | Convert checkpoints, package containers, and push to edge via GitOps |
Agents operate within the same security boundaries, managed identities, and RBAC controls as manual workflows. All agent actions are logged and auditable.
| Question | Answer |
|---|---|
| Are agents required? | No. Every pipeline stage has a manual CLI and API path. Agents are opt-in. |
| Can I use agents for some stages but not others? | Yes. Agents are composable: use them for data collection but run training manually, or vice versa. |
| Are agents opinionated or customizable? | Customizable. Agent behavior is driven by configuration files you control: which stages to automate, compute budgets, approval gates, and evaluation thresholds. |
| What happens if an agent makes a mistake? | Agents request human approval before destructive actions (deploying to production, deleting data). All intermediate artifacts are versioned and recoverable. |
| How are agent actions audited? | Every agent action is logged with the initiating instruction, parameters, and outcome. Logs integrate with Azure Monitor and MLflow. |
| Directory | Purpose |
|---|---|
src/ |
Core Python modules: conversion, validation, training utilities |
infra/ |
Terraform and Bicep templates for Azure resource provisioning |
config/ |
YAML configuration schemas for recording, training, and deployment |
scripts/ |
Setup, benchmarking, and operational helper scripts |
tests/ |
Unit, integration, and end-to-end test suites |
docs/ |
All project documentation |
Prerequisites:
- Python 3.12+
- Docker with NVIDIA Container Toolkit
- Terraform 1.5+ (for infrastructure deployment)
- Azure CLI with an active subscription
- NVIDIA GPU (local development) or Azure GPU VM
Run the test suite (the four component suites mirror the CI split):
# Run every component at once (uses testpaths from pyproject.toml)
uv run pytest
# Or run a single component
uv run pytest training/tests -v
uv run pytest data-management/tools/tests -v
uv run pytest data-pipeline/capture/tests -v
uv run pytest fleet-deployment/inference/tests -vSee prerequisites for the complete setup guide.
Contributions are welcome. Whether fixing documentation or adding new training tasks:
- Read the Contributing Guide
- Review open issues
- See the prerequisites for required tools
All release tags are signed. Verify a release tag before using it in production workflows:
git fetch --tags
git tag -v v1.0.0Note
git tag -v confirms cryptographic integrity and the Rekor entry but does not validate the signer identity. CI gates each v* tag with constrained gitsign verify-tag, binding the signature to the pinned workflow identity. For identity-constrained local verification, run gitsign verify-tag --certificate-identity 'https://github.com/microsoft/physical-ai-toolchain/.github/workflows/main.yml@refs/heads/main' --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' v1.0.0.
This repository uses Sigstore gitsign keyless signing for release tags. For tag signing policy and maintainer guidance, see CONTRIBUTING.md.
See the project roadmap for priorities, timelines, and success metrics.
This toolchain builds upon:
- NVIDIA Isaac Lab: RL task framework
- NVIDIA Isaac Sim: physics simulation
- NVIDIA OSMO: workflow orchestration
- LeRobot: imitation learning dataset format
- Built with HVE Core
Microsoft encourages customers to review its Responsible AI Standard when developing AI-enabled systems to ensure ethical, safe, and inclusive AI practices. Learn more at Microsoft's Responsible AI.
No interfaces are currently deprecated. When deprecations are announced, they appear here with migration guidance and removal timelines.
See the Deprecation Policy for how interface changes are communicated and managed.
This project is licensed under the MIT License.
See SECURITY.md for the security policy and vulnerability reporting.
See GOVERNANCE.md for the project governance model.
See SUPPORT.md for support options and issue reporting.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.
🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.