🦾 Dum-E | The Embodied AI Agent

What if your favorite AI agents have hands?

🚀 Get Started • 📖 Documentation • 🗺️ Roadmap • 🤝 Contributing

---

🎯 Mission

Inspired by Tony Stark's robotic assistant Dum-E, the mission of this project is to create an intelligent, voice & vision enabled AI agent with robotic arm(s) capable of real-time human interaction, physical operations, and orchestration of a wide range of tools and services.

✨ Key Features

• 🎤 Real-time Voice Interface - Natural voice conversation with configurable voice and multi-language support
• 🧠 Long-horizon Tasks Planning - Orchestrated by state-of-the-art VLMs with multi-modal reasoning and tool use
• 📡 Asynchronous Tasks Execution - Support for asynchronous task execution with streaming updates
• 👁️ Hybrid Robot Control - Deep learning policies for generalizable physical operations combined with classical control for precise manipulation
• 🔧 Modular Architecture - Flexible interfaces to add your own custom backends, embodiments and tools
• 🌐 MCP Support - Agents and tools available via MCP for custom client integration

📺 Demo

🔊 Watch with sound to hear voice interactions

Dum-E.Demo.mp4

🚀 Get Started

This project supports two deployment patterns:

1. Single Workstation: Run everything locally on a single machine with GPU
2. Client-Server: Run policy server on a separate GPU server while keeping lightweight client components local, useful when your local machine doesn't meet the GPU requirements

Choose the setup that best matches your needs and hardware availability. The following sections will guide you through the installation process.

⚙️ System Requirements

Common

• Robot: SO100/101 robotic arm assembled with wrist camera
• Webcam: 640p+ USB webcam for additional vision input

For Single Workstation

• GPU: NVIDIA Ampere or later with at least 12GB VRAM (tested on RTX 3060) and driver ≥ 535.0
• OS: Ubuntu 22.04/24.04 LTS or Windows with WSL2

For Client-Server

• Server: Any machine with 12GB+ VRAM NVIDIA GPU, e.g. EC2 g4dn/g5/g6
• Client: Any machine with 4GB+ RAM
• OS: Any OS (MacOS/Windows/Linux)

🔧 Installation

📦 On Single Workstation or Server

Requires 1) NVIDIA GPU 2) Linux or WSL2 3) Docker with the NVIDIA Container Runtime

The GR00T N1.7 policy server runs as a reproducible Docker container (x86 + NVIDIA GPU, or Jetson Orin). The container packages the server's CUDA / torch / flash-attn stack and serves inference on ZMQ port 5555; the checkpoint and Hugging Face cache are mounted at runtime, never baked into the image.

1. Clone the Isaac-GR00T repository (the build wrapper checks it out at a pinned commit):

   git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T

2. Pre-fetch the checkpoint and the gated nvidia/Cosmos-Reason2-2B backbone into the host Hugging Face cache. This is mandatory — the server loads the gated backbone at startup and exits with GatedRepoError if it is not already cached. Request access to nvidia/Cosmos-Reason2-2B on its Hugging Face page first, then:

   export HF_TOKEN=<your-hf-token>
   # SO101 fruit-picking checkpoint (mounted :ro at runtime)
   uv run hf download aaronsu11/GR00T-N1.7-3B-SO101-FruitPicking \
       --local-dir ./checkpoints/GR00T-N1.7-3B-SO101 --exclude "optimizer.pt"
   # Gated backbone (read from the mounted HF cache at runtime)
   uv run hf download nvidia/Cosmos-Reason2-2B

3. Build the image at the pinned commit. The wrapper delegates to the upstream docker/build.sh, failing closed if the checkout does not match the pin. Set GR00T_REPO if your clone is not at /home/<user>/Projects/Isaac-GR00T.

   # x86 + NVIDIA GPU -> image tag: gr00t
   bash scripts/build_gr00t_image.sh
   
   # Jetson Orin (aarch64), run natively on the Orin -> image tag: gr00t-orin
   bash scripts/build_gr00t_image.sh orin

4. Start the policy server. The checkpoint and HF cache are mounted :ro; HF_TOKEN is passed at runtime via -e (never baked into the image).

   docker rm -f gr00t-server 2>/dev/null || true
   docker run -d \
       --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
       -p 5555:5555 \
       -v "$(pwd)/checkpoints/GR00T-N1.7-3B-SO101:/checkpoints/model:ro" \
       -v ~/.cache/huggingface:/root/.cache/huggingface:ro \
       -e HF_TOKEN="$HF_TOKEN" \
       --name gr00t-server \
       gr00t \
       uv run python gr00t/eval/run_gr00t_server.py \
           --model-path /checkpoints/model \
           --embodiment-tag new_embodiment --host 0.0.0.0 --port 5555

   The server is ready once port 5555 is listening and the model has finished loading (GPU memory settles). Keep the container running while you use the policy for inference. Note the server's IP (<policy_host>) and make sure port 5555 is reachable from the client.

Note

Security: the server has no auth by default. Publish/bind :5555 to localhost or a trusted subnet only — do not expose it to untrusted networks. Always build via scripts/build_gr00t_image.sh (SHA-pinned), never an ad-hoc docker build, so the image provenance stays locked to the verified upstream commit.

On Single Workstation or Client

1. Clone the Repository

   git clone https://github.com/aaronsu11/Dum-E.git

2. Install uv (if not already installed):

   curl -LsSf https://astral.sh/uv/install.sh | sh

3. Install Dum-E Dependencies

   cd Dum-E
   uv sync
   source .venv/bin/activate

Note

If you have never set up SO-ARM before:

• Find the wrist_cam_idx and front_cam_idx by running lerobot-find-cameras
• Find the robot_port of by running lerobot-find-port
• Calibrate the robot following the instructions for SO-100 or SO-101 and note down your robot_id. For example with SO-101, run: lerobot-calibrate --robot.type=so101_follower --robot.port=<robot_port> --robot.id=<robot_id>

4. Configure Dum-E

   cp config.example.yaml my-dum-e.yaml

   Edit my-dum-e.yaml:

   • Set controller.robot_type/robot_port/robot_id/wrist_cam_idx/front_cam_idx
   • Set controller.policy_host to your gr00t policy server IP (or localhost)
   • Optionally set agent.profile and voice.mode/profile to use different model presets

5. Test policy execution

   # Uses controller.* from YAML
   python -m embodiment.so_arm10x.controller --config my-dum-e.yaml --instruction "<your-instruction>"

Note

If the robot is not moving, check if the gr00t policy server is running and the port is accessible from the client by running

• nc -zv <policy_host> 5555 (on MacOS/Linux) or
• Test-NetConnection -ComputerName <policy_host> -Port 5555 (on Windows PowerShell).

6. Environment configuration for agent and voice

   Sign up for free accounts at ElevenLabs, Deepgram, Anthropic and obtain the API keys if you are using the default profile, or get your AWS credentials if you are using the aws profile.

   Copy the environment template and update the .env file with your credentials:

   cp .env.example .env

   Edit .env with your credentials (choose one of the following):

   • For default profile:
     • Anthropic API key for LLM
     • ElevenLabs API key for TTS
     • Deepgram API key for STT
   • For aws profile:
     • AWS Access Key ID
     • AWS Secret Access Key
     • AWS Region

Note

You can optionally configure Langfuse observability for agent and voice by setting LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST in the .env file.

7. Test the robot agent

   # One-shot instruction (agent inherits controller.* unless overridden in agent section)
   python -m embodiment.so_arm10x.agent --config my-dum-e.yaml --instruction "<your-instruction>"

8. Start Dum-E

   Finally, start the full stack with the voice interface, MCP server and robot agent:

   python dum_e.py --config my-dum-e.yaml

   This will launch the voice interface at http://localhost:7860 where you can connect and speak to Dum-E using your microphone. Have fun!

   You can also start only the voice interface and MCP servers. Useful for testing the servers independently without the robot hardware:

   python dum_e.py --node servers --config my-dum-e.yaml

🏗️ Architecture Overview

Core Components

graph TB
 subgraph subGraph0["Frontend"]
        A["Voice Interface<br>Pipecat"]
        B["External MCP Clients"]
  end
 subgraph subGraph1["Backend"]
        C["MCP Server<br>Streamable HTTP"]
        D["Fleet Manager<br>Multi-agent Management"]
        E["Task Manager<br>Task Management"]
        F["Message Broker<br>Event Streaming"]
  end
 subgraph subGraph2["Agent Layer"]
        G["Robot Agent<br>Multi-modal Orchestration"]
  end
 subgraph subGraph3["Controller Layer"]
        H["Classical Control<br>MoveIt"]
        I["VLA Policy<br>Gr00t Inference"]
        J["Custom Tools<br>Public MCP Servers"]
  end
 subgraph subGraph4["Hardware Layer"]
        K["Physical Robot<br>SO-ARM10x"]
  end
    A --> C
    B --> C
    C --> D & E & F & G
    G --> H & I & J
    H --> K
    I --> K

Loading

🔄 Data Flow

1. Voice Interaction: Voice Streams → Cascaded / Speech-to-Speech Processor → Conversation and Task Delegation
2. Task Execution: Task Manager → Robot Agent → VLM Reasoning → Robot Controller Tools → Robot Hardware
3. Robot Control: Task Instruction + Camera Images → DL Policy / Classical Control → Joint Commands
4. Streaming Feedback: Agent Streaming Responses → Message Broker → MCP Context → Voice Updates

🗺️ Roadmap

Q3 2025

• Voice Interaction

  • Multi-language support (Mandarin, Japanese, Spanish etc.)
  • Emotional understanding with speech-to-speech models

• MCP Servers

  • Access to robot agent via MCP
  • Configurable MCP server endpoints for Dum-E

• Local Model Support

  • Integration with Ollama for local language model inference

• Upgrade Dependency

  • LeRobot new hardware APIs

Q4 2025

• Cross-Platform Support

  • Docker containers for platform-agnostic deployment (x86 + Jetson Orin)

• ROS2 Integration

  • Native ROS2 node implementation
  • Integration with existing ROS toolkits

🔬 Ongoing: Research Initiatives

• Embodied AI Research

  • Generalizable & scalable policies for physical tasks
  • Efficient dataset collection and training

• Human-Robot Interaction

  • Natural multi-modal understanding
  • Contextual conversation memory
  • Self-evolving personality and skillset

🤝 Contributing

We welcome contributions from the robotics and AI community! Here's how you can help:

🌟 Ways to Contribute

• 🐛 Bug Reports: Found an issue? Create a detailed bug report
• 💡 Feature Requests: Have ideas? Share them in our discussions or Discord
• 📝 Documentation: Help improve our docs and tutorials
• 🧪 Testing: Add test cases and improve coverage
• 🚀 Code: Submit pull requests with new features or fixes

📋 Development Guidelines

1. Fork the Repository

   git fork https://github.com/aaronsu11/Dum-E.git
   cd Dum-E
   git checkout -b feature/issue-<number>/<your-feature-name>

2. Follow Code Standards

   • Use Python 3.12 type hints
   • Follow PEP 8 style guidelines
   • Add comprehensive docstrings
   • Maintain test coverage > 50%

3. Testing Requirements

   # Run tests before submitting
   python -m pytest tests/
   python -m black <your-file-or-directory>
   python -m isort <your-file-or-directory>

4. Pull Request Process

   • Create detailed PR description
   • Link related issues
   • Ensure CI/CD passes
   • Request review from maintainers

👥 Community

• Discord: Join our community
• GitHub Discussions: Share ideas and get help

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

This project builds on top of the following open-source projects:

• Pipecat Framework
• FastMCP
• Strands Agents
• Isaac GR00T
• LeRobot

---

⭐ Star us on GitHub — it motivates us a lot!

🚀 Get Started • 📖 Documentation • 🤝 Join Community • 💼 Commercial Use

Built with ❤️ for the future of robotics