Dexterity Interface for RL

Quick Start

Driver Container (On driver computer)

docker build -f docker/Dockerfile.driver -t dex-driver .

cd <path to>/dexterity-interface-rl
docker run --name handrl-driver --rm -it --privileged -v $(pwd):/workspace --net=host dex-driver

ROS2 Build

cd /workspace/libs/robot_motion_interface/ros
colcon build --symlink-install
source install/setup.bash

ROS2 Run Driver Node

export ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST
export ROS_STATIC_PEERS='192.168.4.9;remote.com'
ros2 run robot_motion_interface_ros rl_driver

If it says libfranka: Move command rejected: command not possible in the current mode ("User stopped"), please check the Franka Control Panel for the robot status and check the stop button.

Open Another Bash and Enter the Running Container

docker exec -it handrl-driver bash

Enter and Source ROS2 Workspace

cd /workspace/libs/robot_motion_interface/ros
source install/setup.bash

ROS2 Run Test Node

ros2 run robot_motion_interface_ros test_pre_grasp

Policy Container (On inference computer)

Build

docker build -f docker/Dockerfile.policy_py312 -t dex-policy .

cd <path to>/dexterity-interface-rl

xhost +local:docker
docker run --name handrl-policy --rm -it --privileged --gpus all -v $(pwd):/workspace --device /dev/bus/usb:/dev/bus/usb --net=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix dex-policy

Dependencies

cd /workspace/dep
pip install -r requirements.txt
cd /workspace

Compile curobo

cd /workspace/dep
git clone https://github.com/AlfredMoore/curobo-HAND.git curobo
cd curobo
pip install -e . --no-build-isolation
python3 -m pytest .
cd /workspace

Prompt depth anything

cd /workspace/dep
git clone https://github.com/DepthAnything/PromptDA.git # 9161547
# This repo is not ready for `pip install -e .`
export PYTHONPATH=/workspace/dep/PromptDA:$PYTHONPATH
cd /workspace

Segment Anything 2

cd /workspace/dep
git clone https://github.com/facebookresearch/sam2.git  # 2b90b9f
cd sam2
pip install -e .
cd /workspace

Efficient SAM 3 (including Segment Anything 3)

cd /workspace/dep
git clone https://github.com/SimonZeng7108/efficientsam3.git -b sam3_litetext   # bef17f5
cd efficientsam3
pip install --force-reinstall --no-deps --no-build-isolation -e .
cd /workspace

Import test

python /workspace/init/test_env.py

Realsense Test

python -m robot_motion_interface.utils.realsense_test.py

Commit the image from the host (optional)

docker commit handrl-policy dex-policy:compiled
# Then you can directly run the full installed container by
xhost +local:docker
docker run --name handrl-policy --rm -it --privileged --gpus all -v $(pwd):/workspace --device /dev/bus/usb:/dev/bus/usb --net=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix dex-policy:compiled

Run Pre-grasp Sampling Modules

cd /workspace
# pinocchio and curobo test
python -m robot_motion_interface.utils.kinematics
# pre-grasp SE3 poses sampling
python -m robot_motion_interface.utils.pose_sampler
# home to a pre-grasp pose traj generation (this takes a long time to filter 50k joint poses)
python -m robot_motion_interface.utils.traj_sampler

ROS2 Build

cd /workspace/libs/robot_motion_interface/ros
colcon build --symlink-install
source install/setup.bash

Test Nodes

# cv node
ros2 run robot_motion_interface_ros cv_node

Curobo TrajOpt Nodes

export ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST
export ROS_STATIC_PEERS='192.168.4.4;remote.com'
ros2 run robot_motion_interface_ros test_curobo

Retargeting Nodes

export ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST
export ROS_STATIC_PEERS='192.168.4.4;remote.com'
ros2 run robot_motion_interface_ros test_retarget_traj_player --ros-args \
  -p traj_path:=models/egodex/traj-retarging/screw_unscrew_bottle_cap/0_curobo_2stage-interpolated.npz

Attach to Policy Container

docker exec -it -e DISPLAY=$DISPLAY handrl-policy bash

Dexterity Interface

Requirements

• For the simulation/interface you will need:
  • Ubuntu Machine with EITHER:
    • Docker Engine. TODO: Add functionality. OR
    • Ubuntu 22.04 or 24.04. TODO: Revise if certain version of ROS.
• You can also have the following hardware requirements:

  • Franka Emika Panda 7 DOF Robot setup with the FCI.

    • Robot system version: 4.2.X (FER pandas)
    • Robot / Gripper Server version: 5 / 3
    • The Realtime Kernel Patch Kernel Patch on Ubuntu Machine.

  • Axio80-M20 Force Torque Sensor installed on the Panda's End Effector.

  • Tesollo 3 Finger Gripper TODO

  • Cameras TODO

If you are running on the robot, be sure to set the proper static IPs and wiring that you need as shown below:

flowchart LR

    %% Panda Controller
    subgraph PC["**Panda Controller**"]
        PC_CONN["Round Pin Connector"]:::power_data
    end

    %% Panda Arm
    subgraph PA["**Panda Robot**"]
        PA_IP["IP: 192.168.1.3 <br> Netmask: 255.255.255.0"]:::ethernet
        PA_CONN["Round Pin Connector"]:::power_data
    end



    %% Ubuntu Machine
    subgraph MA["**Ubuntu Machine**"]
        MA_DESC["Requires Real-Time Kernel Patch if running robot."]:::description
        MA_IP1["IP: 192.168.1.5 <br> Netmask: 255.255.255.0"]:::ethernet
        MA_IP2["IP: 192.168.2.5 <br> Netmask: 255.255.255.0"]:::ethernet
        MA_IP3["IP: 192.168.3.2 <br> Netmask: 255.255.255.0"]:::ethernet
    end



    %% Wall Outlets
    PC_PWR["Wall Outlet"]:::power_data

    %% Connections
    PC_PWR --- PC_CONN --- PA_CONN --- PA
    

    PA_IP --- MA_IP1

    %% Styles
    classDef description fill:none,stroke:none,color:#000;
    classDef ethernet fill:#fff3b0,stroke:#000,color:#000;
    classDef power_data fill:#f5b7b1,stroke:#000,color:#000;

Loading

Python Setup

1. Install Ubuntu dependencies:

   sudo apt update
   sudo apt install libeigen3-dev python3.11 python3.11-venv

2. Create and source python virtual environment

   python3.11 -m venv venv-dex
   source venv-dex/bin/activate

3. Install IsaacSim and IsaacLab.

   pip install --upgrade pip
   pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128
   pip install isaaclab[isaacsim,all]==2.2.0 --extra-index-url https://pypi.nvidia.com

   Test that Isaacsim installed correctly. The first time this is run, make sure to reply YES to the EULA prompt. Also the first time it may take a while to start as it sets up the necessary dependencies.

   isaacsim
   

4. Install our Python packages:

   pip install -e libs/robot_motion
   pip install -e libs/robot_motion_interface
   pip install -e libs/isaacsim_ui_interface/
   pip install -e libs/sensor_interface/sensor_interface_py
   
   pip install -e libs/primitives/primitives_py
   
   pip install -e libs/planning/planning_py

   TODO: Edit the rest of these so they are not so deep

Python Running

```bash
python3 -m robot_motion.ik.ranged_ik

python3 -m robot_motion_interface.isaacsim.isaacsim_interface
python3 -m robot_motion_interface.tesollo.tesollo_interface
python3 -m robot_motion_interface.panda.panda_interface
python3 -m robot_motion_interface.bimanual_interface

python3 -m isaacsim_ui_interface.isaacsim_ui_interface

python3 -m sensor_interface.camera.kinect_interface
python3 -m sensor_interface.camera.realsense_interface

python3 -m primitives.primitive

python3 -m planning.llm.gpt
python3 -m planning.perception.yolo_perception
```

TODO: C++ setup/running

[ALTERNATIVE SETUP] Docker Setup

sudo docker build -t dex-interface .
sudo docker run --rm -it --privileged  -v $(pwd)/libs:/workspace/libs -v $(pwd)/app:/workspace/app --net=host dex-interface

Note: --privileged is not the safest, but it is an easy way to give real-time privileges to the container. TODO: Look into safer way.

System Architecture

flowchart TD

%% --- UI (top) ---
subgraph L1["UI"]
  UIF["UI Frontend (Javascript, HTML, Tailwind CSS)"]
  UIB["UI Backend (Python, ROS Python, Fast API framework)"]
  UIF --- UIB
end

%% --- High-level logic / Primitives ---
subgraph L2["High-level Logic & Primitives"]
  PRIM["Primitives (Python): low and mid-level"]
  ROS_PRIM["ROS Primitive Wrapper (ROS Python)"]
  PLAN["Planning (Python): Breakdown task into primitives"]
  LLM["LLM (Python): task breakdown/chat "]
  PERC["Perception (Python): object localization, manipulation points"]
end

%% --- Interfaces ---
subgraph L3["Hardware/Software Interfaces"]
  ISAACI["IsaacSim UI Interface (Python): add/remove objects, click/drag, extends Isaacsim Robot Motion interface"]
  ROS_RMI["ROS Motion Interface Wrapper (ROS Python)"]
  RMI["Robot Motion Interface (Python, C++): Panda, Tesollo, IsaacSim"]

  SENSI["Sensor Interface (Python, C++): force torque sensor"]
  ROS_SENSI["ROS Sensor Interface Wrapper (Python, C++): cameras, force torque sensor"]
end

%% --- Low level (bottom) ---
subgraph L4["Low-level Control & Models"]
  CTRLS["Controllers (C++): cartesian torque, joint torque, joint pos/vel"]
  IK["IK (Python): Drake or RelaxedIK"]
  RPROPS["Robot Properties (C++): friction, coriolis"]
end

%% --- Wiring (top → bottom) ---
UIB --- ISAACI
UIB --- PLAN
UIB -.- PERC



PLAN --- ROS_PRIM
ROS_PRIM --- PRIM
PLAN --- LLM
PLAN --- PERC
PRIM --- ROS_RMI


RMI --- ROS_RMI
PERC --- SENSI
SENSI --- ROS_SENSI


RMI --- CTRLS
RMI --- IK
CTRLS --- RPROPS

Loading

Mya notes:

• Isaac Sim 5.0 requires Python3.11, Ubuntu 22.04 or 24.04
• Ubuntu 22.04 is most compatible with ROS 2 Humble which by default uses Python3.10. So would need to compile from source for Python 3.11: https://github.com/isaac-sim/IsaacSim-ros_workspaces/blob/main/build_ros.sh
• Could instead update everything to Ubuntu 24.04 and use Jazzy.

Todo:

• Figure out which packages are run on what computers.
• Figure out blocking vs non-blocking movement execution
• Allow partial setpoint updates.

https://docs.omniverse.nvidia.com/extensions/latest/ext_livestream/webrtc.html https://isaac-sim.github.io/IsaacLab/main/source/api/lab/isaaclab.app.html

https://github.com/NVIDIA-Omniverse/web-viewer-sample

http://127.0.0.1:8211/streaming/webrtc-client?server=127.0.0.1 http://192.168.1.209:8211/streaming/webrtc-client?server=192.168.1.209