Unsuccessful execution of trajectories by MoveIt

[naa1824]

Thank you for this amazing repository!

I tried running ex_train.bash file using reach-v0 and grasp-v0 environments, and I am getting this error repeating in the log terminal:

[train.py-9] [INFO] [1738223111.533212688] [joint_trajectory_controller]: Accepted new action goal
[train.py-9] [INFO] [1738223111.558696587] [gripper_trajectory_controller]: Goal reached, success!
[train.py-9] [INFO] [1738223111.695858269] [gripper_trajectory_controller]: Received new action goal
[train.py-9] [INFO] [1738223111.695928880] [gripper_trajectory_controller]: Accepted new action goal
[train.py-9] [ERROR] [1738223111.699003631] [drl_grasping_0]: Action 'joint_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[train.py-9] [ERROR] [1738223111.700511261] [drl_grasping_0]: Action 'gripper_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[train.py-9] [ERROR] [1738223111.701031727] [drl_grasping_0]: Action 'gripper_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[train.py-9] [INFO] [1738223111.725962534] [joint_trajectory_controller]: Received new action goal
[train.py-9] [INFO] [1738223111.726004452] [joint_trajectory_controller]: Accepted new action goal
[train.py-9] [ERROR] [1738223111.877747029] [drl_grasping_0]: Action 'joint_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[train.py-9] [ERROR] [1738223111.878758634] [drl_grasping_0]: Action 'joint_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[train.py-9] [ERROR] [1738223111.882890962] [drl_grasping_0]: Action 'joint_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[train.py-9] [INFO] [1738223111.896224860] [joint_trajectory_controller]: Received new action goal
[train.py-9] [INFO] [1738223111.896261447] [joint_trajectory_controller]: Accepted new action goal
[train.py-9] [INFO] [1738223111.920762726] [gripper_trajectory_controller]: Goal reached, success!
[train.py-9] [ERROR] [1738223112.066170263] [drl_grasping_0]: Action 'joint_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[train.py-9] [ERROR] [1738223112.069397011] [drl_grasping_0]: Action 'joint_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.

[evaluate.py-10] [INFO] [1738217726.937639924] [joint_trajectory_controller]: Received new action goal
[evaluate.py-10] [INFO] [1738217726.937689261] [joint_trajectory_controller]: Accepted new action goal
[evaluate.py-10] [INFO] [1738217726.955191282] [drl_grasping_0]: Objects overlapping, trying new positions
[evaluate.py-10] [INFO] [1738217726.956227882] [joint_trajectory_controller]: Goal reached, success!
[evaluate.py-10] [INFO] [1738217726.969279973] [drl_grasping_0]: Objects overlapping, trying new positions
[evaluate.py-10] [WARN] [1738217727.005667142] [drl_grasping_0]: Command failed because MoveIt Servo is not yet enabled.
[evaluate.py-10] [WARN] [1738217727.016479687] [drl_grasping_0]: Calling '/servo_node/start_servo' service to enable MoveIt Servo...
[evaluate.py-10] [WARN] [1738217727.054877874] [drl_grasping_0]: Command failed because MoveIt Servo is not yet enabled.

The error keeps repeating every 4-5 lines,
Is that normal?

[AndrejOrsula]

Hello @naa1824,

Thank you for opening this issue! Could you please verify that the robot moves properly inside the simulation (smooth-ish motion in the Cartesian space)? This error is expected if the RL policy outputs an action into an invalid configuration that the robot cannot execute. The action simply won't be performed. An alternative would be to use MoveIt Servo (as indicated in the second log), which uses Differential IK that will always result in some motion (although the motion might not match the desired action).

FYI, my efforts are now focused on the Space Robotics Bench that includes many more environments, tasks and workflows (the framework is based on Isaac Sim instead of Gazebo). Therefore, I am not planning to update this repository anymore.

[naa1824]

Hi @AndrejOrsula. Thanks so much for your quick response.

To be on the same page: I used the latest docker image "2.0"

I ran the ex_training.bash: grasp-gazebo-v0, panda, sac.

During the training, I could not see any visualization. rivz and gazebo didn't work.

I stopped the training after reaching 20k time step.

Then I ran ex_evaluation.bash, and I made sure that it has the same configurations as the training

During evaluation I see the robot moving toward the object, then it git stuck and it resets the environment:
and the terminal print the same error:

[evaluate.py-10] [INFO] [1738238749.967540587] [joint_trajectory_controller]: Received new action goal
[evaluate.py-10] [INFO] [1738238749.967607306] [joint_trajectory_controller]: Accepted new action goal
[evaluate.py-10] [INFO] [1738238750.232434008] [joint_trajectory_controller]: Goal reached, success!
[evaluate.py-10] [INFO] [1738238750.327865746] [gripper_trajectory_controller]: Received new action goal
[evaluate.py-10] [INFO] [1738238750.327907053] [gripper_trajectory_controller]: Accepted new action goal
[evaluate.py-10] [ERROR] [1738238750.349975672] [drl_grasping_0]: Action 'gripper_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[evaluate.py-10] [ERROR] [1738238750.351366162] [drl_grasping_0]: Action 'gripper_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[evaluate.py-10] [INFO] [1738238750.361793015] [joint_trajectory_controller]: Received new action goal
[evaluate.py-10] [INFO] [1738238750.361841343] [joint_trajectory_controller]: Accepted new action goal
[evaluate.py-10] [INFO] [1738239401.571743345] [joint_trajectory_controller]: Goal reached, success!
[evaluate.py-10] [INFO] [1738239401.714943379] [joint_trajectory_controller]: Received new action goal
[evaluate.py-10] [INFO] [1738239401.714986878] [joint_trajectory_controller]: Accepted new action goal
[evaluate.py-10] [INFO] [1738239401.818496537] [gripper_trajectory_controller]: Goal reached, success!
[evaluate.py-10] [INFO] [1738239402.039528281] [joint_trajectory_controller]: Goal reached, success!
[evaluate.py-10] [ERROR] [1738239402.152816983] [drl_grasping_0]: Action 'gripper_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[evaluate.py-10] [ERROR] [1738239402.154643356] [drl_grasping_0]: Action 'gripper_trajectory_controller/follow_joint_trajectory' was unsuccessful: 5.
[evaluate.py-10] [INFO] [1738239402.162054072] [joint_trajectory_controller]: Received new action goal
[evaluate.py-10] [INFO] [1738239402.162103514] [joint_trajectory_controller]: Accepted new action goal
[evaluate.py-10] [INFO] [1738239402.492588146] [joint_trajectory_controller]: Goal reached, success!

I know you are busy with other projects, but really this Repo is really master piece

I want to use it to learn how to train the panda to pick and place single ball using RL in simulation. so I want to use this repo as my starting point in this journey

do you know which part of this repo that work 100% using the panda robot?

[naa1824]

I have another question which I posted in the discussion part related to Rivz and RL:

What I understand is, in manipulation applications, Moveit can create the path planning and Motion planning to generate the motion.
But another approach to do so is to use machine learning: Reinforcement Learning to replace Moveit. so the robot will generate the motion based on the trained policy.

How does this Repo merging both together?

Can you teach me what is the role of each one in this project?

[AndrejOrsula]

During evaluation I see the robot moving toward the object, then it git stuck and it resets the environment

In that case, I would say that everything works as expected. The only concern with those errors would be if the robot did not move at all.

terminal print the same error

Yes, it happens when MoveIt cannot find a plan. But that's fine because the agent does not always provide feasible actions.

I stopped the training after reaching 20k time step.

It might require longer training that 20k time steps. Sadly, it is a slow process and this project is not parallelised (only a single environment worker).

I want to use it to learn how to train the panda to pick and place single ball using RL in simulation. so I want to use this repo as my starting point in this journey

That sounds great! As mentioned above, this repository is sadly not very performant when it comes to the speed of convergence due to simulating with a single worker. Therefore, don't be afraid to try alternatives if you find that it limits your work.

I have another question which I posted in the discussion part related to Rivz and RL:

I answered in the Discussion#116.

[naa1824]

Thank you so much,

Wish you all the best with your projects.

[naa1824]

Hello @AndrejOrsula,
I hope you are doing well.

I have trained the model for 6 days, it finished 1 million time steps.

the success rate is zero, but I can see some grasping reward at some stages.

but going to the evaluation, it act as the last time, no improvement.

the robot start moving closer to the object then it git stuck and it starts going up and down. through my observation, I think the gripper (last motor) is trying to rotate 360 degrees which exceed the limit of the motor.

Is there something that I can do? what files of the project that I have to modify to get better output?

Could you give more details about this solution that you suggested earlier?

An alternative would be to use MoveIt Servo (as indicated in the second log), which uses Differential IK that will always result in some motion (although the motion might not match the desired action).

How can I do it?

[naa1824]

Another quick question,

which part of the code that control the order of the grasping when there are multiple objects?

[AndrejOrsula]

What algorithm are you using for training the agent? It looks like SAC from the logs, but I'm not certain.

It is surprising, but you could try a different algorithm from SB3 (e.g. TQC) to see if anything changes.

drl_grasping/examples/ex_train.bash

Lines 41 to 44 in b228354

	## Selection of RL algorithm
	ALGO="sac"
	# ALGO="td3"
	# ALGO="tqc"

There is also Dreamer integration, but I have not tested this one much.

Could you give more details about this solution that you suggested earlier?

An alternative would be to use MoveIt Servo (as indicated in the second log), which uses Differential IK that will always result in some motion (although the motion might not match the desired action).

How can I do it?

You can modify the config here to enable MoveIt Servo for a specific task (but I am not sure if it will lead to any improvements):

drl_grasping/drl_grasping/envs/__init__.py

Line 330 in b228354

# "use_servo": True,

which part of the code that control the order of the grasping when there are multiple objects?

None, the agent gets reward for grasping ANY object. If you want to grasp a specific object, you would need to embed this information into observations and modify the reward computation accordingly.

[naa1824]

Thank you so much