Hyperparameters for biped walk

[apirrone]

Hello !

I'm trying to use AMP_for_hardware to learn a nice walk on a small bipedal robot. I have produced motion example data in the same format as you used here.

I tried running trainings with pretty much the same hyperparameters as the ones in your example a1_amp_config.py. I had to tweak some things to match the size and weight of my own robot, but not a lot.

To validate that there is no bug in my adaptations, I first train with only a walking forward example motion and only positive x linear velocity command and no noise/randomization. The robot progressively learns to move forward a little bit, but by shaking its feet and making small jumps, not imitating the walking example motion. Here is an example (after 1000 steps of 8000 envs)

amp_for_hardware-2024-08-06_15.53.49.mp4

And here is what the reference motion looks like (using your replay_amp_data.py script)

amp_for_hardware-2024-08-06_15.55.33.mp4

The training curves look like this :

I am able to run your a1_amp environment with the provided checkpoint and it runs great, it's very fun ton control it with a joystick :)

My questions are :

• Did you use the parameters that are in a1_amp_config.py to train the provided policy ? Meaning only the velocity tracking rewards and amp_reward_coef = 2.0 ?
• Do you think the behavior I get is a symptom of a bug or bad parameters ?
• Do I just need to train for much longer ?
  • I tried letting the training run over night, I did not get much better results

Thank you very much !

[WoohyunCha]

I have been training a bipedal robot as well.
Although my case is a bit different; setting the velocity tracking reward scales to (1.0, 0.5) instead of the 1.0/(0.005*6) or so, leads to the robot following the reference motion too much. No matter what command I give it would move in the same speed.
Increasing the velocity tracking reward scale or the task_lerp does help better tracking of command, but it leads to instability of the robot such as shaking or simply tipping over.
Someone else in my lab has succeeded in training a policy for the same robot using AMP without much tuning effort, but the code is based on rl_games instead of legged gym.
I hope my case helps you in solving your problem

[apirrone]

That's very interesting, thank you @WoohyunCha !

I think I may still have some bugs, even when I keep only the amp reward (no velocity tracking) and only the forward walk motion example, the robot has trouble learning something clean. There is shaking and it's not really a regular walking gait.
Before trying AMP_for_hardware, I was using IsaacGymEnv's implementation of AMP, and in their documentation, they say that the disc_grad_penalty parameter is critical:

If I am not mistaken, this parameter is not "parameterized" in AMP_for_hardware, so I dug into the code and I think I found it, so I parameterized it to play with it. I think it is this (in amp_ppo.py) but I am not sure.

@WoohyunCha Did you tune this parameter as well or did you keep it at 10 ? In this repo https://github.com/inspirai/MetalHead they seem to use a very low disc_grad_penalty of 0.01

Also, I record my motion examples at 60fps. My understanding is that AMPLoader interpolates the example motion to match the dt set in the config. Is that right ?

[WoohyunCha]

Hi.
I figured lambda=10 is reasonable because all the papers I have read uses 10. However, the actual weight should be 5, because the papers formulate

So I just divided it by 2, which still did not make things any better.

About the motion loader, my understanding is also the same, so the fps shouldn't be much of a problem. My reference motion are at 2000fps and my policy runs in 250Hz, but the imitation worked just fine.

My colleague who has succeeded in training a stable policy also used IsaacgymEnvs. I tried to match the parameters in this framework to those of IsaacgymEnvs, but the training does not improve at all. One key difference was that IsaacgymEnvs had a parameter named "disc_coef", which scales up the amp_loss and grad_pen_loss.

It is usually set as 5 in IsaacgymEnvs, which means if you set disc_grad_penalty to 5, the w_gp from the paper would be 50, which is much higher than what the papers suggest. Maybe this is why the lower disc_grad_penalty works.

[apirrone]

Ok, I'll try introducing the disc_coef parameter and keep playing with disc_grad_penalty.

Thank you for your valuable help @WoohyunCha !

[WoohyunCha]

Also, my colleague says including the b_loss from IsaacgymEnvs is important so maybe take a look into it as well.
Thanks for the discussion! It would be really helpful if you could tell me the stuffs that worked after you succeed :)

[apirrone]

I'll look into the b_loss !

Yeah I'll report back there, it could be useful for other people too :)

[apirrone]

So I have been playing with disc_coef, disc_grad_penalty, amp_task_reward_lerp, different example motion speeds, the scale of the tracking reward, the stiffness and damping parameters and action_scale, and I can't get anything very good. I have implemented the b_loss but I have not tried it on a long run yet.

This makes me think there is a bug somewhere, or that some of the parameters are much too far from what they should be. @WoohyunCha @Alescontrela any ideas ?

[apirrone]

I added clipping for the observations (5) and actions (1) like in IsaacGymEnvs, It seems to have a drastic effect on learning behavior. I'm starting to see things I like :)

amp_for_hardware-2024-08-07_22.57.30.mp4

This is with only motion imitation (amp_task_reward_lerp = 0) and not a lot of steps. I'm going to run a training with the velocity tracking overnight and hope for the best 🤞

[WoohyunCha]

Congrats @apirrone !!
I have been running things overnight too and finally made the policy imitate reference motion, and act differently according to the velocity commands although it does not perfectly track it. At least it tries to walk faster at higher commands and stands still at 0 commands.
It seems like either one of bound_loss or disc_grad_penalty was indeed the most crucial parameter.

Also, about clipping the observations and actions,
could you tell me more about how you implemented the clipping and scaling on observations?
By like in IsaacgymEnvs, do you mean you keep running estimates of mean and variance of observations, then normalize the observations instead of using fixed observation scales?

[apirrone]

Well this night's run did not yield the results I hoped for. Still no clean walk :'(

I'll keep playing with the parameters, and try b_loss out.

Concerning the obs and action clipping, I just set action clipping to 1 and observation clipping to 5, like it was by default in the IsaacGymEnv env I based mine on.

Btw, how did you find the values for min_normalized_std ? I just set it to None

[apirrone]

So I've been messing around all day, no luck.

Now, I'm trying to match all the parameters I had in IsaacGymEnvs, but I don't know if it's such a good idea since the a1_amp example of this repo works just fine ...

If anyone has any idea of things I could try, that would be great. I'm a little out of ideas 😅

[WoohyunCha]

Hi.
I successfully trained my policy to both imitate and track velocity commands in a pretty stable manner. The training nearly converges at 3000 iterations, so the training stability itself is pretty good too.

tocaboi_walk_05m.mp4

I actually did not change anything in min_normalized_std (it's the default value from this repository)

Don't mind the wgan part as I'm still working on it.

These are the parameters I used to train the stable version. After implementing bound loss and disc_coef along with the smaller disc_grad_pen, changes in other parameters did not affect the training that much. So maybe try my parameters? I have updated my version on my github repo so might as well check that.

If using the same parameters as me does not work, maybe we should also take a look into the reference motions and the commands you are giving. Could you describe what kind of reference motions you are using, how you set the policy observations and the discriminator states, and the ranges of commands (velocities) you are giving?

And also, I haven't run the a1_amp example. Does it work without all the additional stuffs we have discussed so far? (like disc_coef or bound loss) Matching everything with IsaacGymEnvs (especially the loss functions) worked for me so I think it's worth the try.

[apirrone]

That's a great result @WoohyunCha, congrats !

I have tried with pretty much the exact same parameters as in your screenshot, it wouldn't imitate the motion properly.

I generate the reference motion using a procedural walk engine, here is an example

amp_for_hardware-2024-08-09_10.02.39.mp4

I know these motions are somewhat realistic, as I can make the robot walk in mujoco using them

amp_for_hardware-2024-08-09_10.04.27.mp4

I also know I formatted them correctly for AMP_for_hardware's motion_loader.py because I can use the provided replay script

amp_for_hardware-2024-08-06_15.55.33.mp4

For now, I try to only make it walk forward properly, so my velocity command is always 0.1 m/s (I looked at the avereage linear velocity in mujoco when walking forward)

I matched the actual torque of the motors I'm using in the real robot (given by the specs), they are not very powerful. I was thinking maybe the robot is simply under actuated and it's hard to follow this walk properly, but I was able to learn a very clean walk with these specs in IsaacGymEnvs so I don't think it's that

isaac_walk-2024-07-16_17.52.32.mp4

I'll keep trying here, but at some point I will probably go back to IsaacGymEnv.

[BruceXing24]

@WoohyunCha can you share your reward scale?

[WoohyunCha]

@WoohyunCha can you share your reward scale?

I used 50 for linear velocity tracking and 25 for angular velocity tracking.