train PVPTD3 and TD3 issue

[yangsl-njfu]

When using the algorithms TD3 and PVPTD3 from the baseline3 library for training, the vehicle starts spinning from the beginning

[pengzhenghao]

Seems like the hyperparameter problem. We've successfully trained TD3 agents. I haven't tried out DDPG yet.

Please refer to:

https://github.com/metadriverse/pvp/blob/main/pvp/experiments/metadrive/train_td3_metadrive.py

This is the TD3 script I used.

Also, in the very begining of the training the agent might have some wierd behaviors, this is expected.

[yangsl-njfu]

Thank you for your reply, but when I run with this code (parameters of the source code), I found that when training PVPTD3 without manual intervention, the autorickshaw will start steering to hit the sidewalk (spinning) a few steps into the first round of exploring, and then the second round backward as soon as it starts to hit the sidewalk, and my question is whether this is normal or not? Also, I'm using TD3 and DDPG from baseline3, and training in metadrive I found that a similar problem occurs (starts spinning in place after a few rounds of training), so I'm asking for advice.

[pengzhenghao]

Could you just wait and see if TD3 converges after 1M steps? It's meaningless to study the initial behavior in the very begining of the training.

[yangsl-njfu]

Before discovering this problem, we have used the TD3 algorithm in stable-baselines3 to test different number of steps, and modified the relevant parameters such as noise and learning rate, but the result is that the car rotates in place. In addition, the algorithm was also trained in highway-env to eliminate the simulation environment problem, but the same problem occurred. Meanwhile, using the source code provided in this repository for testing, the car always starts to turn and hit the sidewalk after 50,000 steps of training without human intervention. I would like to ask the authors whether this is a normal phenomenon without human intervention, or is the rotation problem caused by the influence of the TD3 algorithm? In addition, I would like to ask the authors whether they have encountered similar problems when using the TD3 algorithm in stable-baseline3 (the DDPG algorithm also has this problem).

[pengzhenghao]

Thanks for reporting this. We will try to launch TD3 with latest MetaDrive to see if that happens.

In addition, the algorithm was also trained in highway-env to eliminate the simulation environment problem, but the same problem occurred.

This information makes me believe it's a hyperparameter problem of TD3.

Meanwhile, using the source code provided in this repository for testing, the car always starts to turn and hit the sidewalk after 50,000 steps of training without human intervention.

We will launch TD3 to see if that happens with latest MetaDrive. Maybe we should add some action noise to TD3?

I would like to ask the authors whether this is a normal phenomenon without human intervention, or is the rotation problem caused by the influence of the TD3 algorithm?

So you means the TD3 agent always be like this after 50000 steps, even you've trained it for 1M steps? According to my experience, you can try out (while we trying to reproduce the result) 1. lower the learning rate, 2) add some random action noise, 3) have a larger batch size, 4) have larger warmup steps.