continuous-space algorithms with highway-env

[yangsl-njfu]

Hello author! The discrete-space algorithm (DQN) in stable-baselines3 can be trained in highway environment, but the continuous-space algorithms (PPO, SAC) find that the autocar always turns in place. How to solve this problem?

[seo-minseok]

Hello. I’m experiencing the same issue—have you found a solution by any chance?

[xpotato018-gif]

have you found a solution by any chance?

[eleurent]

Ah yes, the reward function in the highway env is to have positive speed, which driving in circle does achieve I guess (reward hacking). This env was primarily designed for discrete actions (which do not allow turning in place). Best would be to edit the reward function to only reward the X component of the velocity vector, rather than absolute speed.