Observation Space of the Agents

[odyssey33]

Hi,
https://github.com/OmniSafeAI/safety-gymnasium/blob/main/docs/components_of_environments/agents/ant.rst
From the page, I was able to know what state variables and ranges are there for each agent.
However, I noticed that the number of the observation space is not the same as the one that I actually printed with my code.
(See below.)
Where can I find the correct information about what each element of the observation state vector corresponds to?
Thanks.

[muchvo]

It's a good point. The original information for this came from the mechanism of Safety-Gym for observation space, while it sets all ranges to [-inf, inf]. We thought it is not appropriate and then fixed it in our implementation. Thanks for your reminder. We will correct the documentation.
Regarding your question, the information from the code is correct.

After we correct the information, the reason why the observation information from the code is different from that in the documentation is that each environment consists of three main elements: task, agent, and objects. In the documentation, we present the observation space as the specific observation space, which means that the entire observation space in the current environment is a union of all specific observation spaces of each element. In short, there is more information in the observation space in your code because some other objects depend on your task. For additional information that will be added to the observation space, apart from what we present in the agent part, you can refer to the specific observation space for each task in our docs.
If you want to get more details about each element of the observation state vector corresponds to what directly in code, use:

print(env.obs_space_dict)
# Dict('accelerometer': Box(-inf, inf, (3,), float64),
#     'velocimeter': Box(-inf, inf, (3,), float64),
#     'gyro': Box(-inf, inf, (3,), float64),
#     'magnetometer': Box(-inf, inf, (3,), float64),
#     'circle_lidar': Box(0.0, 1.0, (16,), float64))

[odyssey33]

Here is the screenshot.

[odyssey33]

also, I wanted to ask you how you made the reward plot. As far as I know, the reward is zero when you reset the environment, but your plot in the benchmark shows "continuously" increasing reward.

[odyssey33]

Sorry. Can I know how to resolve the time limit issue? As you said, I set the time step over each epoch to 20,000. But the environment did required me to reset itself at 1,000th time step.

[muchvo]

Thanks! I have one question, sorry. Can I ask you how many time-step you set for every episode when you trained the agent?

The time step for every episode is 1000 as a constant value. You can check out here.

For benchmarks provided, the answer is 2048(time steps per epoch) * 500(total epoch) in the 1e6 setting, and 20000(time steps per epoch) * 500(total epoch) in the 1e7 setting.

In this reply, the setting is for the training process of the algorithm.

Also, for the graphs, the solid line shows the mean of reward value and cost value in the training process over 5 seeds, and the shadow area represents the standard deviation. For example, if the agent ran 2 episodes in epoch=400, then we calculate the mean and std of total reward and cost in 2 episodes * 5 seeds = 10 episodes. It is a general setting in the RL community.

I also want to clarify that in reinforcement learning algorithms, the concepts of environment maximum time step and steps per epoch are different. The one you mentioned belongs to the former. environment maximum time step is typically predetermined, such as the mentioned 1000, which represents the maximum number of interactions before a reset. On the other hand, steps per epoch, like the suggested values of 2048 and 20000, can be considered as a defined parameter. It signifies the number of steps the algorithm interacts with the environment before returning for an update.

To be specific, if the steps per epoch is set to 2048, the process would be as follows: interact with the environment for 1000 steps -> reset the environment -> interact with the environment for 1000 steps -> reset the environment -> interact with the environment for 48 steps -> return for an update in the algorithm -> start the next round of interactions. If steps per epoch is set to 20000, the aforementioned process of interacting with the environment for 1000 steps and resetting will repeat 20 times.

[muchvo]

It's a good point. The original information for this came from the mechanism of Safety-Gym for observation space, while it sets all ranges to [-inf, inf]. We thought it is not appropriate and then fixed it in our implementation. Thanks for your reminder. We will correct the documentation. Regarding your question, the information from the code is correct. If you want to get more details about each element of the observation state vector corresponds to what, use:

print(env.obs_space_dict)
# Dict('accelerometer': Box(-inf, inf, (3,), float64),
#     'velocimeter': Box(-inf, inf, (3,), float64),
#     'gyro': Box(-inf, inf, (3,), float64),
#     'magnetometer': Box(-inf, inf, (3,), float64),
#     'circle_lidar': Box(0.0, 1.0, (16,), float64))

Hi, I realized that I made a mistake in my previous response. The reason why the observation information from the code is different from that in the documentation is that each environment consists of three main elements: task, agent, and objects. In the documentation, we present the observation space as the specific observation space, which means that the entire observation space in the current environment is a union of all specific observation spaces of each element. In short, there is more information in the observation space in your code because some other objects depend on your task. For additional information that will be added to the observation space, apart from what we present in the agent part, you can refer to the specific observation space for each task in our docs.

I apologize for providing misleading information. I will revise my previous response to ensure clarity for other readers.