Segmentation fault occurring after n random timesteps

[faizansana]

The environment exits with a segmentation fault that occurs after n timesteps. This is an indeterministic number failing within even 2000 timesteps and as late as never as seen.

The following has been observed.

• It is seen to fail faster in python 3.8.16 (environment.yml) as compared to 3.7
• It was thought that this might be due to the collision sensor. However, the collision sensor was completely removed from the script and it stil continued to fail.
• This seg fault might be occurring due to failure of destroying the actor. It seems that the seg fault is occurring due to the carla library. However since the carla python library is a wrapper for C code, it is hard to debug exactly what is going wrong.
• After the traffic manager instantiation was moved to after setting it to synchronous mode, it seems these seg faults occur less. However, this could be unrelated.

[faizansana]

An instance of when the seg fault occured when a proper error code was given.

In most cases, it just says "Segmentation Fault" and exits or "Core dumped".

[faizansana]

Could also probably try to reload world at each reset. This might get rid of all actors since it seems that the GPU memory occupied by CARLA slowly increases with time. This could potentially be due to actors/sensors/other artifacts not being destroyed properly.

[faizansana]

Could also potentially try storing actor ids of all actors and then using batch to destroy them as seen in the Walker Batch Recipe

[faizansana]

Looks like seg fault being caused by traffic manager carla-simulator/carla#6358

[faizansana]

When using gdb on SHEVS laptop with RTX 3070 but algorithm deployed on CPU, the following was obtained.

Thread 40 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f17aa7fc700 (LWP 7911)]
0x00000000004e5450 in _PyObject_GC_UNTRACK_impl (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, 
    lineno=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, 
    filename=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.16/Modules/gcmodule.c:1951
1951    /usr/local/src/conda/python-3.8.16/Modules/gcmodule.c: No such file or directory.

[faizansana]

Output from faulthandler:

Current thread 0x00007f29b99d4280 (most recent call first):
  File "/home/docker/src/src/carla_env_custom.py", line 638 in _try_spawn_random_walker_at
  File "/home/docker/src/src/carla_env_custom.py", line 568 in _spawn_surrounding_pedestrians
  File "/home/docker/src/src/carla_env_custom.py", line 492 in reset
  File "/home/docker/src/src/carla_env_custom.py", line 750 in <module>
Segmentation fault (core dumped)

Shows issue is with
walker_controller_actor.go_to_location(self.world.get_random_location_from_navigation())

[faizansana]

In another machine, output from faulthandler:

Thread 0x00007f20c8141280 (most recent call first):
  File "/home/docker/src/src/carla_env_custom.py", line 635 in _try_spawn_random_walker_at
  File "/home/docker/src/src/carla_env_custom.py", line 567 in _spawn_surrounding_pedestrians
  File "/home/docker/src/src/carla_env_custom.py", line 491 in reset
  File "/home/docker/src/src/carla_env_custom.py", line 749 in <module>
Segmentation fault (core dumped)

Shows issue with following line:
walker_controller_actor.start()

[faizansana]

After testing 26 times on delta-ubuntu1, 40 times on shevs-laptop, it seems that the seg fault occurs everytime with respect to the following line:

walker_controller_actor.go_to_location()

This is also confirmed in the issue at carla-simulator/carla#4350 and carla-simulator/carla#6043