An issue about the evaluation script running in Docker

[kevinqyh0827]

Hi, I was trying run evaluation with pretrained models in Docker but have some errors. Here is the error information:

"Loading /root/data/objaverse_houses/houses_2023_07_28/val.jsonl.gz: 16000it [00:02, 6342.99it/s]
200 tasks in queue
Starting worker 0
Starting worker 1
Loading ckpt /root/data/pretrained_models/SigLIP-ViTb-3-double-det-CHORES-S/checkpoint_final.ckpt using ckpt_prefix='model.' ...
WARNING: worker 0 failed to stop with non-None task_sampler
Process ForkServerProcess-1:
Traceback (most recent call last):
File "/root/spoc/tasks/abstract_task_sampler.py", line 131, in controller
return self.controller_type(**self.controller_args)
File "/root/spoc/environment/stretch_controller.py", line 64, in init
self.controller = Controller(**kwargs)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 545, in init
self.start(
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 1545, in start
self._start_unity_thread(env, width, height, unity_params, image_name)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 1268, in _start_unity_thread
raise Exception(message)
Exception: Unity process has exited - check ~/.config/unity3d/Allen\ Institute\ for\ Artificial\ Intelligence/AI2-THOR/Player.log for errors. Confirm that Vulkan is properly configured on this system using vulkaninfo from the vulkan-utils package. returncode=-11

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/miniconda3/envs/spoc/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/miniconda3/envs/spoc/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/spoc/online_evaluation/online_evaluator_worker.py", line 57, in start_worker
worker.distribute_evaluate(agent, tasks_queue, results_queue)
File "/root/spoc/online_evaluation/online_evaluator_worker.py", line 484, in distribute_evaluate
task = self.task_sampler.next_task()
File "/root/spoc/tasks/multi_task_eval_sampler.py", line 165, in next_task
self.increment_task_and_reset_house(
File "/root/spoc/tasks/multi_task_eval_sampler.py", line 150, in increment_task_and_reset_house
self.reset_controller_in_current_house_and_cache_house_data(
File "/root/spoc/tasks/abstract_task_sampler.py", line 168, in reset_controller_in_current_house_and_cache_house_data
self.reset_scene_with_timeout_handler()
File "/root/spoc/tasks/abstract_task_sampler.py", line 237, in reset_scene_with_timeout_handler
self.controller.reset(scene=self.current_house)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/torch/distributions/utils.py", line 112, in get
value = self.wrapped(instance)
File "/root/spoc/tasks/abstract_task_sampler.py", line 134, in controller
raise TaskSamplerInInvalidStateError("Controller has closed.")
utils.data_generation_utils.exception_utils.TaskSamplerInInvalidStateError: Controller has closed.
Logging and waiting for proccesses to finish
Loading ckpt /root/data/pretrained_models/SigLIP-ViTb-3-double-det-CHORES-S/checkpoint_final.ckpt using ckpt_prefix='model.' ...
WARNING: worker 1 failed to stop with non-None task_sampler
Process ForkServerProcess-2:
Traceback (most recent call last):
File "/opt/miniconda3/envs/spoc/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/miniconda3/envs/spoc/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/spoc/online_evaluation/online_evaluator_worker.py", line 57, in start_worker
worker.distribute_evaluate(agent, tasks_queue, results_queue)
File "/root/spoc/online_evaluation/online_evaluator_worker.py", line 484, in distribute_evaluate
task = self.task_sampler.next_task()
File "/root/spoc/tasks/multi_task_eval_sampler.py", line 165, in next_task
self.increment_task_and_reset_house(
File "/root/spoc/tasks/multi_task_eval_sampler.py", line 150, in increment_task_and_reset_house
self.reset_controller_in_current_house_and_cache_house_data(
File "/root/spoc/tasks/abstract_task_sampler.py", line 168, in reset_controller_in_current_house_and_cache_house_data
self.reset_scene_with_timeout_handler()
File "/root/spoc/tasks/abstract_task_sampler.py", line 237, in reset_scene_with_timeout_handler
self.controller.reset(scene=self.current_house)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/torch/distributions/utils.py", line 112, in get
value = self.wrapped(instance)
File "/root/spoc/tasks/abstract_task_sampler.py", line 131, in controller
return self.controller_type(**self.controller_args)
File "/root/spoc/environment/stretch_controller.py", line 64, in init
self.controller = Controller(**kwargs)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 545, in init
self.start(
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 1545, in start
self._start_unity_thread(env, width, height, unity_params, image_name)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 1237, in _start_unity_thread
command = self.unity_command(width, height, self.headless)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 1147, in unity_command
raise RuntimeError(
RuntimeError: vulkaninfo failed to run, please ask your administrator to install vulkaninfo (e.g. on Ubuntu systems this requires running sudo apt install vulkan-tools).
"

So following the default suggestion, I modify the Dockerfile add this line below to make sure vulkan-tools is successfully installed. However, after I rebuild this image, the problem is still there. Any ideas of this issue? Or should I install the vulkan-tools on the host side not in the docker side?

Thanks so much!

[ehsanik]

@Lucaweihs @AlvaroHG Seems like this might be a THOR issue?

[kevinqyh0827]

Hi, there. I would like to give some updates about this issue.

Previously, I ssh to a remote Ubuntu server from my laptop with a Windows 11 OS. I am using the default powershell terminal. Now I switched to the MobaXterm (https://mobaxterm.mobatek.net/) which support the X11 forwarding feature. After rebuild the docker image and run the evaluating script, here are some screeshots of the output:

So the error message about "vulkaninfo" has disappeared. It looks like the clip model is also imported correctly and the CloudRendering is working. However, we can notice that the Rate of tasks is always zero, leading to the ETA to a very large number.

I hope this info could be helpful for troubleshooting. Thanks!

[kevinqyh0827]

Hi, there. I would like to give more updates about this.

I tested solely on AI2THOR and find the core reason seems from the "CloudRendering" platform, which is used as the headless setup option.

To start an AI2THOR controller, I tried this:

from ai2thor.controller import Controller 
controller = Controller(scene="FloorPlan10")

Then, a unity window poped up and everything looks normal/

However, when I switched the second line to:
controller = Controller(scene="FloorPlan10", platform=CloudRendering)
The error message is:

Traceback (most recent call last):
File "", line 1, in
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 545, in init
self.start(
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 1545, in start
self._start_unity_thread(env, width, height, unity_params, image_name)
File "/opt/miniconda3/envs/spoc/lib/python3.10/site-packages/ai2thor/controller.py", line 1268, in _start_unity_thread
raise Exception(message)
Exception: Unity process has exited - check ~/.config/unity3d/Allen\ Institute\ for\ Artificial\ Intelligence/AI2-THOR/Player.log for errors. Confirm that Vulkan is properly configured on this system using vulkaninfo from the vulkan-utils package. returncode=-11

It is the part of the bulky error mentioned above. The Player.log looks like:

One last question is, what is a example output of this evaluation program on pretrianed models? Is it video demonstrating the agent completing a task or any other thing?

Hope this helps for debugging and looking forward for any kind of suggestions.

[Galaxyphan]

Hello, I'm currently working with this project, and I've encountered the same issue.

Have you found a solution?

Any guidance would be greatly appreciated.

Thank you very much for your time and help!