TypeError and ValueError during HM3D evaluation

[AScicluna]

Hi there, thank you for releasing this work!

I am trying to run evaluation on the HM3D dataset (python -m vlfm.run), however have run into two sequential errors:

1. TypeError in blip2itm.py

When running the command, I first encounter this error:

  File "/home/ascic/Projects/uiap-ogn/uiap-ogn/vlfm/policy/vlplan_itm_policy.py", line 233, in act
    self._update_value_map()
  File "/home/ascic/Projects/uiap-ogn/uiap-ogn/vlfm/policy/vlplan_itm_policy.py", line 170, in _update_value_map
    mean = self._itm.cosine(rgb, processed_prompt)
  File "/home/ascic/Projects/uiap-ogn/uiap-ogn/vlfm/vlm/blip2itm.py", line 95, in cosine
    return float(response["response"])
TypeError: float() argument must be a string or a number, not 'list'

To fix this, I changed Line 95 of vlfm/vlm/blip2itm.py to remove the float() conversion, since response["response"] seems to be a list. However, this led to the next error, which I believe is related:

2. ValueError in vlplan_context_map.py

  File "/home/ascic/Projects/uiap-ogn/uiap-ogn/vlfm/policy/vlplan_itm_policy.py", line 233, in act
    self._update_value_map()
  File "/home/ascic/Projects/uiap-ogn/uiap-ogn/vlfm/policy/vlplan_itm_policy.py", line 177, in _update_value_map
    self._value_map.update_map(means, variances, depth, tf, min_depth, max_depth, fov)
  File "/home/ascic/Projects/uiap-ogn/uiap-ogn/vlfm/mapping/vlplan_context_map.py", line 139, in update_map
    self._fuse_new_data(curr_map, values, variances)
  File "/home/ascic/Projects/uiap-ogn/uiap-ogn/vlfm/mapping/vlplan_context_map.py", line 474, in _fuse_new_data
    obs_mean[..., c] = new_map * values[c]
ValueError: operands could not be broadcast together with shapes (1000,1000) (2,) 

This traceback suggests self._itm.cosine(rgb, processed_prompt) is returning a list (I assume containing the [mean, variance] of the ensembled BLIP results?). The current implementation of _update_value_map inside vlfm/policy/vlplan_itm_policy.py seems to only expect the mean from this function.

To fix this error, I commented Lines 170,171,174 and uncommented Lines 167,168,169,175 inside vlfm/policy/vlplan_itm_policy.py to reflect the expectation of using the mean and variance from the returned list.

After this the evaluation seems to run, but I would like to know if the current implementation of _update_value_map is purposeful? I want to confirm that making this change won't affect the evaluation results downstream.

I created the conda environment following the steps outlined in the README.md and have been able to successfully load the models. The only changes I have made was to vlfm/vlm/server_wrapper.py, to increase the timeout on Lines 136 and 147, and the device type for BLIP + YOLOv7 to temporarily run them through my CPU rather than the GPU.

If you need any more info regarding my environment setup, specs, etc, let me know. Would appreciate any insight on this. Cheers!