Not able to reproduce VLABench metrics

[serser]

While setting VLABench env, I faced a segmentfault, for which I apply the following walkaround patch.

diff --git a/VLABench/envs/dm_env.py b/VLABench/envs/dm_env.py
index 7c12f0e..cc3d82c 100644
--- a/VLABench/envs/dm_env.py
+++ b/VLABench/envs/dm_env.py
@@ -14,7 +14,7 @@ class LM4ManipDMEnv(composer.Environment):
             height=480,
             width=480,
         )
-        self.register_pcd_generator()
+        # self.register_pcd_generator()  # Disabled due to Open3D segfault in container
         
     def reset(self):
         self.timestep = 0
@@ -26,7 +26,7 @@ class LM4ManipDMEnv(composer.Environment):
         self.reset_gravity_and_fluid(self.physics)
         for _ in range(self.reset_wait_step):
             self.step()
-        self.register_pcd_generator()
+        # self.register_pcd_generator()  # Disabled due to Open3D segfault in container
         return timestep
         
     def render(self, **kwargs):
@@ -135,7 +135,9 @@ class LM4ManipDMEnv(composer.Environment):
         observation["robot_mask"] = np.where((observation["segmentation"][..., 0] <= 72)&(observation["segmentation"][..., 0] > 0), 0, 1).astype(np.uint8)
         observation["instrinsic"] = np.array(instrinsic_matrixs)
         observation["extrinsic"] = np.array(extrinsic_matrixs)
-        self.pcd_generator.physics = self.physics
+        # Only access pcd_generator if require_pcd is True and it exists
+        if require_pcd and hasattr(self, 'pcd_generator'):
+            self.pcd_generator.physics = self.physics
         if require_pcd:
             observation["masked_point_cloud"] = self.pcd_generator.generate_pcd_from_rgbd(target_id=list(range(self.physics.model.ncam - 1)), 

Then I was able to run the evaluation on 2toinf/X-VLA-VLABench, however, the results are quite different from what is written in the paper.

Metric	track_1_in_distribution	track_2_cross_category	track_3_common_sense	track_4_semantic_instruction	Average
Progress score	0.4433	0.1872	0.3283	0.3922	0.3377
Intention Score	0.8188	0.4422	0.7200	0.7899	0.6927
Success Rate	0.11	0.0	0.04	0.0877	0.0594

Your paper shows,

Track	Progress Score?
In Distribution	67.8
Cross Category	25.1
Common Sense	48.2
Semantic Instruction	63.1
Average	51.1

Could you confirm whether the paper provides the progress score and how can I fix to reproduce the result? Is it related to this Open3D segfault and how did you fix this?

[serser]

It turns out to be a numpy version problem. After resolving this issue by downgrading numpy to the recommended version 1.25.0, the reproduced metrics remained the similar as the first attempt. Could you provide further directions to reproduce your results?