ValueError: Requested RearrangeDataset config paths 'data/datasets/replica_cad/rearrange/v2/train/rearrange_easy.json.gz' or 'data/replica_cad/' are not downloaded locally. Aborting.

[yashwant-ies]

following the readme instruction as it is described to setup, install and run example.py and ended up with this error. retried several times by removed data directory but no luck:

(habitat) yashwant@octopus:~/Downloads/habitat-lab$ python examples/example.py
pybullet build time: Nov  2 2024 11:43:58
PluginManager::Manager: duplicate static plugin StbImageImporter, ignoring
PluginManager::Manager: duplicate static plugin GltfImporter, ignoring
PluginManager::Manager: duplicate static plugin BasisImporter, ignoring
PluginManager::Manager: duplicate static plugin AssimpImporter, ignoring
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring
PluginManager::Manager: duplicate static plugin AnyImageImporter, ignoring
2024-11-02 11:51:19,396 Initializing dataset RearrangeDataset-v0
Traceback (most recent call last):
  File "/home/yashwant/Downloads/habitat-lab/examples/example.py", line 31, in <module>
    example()
  File "/home/yashwant/Downloads/habitat-lab/examples/example.py", line 15, in example
    with gym.make("HabitatRenderPick-v0") as env:
  File "/home/yashwant/anaconda3/envs/habitat/lib/python3.9/site-packages/gym/envs/registration.py", line 676, in make
    return registry.make(id, **kwargs)
  File "/home/yashwant/anaconda3/envs/habitat/lib/python3.9/site-packages/gym/envs/registration.py", line 520, in make
    return spec.make(**kwargs)
  File "/home/yashwant/anaconda3/envs/habitat/lib/python3.9/site-packages/gym/envs/registration.py", line 140, in make
    env = cls(**_kwargs)
  File "/home/yashwant/Downloads/habitat-lab/habitat-lab/habitat/gym/gym_definitions.py", line 91, in _make_habitat_gym_env
    env = make_gym_from_config(config)
  File "/home/yashwant/Downloads/habitat-lab/habitat-lab/habitat/gym/gym_definitions.py", line 60, in make_gym_from_config
    return make_env_fn(env_class=env_class, config=config, dataset=dataset)
  File "/home/yashwant/Downloads/habitat-lab/habitat-lab/habitat/utils/env_utils.py", line 36, in make_env_fn
    dataset = make_dataset(config.dataset.type, config=config.dataset)
  File "/home/yashwant/Downloads/habitat-lab/habitat-lab/habitat/datasets/registration.py", line 22, in make_dataset
    return _dataset(**kwargs)  # type: ignore
  File "/home/yashwant/Downloads/habitat-lab/habitat-lab/habitat/datasets/rearrange/rearrange_dataset.py", line 59, in __init__
    raise ValueError(
ValueError: Requested RearrangeDataset config paths 'data/datasets/replica_cad/rearrange/v2/train/rearrange_easy.json.gz' or 'data/replica_cad/' are not downloaded locally. Aborting.
(habitat) yashwant@octopus:~/Downloads/habitat-lab$ ls

[walden233]

Same for me

[linyx700]

same for me, I installed the habitat-lab completely according to the official tutorial. Sincerely hope to get help

[aclegg3]

Hey folks, how did step 1 of Testing go here?
Part of installation is getting the data (assets and episodes) from the downloader tool:

python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/
python -m habitat_sim.utils.datasets_download --uids habitat_test_pointnav_dataset --data-path data/

In the above code snippet we're downloading the data to ./data/ so it should be run within the habitat-lab directory. If you put the data elsewhere you'll need to modify configs to correct the paths.

[shepera]

same for me. I installed on linux server(Ubuntu 24.04) last week and it works perfectly fine. But this week, I met this problem when doing the same thing on my MacBook. Now I find this error also appears when I try to install another one on the linux server.

[syw-robotics]

Hey folks, how did step 1 of Testing go here? Part of installation is getting the data (assets and episodes) from the downloader tool:

python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/
python -m habitat_sim.utils.datasets_download --uids habitat_test_pointnav_dataset --data-path data/

In the above code snippet we're downloading the data to ./data/ so it should be run within the habitat-lab directory. If you put the data elsewhere you'll need to modify configs to correct the paths.

Step 1 has been run as indicated, ValueError: Requested RearrangeDataset config paths 'data/datasets/replica_cad/rearrange/v2/train/rearrange_easy.json.gz' or 'data/replica_cad/' are not downloaded locally. Aborting. still arises whatever I install habitat stable or habitat-nightly within conda.

[syw-robotics]

I fixed ValueError: Requested RearrangeDataset config paths 'data/datasets/replica_cad/rearrange/v2/train/rearrange_easy.json.gz' or 'data/replica_cad/' are not downloaded locally. Aborting.

• machine: ubuntu 22.04 (both wayland and x11 return same results)
• habitat-sim version: stable(0.3.2) or nightly via conda
• python version of conda venv: Python 3.9.20

Solution

python -m habitat_sim.utils.datasets_download --uids <replica_datasets> --data-path <path_to_cloned_habitat-lab_repo>/data

where <replica_datasets> includes :

"replica_cad_dataset",
"hab_fetch",
"ycb",
"rearrange_pick_dataset_v0",
"rearrange_dataset_v2",

(see, https://github.com/facebookresearch/habitat-sim/blob/dfb388e29e5e1f25da4b576305e85bdc0be140b8/src_python/habitat_sim/utils/datasets_download.py#L341. Here rearrange_dataset_v1 should be replaced with rearrange_dataset_v2)

Feedback

Now, python examples/example.py should work perfectly, but python examples/interactive_play.py --never-end throws X Error of failed request: BadAccess (attempt to access private resource denied) as the README said which is same on a Ubuntu 20.04 machine.

Not sure the perfact solution for this currently, but it do keep people from using habitat as a research platform.

[mlohbihler]

Is anyone else getting this error now?

Data download failed, no datasource named rearrange_dataset_v2

The example.py script is still failing with the same error.

Note that running python -m habitat_sim.utils.datasets_download --list shows that all of the datasets exist except "rearrange_dataset_v2" (atow). I tried downloading "rearrange_dataset_v1" instead, but that did not work either in that the download worked but the example script still did not.

[mlohbihler]

Update: changing "v2" to "v1" in habitat/config/benchmark/rearrange/skills/pick.yaml got it pretty much working:

  dataset:
    data_path: data/datasets/replica_cad/rearrange/v1/{split}/rearrange_easy.json.gz

"Pretty much" because there were a lot of warnings:

PluginManager::Manager: duplicate static plugin StbImageImporter, ignoring
PluginManager::Manager: duplicate static plugin GltfImporter, ignoring
PluginManager::Manager: duplicate static plugin BasisImporter, ignoring
PluginManager::Manager: duplicate static plugin AssimpImporter, ignoring
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring
PluginManager::Manager: duplicate static plugin AnyImageImporter, ignoring
2024-11-25 17:19:30,369 Initializing dataset RearrangeDataset-v0
2024-11-25 17:19:47,577 initializing sim RearrangeSim-v0
[00:15:18:239737]:[Warning]:[Metadata] AttributesManagerBase.h(457)::buildAttrSrcPathsFromJSONAndLoad : <Articulated Object> : No Glob path result found for `data/hab_fetch_1.0/robots/fetch_no_base.urdf` so unable to load templates from that path.
[00:15:18:296524]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_00.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296565]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_01.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296588]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_02.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296823]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_03.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296862]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_04.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296884]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_05.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296906]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_06.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296925]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_07.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296947]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_08.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296967]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_09.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:296986]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_10.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297008]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_11.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297034]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_12.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297056]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_13.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297076]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_14.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297096]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_15.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297116]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_16.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297135]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_17.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297156]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_18.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297175]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_19.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
[00:15:18:297195]:[Error]:[Metadata] SceneDatasetAttributesManager.cpp(304)::validateMap : `navmesh_instances` Value : `navmeshes/v3_sc4_staging_20.navmesh` not found on disk as absolute path or relative to `data/replica_cad`
PluginManager::Manager: duplicate static plugin StbImageImporter, ignoring
PluginManager::Manager: duplicate static plugin GltfImporter, ignoring
PluginManager::Manager: duplicate static plugin BasisImporter, ignoring
PluginManager::Manager: duplicate static plugin AssimpImporter, ignoring
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring
PluginManager::Manager: duplicate static plugin AnyImageImporter, ignoring
Renderer: AMD Radeon Pro 5300M OpenGL Engine by ATI Technologies Inc.
OpenGL version: 4.1 ATI-6.1.13
Using optional features:
    GL_ARB_vertex_array_object
    GL_ARB_ES2_compatibility
    GL_ARB_separate_shader_objects
    GL_ARB_texture_storage
    GL_EXT_texture_filter_anisotropic
    GL_EXT_debug_label
    GL_EXT_debug_marker
Using driver workarounds:
    no-layout-qualifiers-on-old-glsl
    apple-buffer-texture-unbind-on-buffer-modify
MeshTools::compile(): ignoring Trade::MeshAttribute::TextureCoordinates 1 as its binding slot is already occupied by Trade::MeshAttribute::TextureCoordinates 0
MeshTools::compile(): ignoring Trade::MeshAttribute::TextureCoordinates 1 as its binding slot is already occupied by Trade::MeshAttribute::TextureCoordinates 0
MeshTools::compile(): ignoring Trade::MeshAttribute::TextureCoordinates 1 as its binding slot is already occupied by Trade::MeshAttribute::TextureCoordinates 0
MeshTools::compile(): ignoring Trade::MeshAttribute::TextureCoordinates 1 as its binding slot is already occupied by Trade::MeshAttribute::TextureCoordinates 0
[00:15:19:299558]:[Warning]:[Sim] Simulator.cpp(594)::instanceStageForSceneAttributes : The active scene does not contain semantic annotations : activeSemanticSceneID_ = 0
2024-11-25 17:19:51,294 Initializing task RearrangePickTask-v0
/opt/anaconda3/envs/habitat/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
Environment creation successful
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Tbl1_Top1_frl_apartment_table_01, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_counter_left_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_sink_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_sink_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_drawer_right_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_TvStnd1_Top1_frl_apartment_tvstand, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_TvStnd1_Top1_frl_apartment_tvstand, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_drawer_right_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Tbl2_Top1_frl_apartment_table_02, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Chr1_Top1_frl_apartment_chair_01, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Tbl2_Top1_frl_apartment_table_02, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_drawer_middle_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Tbl1_Top1_frl_apartment_table_01, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_counter_right_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Chr1_Top1_frl_apartment_chair_01, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Tbl1_Top1_frl_apartment_table_01, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_drawer_right_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_drawer_middle_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_drawer_left_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_counter_left_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,716 Cannot find active receptacle receptacle_aabb_Sofa_frl_apartment_sofa, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_drawer_middle_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_counter_left_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_drawer_right_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_TvStnd1_Top1_frl_apartment_tvstand, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_drawer_middle_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_middle_topfrl_apartment_refrigerator, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_TvStnd1_Top1_frl_apartment_tvstand, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_drawer_right_top_frl_apartment_kitchen_counter, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
2024-11-25 17:19:54,717 Cannot find active receptacle receptacle_aabb_TvStnd1_Top1_frl_apartment_tvstand, so cannot create a parent relationship. Skipping. Note that episode is likely invalid.
Agent acting inside environment.
Episode finished after 300 steps.

[Lisalsj]

I encountered the same problem as you. May I ask if you have resolved it?

[mlohbihler]

I encountered the same problem as you. May I ask if you have resolved it?

As i mentioned i have pretty much resolved it in that i was able to continue with the instructions, including the interactive testing, but it was with v1 of the rearrange env. I don't know how different v2 is.

[Lisalsj]

I encountered the same problem as you. May I ask if you have resolved it?

As i mentioned i have pretty much resolved it in that i was able to continue with the instructions, including the interactive testing, but it was with v1 of the rearrange env. I don't know how different v2 is.

Could you please let me know which document you are referring to for this operation? It appears that I do not have the document you mentioned.

[mlohbihler]

Could you please let me know which document you are referring to for this operation? It appears that I do not have the document you mentioned.

What i wrote above:

Update: changing "v2" to "v1" in habitat/config/benchmark/rearrange/skills/pick.yaml got it pretty much working:

  dataset:
    data_path: data/datasets/replica_cad/rearrange/v1/{split}/rearrange_easy.json.gz

I also made the following additional changes:

In habitat/config/benchmark/rearrange/play/play.yaml change "v2" to "v1:

  dataset:
    data_path: data/datasets/replica_cad/rearrange/v1/{split}/rearrange_easy.json.gz

In habitat/examples/example.py change line 14 (the gym.make call) to:

    config = habitat.get_config(
    "benchmark/rearrange/skills/pick.yaml",
    overrides=["habitat.environment.max_episode_steps=20"]
    )

    # with gym.make("HabitatRenderPick-v0") as env:
    with gym.make_gym_from_config(config) as env: