Some question about the data

[denyingmxd]

Thanks for thr great work and dataset. Yet, i meet a few problems when i try to use the dataset.

1. It is claimed that the synthetic datasets have 21 scenes which is listed in the paper and github readme file. Yet, there are only 20 in the download_spherecraft.py and the 'rainbow' is missing.

2. You mention that 5 synthetic scenes are for tesing but you also have specific train and test files in the form of [keypoint detector]_train.txt. If the txt file is used why we still need to divide scnes into training and testing scenes?

3. For classroom 00000000, the depth of window seems to be wrong and is the depth of what is behind

4. For urbanCanyon 00000000, the depth of buildings in this outdoor scene, seems to be too small

[cgava25]

Hi denyingmxd,

1. Thank you for pointing that out. It is fixed now.

2. We provide train and test files for all synthetic scenes simply for the sake of completeness and flexibility. If someone wishes to train their model on our entire dataset and then evaluate on another dataset, all files are readily available. Or, if they wish to use a different train-test split than the one we suggested, again all necessary files are available. Having the train and test files for all synthetics scenes simply enforces training and testing different models on exactly the same data -- so that these models can be fairly compared.

3. I am not sure I understand your point. The depth should correspond to what is behind the window if they ray of light passes through glass, which is what a downstream vision application (like 3D reconstruction) is supposed to sense. The depth values in the classroom scene are consistent with that.

4. In Table 2 of the paper we clearly marked Rainbow and Urban Canyon and stated "Scales used for Rainbow and Urban Canyon in their original Blender projects are unfortunately over- and undersized, respectively.", so these two scenes are exceptions.

[denyingmxd]

Thank you for your reply and here are some further question/reply regarding the spherecraft dataset for anyone else who is interested:

Q1. As for the maximum depth, you mention that they may vary depending on the scene. Nevertheless, mannually choosing one for each scene seems laborious. Maybe 100m is ok for all scenes? Yet. this may require that for small scenes, every pixels that has depth smaller than 100m is valid. Is this also true?
Q2. For the question on depth of what is behind the glass or window, from my experience on depth estimation, maybe depth of the glass can be important as this is considered as one challenging case for depth estimation and there are works intentionally on estimating the depth of transparent surfaces (e.g., glasses).
Q3. for the scale problem of rainbow and urbanCanyon, if we do not know the actual scale, how is the augmented satellite image generated. I think we need a tranlation vector to determine the location of the new camera. Yet, a small translation vector may not be sufficient for large scenes and large ones are not good for small scenes.

A1. You can use 100m for all scenes, and for small scenes this should do the trick, i.e. all pixels whose depth is smaller than 100m should be valid. But keep in mind that for the larger ones this will invalidate pixels that are otherwise valid but are far away (and that may or may not matter depending on the task).
A2. I see, but that is not the goal of SphereCraft. In keypoint detection, matching and camera pose estimation (SfM), it is important to see through glass and consider what is behind it.
A3. We initially used a maximum translation of 0.5m for all scenes for the rendering of satellite images, but we noticed that for Rainbow and Urban Canyon the result was not nice. So we adapted the maximum translation for these scenes to render satisfactory satellite images. The values are 1.5m and 0.1m, respectively.

[denyingmxd]

there are some artifacts in depth map, any way to fix this?