Specify folders only in test/train json

[Necr0x0Der]

While it is convenient for debugging to have small test and train datasets defined via specifying separate files, it's not convenient for full benchmarking. Thus, in addition to the possibility of defining datasets on the per-file basis, it would be nice to have the possibility of defining them on the per-folder basis. It is desirable to add this option and create corresponding test and train datasets jsons.

[DaddyWesker]

I'm not quite sure I've understood what you're asking. So, currently make_dataset creates two json files which contains labels and pathes to the files in form of dict:

{
    "ice_block" : [list of pathes to files]
    ...
}

What you want to be changed in this case? Something like:

{
    "ice_block" : [folder which contains only ice_block related files]
    ...
}

?

[Necr0x0Der]

I don't like make_dataset for two reasons.

1. I don't see why we need to create json-files, when they are created in such the way. Why this function couldn't be called in runtime and return dicts.
2. My idea was to have some folders for testing generalization. Images from these folders should not appear in the training set. Currently, everything is shuffled.
   Maybe, we can have 3 datasets - train and test with shuffled folders and validation (or generalization test) dataset with novel folders. Basically, the training config should contain information about these folders. I don't see the need to list all the files and prepare this jsons separately. And the file lists can be loaded and separated into three datasets in runtime.

[DaddyWesker]

1. Well, my thoughts were that we doesn't want to wait every time we want to benchmark for all dataset preparation process to be done. Since make_dataset is not quite fast since it loops through all folders and then loops some more I thought it is better to call make_dataset, create json_files, and then, in a separate launch, call benchmark with ready-to-be-used json files.

And the file lists can be loaded and separated into three datasets in runtime.

Of course it is possible. As I said, I just thought it will be slow.

Maybe, we can have 3 datasets - train and test with shuffled folders and validation (or generalization test) dataset with novel folders.

So, suggestion is that third dataset is the dataset prepared from a folder without any split. Just load all files form a folder as is in the runtime and use it for validation purposes?