Create a validation suite

[iparask]

The use case requires a validation suite that allows us to run the pipelines and find out whether there are differences or not.

• Brad ToDo: Provide link with images and necessary output for validation
• Giannis ToDo: Create a script to run the pipeline and validate the output.

[bspitzbart]

The predicted_shapefiles output needed for validation are usually found in a subdirectory of the input images. (given by --test_dir argument) It looks like:
$test_dir/WideResnetCount/predicted_shapefiles/
They should be on the same directory level as the csv output.

[iparask]

That would be where I find them, do you have some precalcualted that I can use for validation?

[iparask]

There is a point on the project drive that has the validation input as well as the validation output for Seals

[iparask]

This is blocking by #24 and #34

[iparask]

We have to finalize this aspect of the project.

[bspitzbart]

I added 5 images to bridges to use for validation:
/pylon5/mc3bggp/bspitz/Seals
WV03_20170301144508_104001002A465700_17MAR01144508-P1BS-501556087040_01_P001_u08rf3031.tif
WV03_20170204000736_104001002678E600_17FEB04000736-P1BS-501513717060_01_P001_u08rf3031.tif
WV03_20170203044432_1040010028C45900_17FEB03044432-P1BS-501172834040_01_P001_u08rf3031.tif
WV03_20170202221638_104001002983B700_17FEB02221638-P1BS-501174626090_01_P002_u08rf3031.tif
WV03_20170202190457_1040010028965400_17FEB02190457-P1BS-501172778010_01_P004_u08rf3031.tif

[iparask]

Can you move them to a folder named validation? I think this is the same folder as the one used for experiments, right?

Furthermore, I think we should find a set of unclassified images that we can use for validation. We will need to run them multiple times as we develop the ICEBERG framework.

[bspitzbart]

Ok. Moved to /pylon5/mc3bggp/bspitz/Validation
What do you mean by unclassified?

[iparask]

Images that we can have in a public web server with open access.

Let's say we need the space those images are occupying for another use case ( I know it sounds extreme, but this is hypothetical) and we remove them. At the same time, I make a change in the EnTK scripts because the API changed. I execute the validation suit as the last level of testing and it fails. We will probably get to an investigation of trying to find a bug that does not exist, because the input was not present.

I know this is an extreme case, but I think it is going to happen at some point in the lifetime of the software.