Using model weights on own dataset

[CSteele97]

Hi, @jakubczakon

I would like to use the model weights to detect buildings from my own imagery, but I'm not entirely sure how to do this. I notice there are two files on the following website (https://ui.neptune.ai/neptune-ai/Mapping-Challenge/e/MC-1057/artifacts) - but I am not sure which file is the model weights and how to implement it on my own imagery. I have also seen the 'Predict on new data' section of REPRODUCE_RESULTS but I do not know what the pipeline_name would be or the prediction_path.

I hope this makes sense, I am very new to machine learning so do not yet understand a lot of things.

I would really appreciate it if you could provide some instructions on how I can achieve this. Thank you.

[kamil-kaczmarek]

hey @CSteele97,

1. unet is for segmentation task. Please take a look at this section for more info about unet and second level model. In general, you simply load trained model and use it for your own task.
2. prediction_path is the path where results will be stored as json file.

Hope this helps.

[kamil-kaczmarek]

For the simple case of predicting on some new data, prepare sources and environment, then follow this section: https://github.com/neptune-ai/open-solution-mapping-challenge/blob/master/REPRODUCE_RESULTS.md#predict-on-new-data

[CSteele97]

Hi @kamil-kaczmarek thank you for your reply.

In the case of the REPRODUCE_RESULTS section for predict on new data, would the pipeline_name therefore be unet, as this is the trained model?

Thank you

[kamil-kaczmarek]

Hey @CSteele97,

There is a full command provided in the aforementioned section. It looks like this:

python main.py predict_on_dir \
--pipeline_name unet_tta_scoring_model \
--chunk_size 1000 \
--dir_path path/to/inference_directory \
--prediction_path path/to/predictions.json

There is a pipeline name provided: unet_tta_scoring_model

Cheers,
Kamil

[CSteele97]

Thanks @kamil-kaczmarek

I have been trying to run the command you mentioned, but I get an error 'no module named neptune'. I have followed all the previous steps (without a Neptune registration) and am not sure why I am getting this error or how to resolve it.

I appreciate your time in helping me figure all of this out!

Thank you

[kamil-kaczmarek]

did you install neptune?

[kamil-kaczmarek]

It will be simplest workaround

[CSteele97]

I have managed to solve the neptune issue using pip install neptune-cli, thanks

[CSteele97]

I have tried to run the above command however I am now receiving 'Error: No such command 'predict_on_dir'

[kamil-kaczmarek]

I see that you installed neptune-cli. This will very likely not work as neptune-cli is our heritage library that we no longer support.

The best solution here is to create an environment using conda. Here is full specification of the conda environment: https://github.com/neptune-ai/open-solution-mapping-challenge/blob/master/environment.yml
Conda docs about managing environments: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Regarding Error with predict_on_dir. Please make sure that you run this command from the repo root. I think that it will solve the problem. This method is defined in the main file: https://github.com/neptune-ai/open-solution-mapping-challenge/blob/master/main.py#L51

Hope this helps :)

[CSteele97]

Thanks Kamil,

I have updated my environment which seems to now be working.

I have been running the command from the open-solution-mapping-challenge directory - is this correct?

Thank you

[kamil-kaczmarek]

Hey @CSteele97,

Yep, it should work.

[CSteele97]

Thanks Kamil,

I've tried running the command again from the aforementioned directory but it's still giving the predict_on_dir error - any idea why this might be?

[kamil-kaczmarek]

Hey,

Can you paste full error massage?

[CSteele97]

/anaconda3/envs/mapping/lib/python3.6/site-packages/sklearn/externals/joblib/init.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
warnings.warn(msg, category=FutureWarning)
/Users/open-solution-mapping-challenge/src/utils.py:132: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
/anaconda3/envs/mapping/lib/python3.6/site-packages/lightgbm/init.py:46: UserWarning: Starting from version 2.2.1, the library file in distribution wheels for macOS is built by the Apple Clang (Xcode_9.4.1) compiler.
This means that in case of installing LightGBM from PyPI via the pip install lightgbm command, you don't need to install the gcc compiler anymore.
Instead of that, you need to install the OpenMP library, which is required for running LightGBM on the system with the Apple Clang compiler.
You can install the OpenMP library by the following command: brew install libomp.
"You can install the OpenMP library by the following command: brew install libomp.", UserWarning)
/Users/open-solution-mapping-challenge/src/utils.py:132: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
Usage: main.py [OPTIONS] COMMAND [ARGS]...
Try 'main.py --help' for help.

Error: No such command 'predict_on_dir'.