Could not load ResNet model

[ProGamerGov]

ubuntu@ip-Address:~/fast-neural-style$ th slow_neural_style.lua -content_image content.jpg -style_image style.jpg -output_image out.png -save_every 50 -print_every 50 -preprocessing resnet -loss_network models/resnet_50_1by2_nsfw.caffemodel -num_iterations 1550 -gpu 0 -backend cuda -use_cudnn 1 -optimizer adam
ERROR: Could not load loss network from models/resnet_50_1by2_nsfw.caffemodel
You may need to download the VGG-16 model by running:
bash models/download_vgg16.sh
ubuntu@ip-Address:~/fast-neural-style$

[jcjohnson]

fast-neural-style expects Torch checkpoints instead of Caffe checkpoints; if you want to use ResNets then you should use these model checkpoints:

https://github.com/facebook/fb.resnet.torch/tree/master/pretrained

[ProGamerGov]

@jcjohnson Do you know of any projects I could use to convert .caffemodels to t7 .models?

[jcjohnson]

@ProGamerGov This script of mine uses loadcaffe and can convert some Caffe models to Torch for use in fast-neural-style, but it won't work with models like ResNet or GoogLeNet:

https://github.com/jcjohnson/cnn-benchmarks/blob/master/convert_model.lua

[ProGamerGov]

@jcjohnson So the traditional Caffe models that worked with the Classic (Original) Neural-Style, should work for conversion to Torch models?

I setup cnn-benchmarks, but I seem to receive the same error with any model I try to convert: jcjohnson/cnn-benchmarks#11

[jcjohnson]

Yes, any model that worked with original neural-style should be convertable to Torch format using the conversion script.

[jcjohnson]

I think this is resolved?

[ProGamerGov]

@jcjohnson Sorry for asking a question that is mostly unrelated to this issue, but what is the difference between .t7 and .t7b models? I'd like to convert your pre-trained model from: https://github.com/jcjohnson/densecap from it's .t7 format to a caffemodel format for use in Neural-Style, and the various model conversion guides I have found, use a .t7b model instead of a .t7 model.

[jcjohnson]

@ProGamerGov I don't think there is much of a difference; I'd guess that maybe .t7b is to emphasize that the model was saved in binary mode rather than ascii mode, but in practice almost all Torch checkpoints are saved in binary mode anyway since it is much more efficient.

BTW if you want to use the densecap model in neural-style then converting .t7 to caffemodel back to Torch seems overly complicated; it might be easier just to tweak neural-style so it can work directly with a .t7 file; this is for example how slow_neural_style.lua works.

[ProGamerGov]

@jcjohnson Thanks for the clarification on the Torch7 model file extensions.

I think you may be right that modifying Neural-Style is my best bet. I tried to convert the model using fb-caffe-exts, but that resulted in the error: unknown Torch class <DenseCapModel>, which I posted in this issue on the project. I'm not sure how difficult solving this issue is, and whether or not it's worth it to try and resolve the issue. Would this error indicate that there would be any issues with using the model in Neural-Style?

How difficult would it be for someone to implement the .t7 model support from slow_neural_style.lua into neural_style.lua? How similar are the two?

[jcjohnson]

@ProGamerGov

Adding .t7 model support to neural-style should be pretty straightforward; the main difference is that .t7 models don't have string layer names, so you'd need to index the style and content layers by index like in slow_neural_style.lua.

[htoyryla]

There was a pull request by @szagoruyko last March which allows loading a .t7 model into neural-style jcjohnson/neural-style#169 . I used it back then to try out ResNet.

[htoyryla]

As to the unknown Torch class issue, it happens all the time nowadays when trying to use models across projects (projects have implemented their own classes in addition to the standard ones). The solution is to copy the implementation of the missing class into the project where it is needed and require it (and hope that there are no conflicts).

I guess importing the missing class implementation would not help when converting into Caffe format, as Caffe would not understand the additional class anyway, but it would probably work in neural-style which is running in Torch.

[ProGamerGov]

@htoyryla Using @szagoruyko's Neural-Style modification here: jcjohnson/neural-style#169, results in the same issue as FB's Torch2Caffe converter.

/home/ubuntu/torch/install/bin/luajit: /home/ubuntu/torch/install/share/lua/5.1/
torch/File.lua:343: unknown Torch class <DenseCapModel>
stack traceback:
        [C]: in function 'error'
        /home/ubuntu/torch/install/share/lua/5.1/torch/File.lua:343: in function 'readObject'
        /home/ubuntu/torch/install/share/lua/5.1/torch/File.lua:369: in function  'readObject'
        /home/ubuntu/torch/install/share/lua/5.1/torch/File.lua:409: in function  'load'
        neural_style_t7.lua:77: in function 'main'
        neural_style_t7.lua:526: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50