Linker warnings give missplaced output

[JamesRamm]

When running the mapnik build, I'm getting a warning for each visual studio solution that was created by gyp, along the lines of:

MSB8012 TargetPath does not match the Linkers OutputFile property

The result is that there seems to be files missing from the output...
It seems this issue is caused by upgrading a solution from one version of VS to another (I presume gyp is handling this or something as the sln files are autogenerated). I don't know enough about gyp to fix it in the gyp file, so I tried manually editing the .sln files in VS2014, but could not get it going.

It would be helpful if the instructions gave some indication of the expected output and folder structure to help with determining what went wrong and where.

[wilhelmberg]

@JamesRamm these warnings shouldn't be a problem.

Do you have the mapnik-sdk already in place (libraries, that mapnik depends on, e.g. boost, icu, cairo, png, libxml2, ....)?

If not, a good starting place is mapnik-dependencies.

The docs are a bit out of date, as we have been changing a lot lately, but these steps should get you going:

• have VS2014 CTP4 installed
• have git installed
• have Python 2.7 installed
• have cmake on your PATH
• have wget on your PATH
• have 7z on your PATH

git clone https://github.com/BergWerkGIS/mapnik-dependencies.git
cd mapnik-dependencies
settings 64 14
scripts\build

This will download and build all dependencies and mapnik itself, which you will then find in

<your-clone-dir>\packages\mapnik-master\mapnik-gyp\mapnik-sdk

Be patient, depending on your machine and internet connection, the whole process might take some time.

The script will also try to build node-mapnik, which at the moment fails, but as you don't need it, that shouldn't be a problem.

[JamesRamm]

Yes, mapnik dependencies all built successfully following the instructions here:
https://github.com/BergWerkGIS/mapnik-dependencies
I had to make a few changes to the .bat files to sort out some paths, but all ran successfully.

My only problems are with the final mapnik build

Mapnik does build and the automatic tests seem to run successfully, but I get all the linker warnings and after looking at the folders/files generated it does seem like things are not where I would expect them to be.

I.e the resulting folder structure is not like what I would expect based on the currently available binaries/SDK.
The final mapnik-sdk folder doesn't contain the expected python bindings, but I do get a build folder which has a lib and Release folder

lib has the mapnik.dll, _mapnik.pyd, _mapnik.lib and a input folder of the plugins (.input and .lib, plus some other extensions), but no python files.

The Release folder has a lib folder which has the .py files in it, plus mapnik_json.lib and mapnik_wkt.lib.

I suspect that these two lib folders should really be one (either in Release or build?) and that it shouldve ended up in mapnik-sdk but did not due to the linker warning??

It may be that I simply don't understand the final folder structure and how to package that into something I can distribute with a python program. In any case, this is where the documentation ends, so it might be worth adding a little more to explain how the final output relates to what mapnik users are used to seeing?

My main purpose is to build the python bindings with cairo & pgraster support - we may move to C++ in the future but we really need the python bindings going with Cairo and pgraster for now.

[wilhelmberg]

Yes, you are correct, the Python bindings are not copied to the sdk directory.
Maybe @springmeyer could chime in, if it's just the copying that's missing, then I can take care of that during the next days.

[springmeyer]

Thanks for helping test this new build workflow @JamesRamm. We've made no effort yet to copy around the built results to be useable via python, but this should not be to hard. @BergWerkGIS would be great if you want to take a look at that this week. The only tricky bit is the generation of the paths.py file and ideally using relative paths in that file so that people can put the Mapnik SDK anywhere on their system and still have the python bindings be able to register fonts and input plugins correctly.

[springmeyer]

I'll add that this is the settings file I use on linux to test the python bindings in-place (https://github.com/mapnik/mapnik-gyp/blob/master/localize.sh) without having to move them to a more reasonable structure. But yeah, we should start doing this so that you can more easily just set PYTHONPATH and have it work.

[JamesRamm]

That would be great...
As the tests all ran and generated image files, the python bindings must have built fine...it is just figuring out how it should be packaged up

I did try moving the whole mapnik build folder (mapnik-master) off the VM and to my development system, setting the PATH variables as I thought it should be to get python to work (by looking at the various changes the bat scripts make to the PATH), but I get a missing DLL error that I couldn't resolve when importing from _mapnik, so I am clearly missing some steps.

I'll have some more play around and see if I can't make a batch file to shift the necess. files around.

[springmeyer]

@JamesRamm if you can push this forward that would be great. As far as debugging missing DLLs I presume you know about depedency walker? (http://www.dependencywalker.com/). Another trick is to try importing the module with ctypes and see what the error is: https://github.com/mapnik/node-mapnik/wiki/Troubleshooting#the-specified-module-cannot-be-found

[JamesRamm]

Ok, solution.
I did this manually, but it seems simple enough to add to the batch file.

1). mapnik-gyp/build/lib/python2.7 should be moved to mapnik-sdk. If it is like mine it will contain the _mapnik.pyd but not the .py files.

2). All the .py files in mapnik-gyp/build/Release/lib/python2.7/mapnik should be moved to the mapnik folder in mapnik-sdk/python2.7. I expect this to be __init__.py and printing.py

3). Create or edit the PYTHONPATH environment variable. Add the path to the python directory in mapnik-sdk... i.e C:\...\mapnik-sdk\python2.7. On my machine it is Z:\mapnik-dev\packages\mapnik-master\mapnik-gyp\mapnik-sdk\python2.7

4). Add the path to ..mapnik-sdk\libs and ...mapnik-sdk\bin to the system PATH variable.

5). Open a python interpreter and import mapnik. It should hopefully fail with this error:
ImportError: No module named paths.
This is good as it means we successfully imported _mapnik.pyd and now it is just failing because we now need to create paths.py.

On our dev/testing/whatever machine, we have two options. We can simply specify the environment variables MAPNIK_INPUT_PLUGINS_DIRECTORY and MAPNIK_FONT_DIRECTORY, pointing to the respective directories and it should work, or we can manually create paths.py which __init__.py turns to if the environment variables do not exist.

For release, we probably want to set something up so paths.py is generated.

The contents of my paths.py (manually created) is:

from os import path
mapniklibpath = path.normpath(path.join(__file__, '../../libs'))
inputpluginspath = path.join(mapniklibpath, 'mapnik/input')
fontscollectionpath = path.join(mapniklibpath, 'mapnik/fonts')
__all__ = [mapniklibpath, inputpluginspath, fontscollectionpath]

OK..that done, restart the interpreter and try importing mapnik again. All should be fine.
My next steps will be to test that it actually works :)

[JamesRamm]

Oo, should also note that (and this is quite important actually), the msvcp140.dll, appcrt140.dll, desktopcrt140.dll and vcruntime140.dll need to be on the users' path, which is unlikely as visual studio 2014 is not even released yet...

These last 3 DLL's replace the msvcrXX.dll of previous VS editions...
I don't know what you could do about that for releasing it (other than warn users they need these...)?

[JamesRamm]

@springmeyer , @BergWerkGIS FYI, I figured this issue would be better on the main project as it is not strictly a build issue:

mapnik/mapnik#2570

But you may be able to chime in as it is potentially (?) due to some build problem...

[JamesRamm]

I'm finding that I have to set the PROJ_LIB environment variable on my deployment machine to get stuff like "+init=epsg:4326" to work.
Not a problem at all, but it wasn't necessary for my downloaded mapnik binaries (v2.2) so I'm wondering what causes it?

[springmeyer]

I'm finding that I have to set the PROJ_LIB environment variable on my deployment machine to get stuff like "+init=epsg:4326" to work.
Not a problem at all, but it wasn't necessary for my downloaded mapnik binaries (v2.2) so I'm wondering what causes it?

This is expected. The needed fix is #7. Want to take a crack at that?

[springmeyer]

Oo, should also note that (and this is quite important actually), the msvcp140.dll, appcrt140.dll, desktopcrt140.dll and vcruntime140.dll need to be on the users' path

Correct, for now the documentation for windows needs to note that vcredist.exe needs to be installed. Want to add something like this to the readme?

To distribute the windows binaries built by mapnik-gyp your users will need to first install (just once) the Visual C++ Redistributable Packages for Visual Studio 2014 and the architecture you are shipping:

https://mapbox.s3.amazonaws.com/node-cpp11/vcredist_x64.exe
https://mapbox.s3.amazonaws.com/node-cpp11/vcredist_x86.exe

[springmeyer]

I did this manually, but it seems simple enough to add to the batch file.

Absolutely I think we should add it to the bat / programmatically set everything up just like the final package will be. The use of the environment variables of MAPNIK_INPUT_PLUGINS_DIRECTORY/etc are just for testing and ideally we don't use them.