Building ImageMagick

[SimonDanisch]

Hi all,
I want to finally have a clean build for ImageMagick.
There seem to be a lot of issues when building ImageMagick.
It seems to me as if the current build system relies on a lot of uncertainties,.
E.g.:

• using __init__, in an automatically generated include. Depending on where included and what Julia version you have, __init__ will be called at different points, which makes it hard to reason about
• onload hooks are not guaranteed to get executed. Since there is no documentation its pretty opaque to me, but seems they don't get executed when the library is already present. But e.g. the variable ENV["MAGICK_CODER_MODULE_PATH"] should also be set even if ImageMagick is already present, as some of my experiments show. So putting this into onload is asking for trouble.
• why and when exactly ccall((:MagickQueryConfigureOption, libwand), Ptr{UInt8}, (Ptr{UInt8},), "LIB_VERSION_NUMBER") segfaults on OSX wasn't really found out?! Build segfaults on OS X JuliaImages/Images.jl#188
• I'm not sure why the version check is in a Module. Is this some hack for __init__ ?

From what I gathered we need these:

• always set ENV["MAGICK_CODER_MODULE_PATH"] and ENV["MAGICK_CODER_MODULE_PATH"] before calling any function from ImageMagick
• it seems osx needs also the path to be set!?
• libversion is only needed for sRGB vs RGB
• MagickWandGenesis and the ENV variables must be called before getting the version
• MagickWandGenesis only needs to get called one time after the build?! And on OSX every time the module loads!? I don't know :D

Then there is also this segfault with ImageMagick: JuliaLang/julia#13318
Because of all these issues, I started looking into polishing DevIL.jl a little more, as it is more lightweight and easier to setup compared to ImageMagick.
There I have problems with the Linux and OSX build. Any help would be appreciated:
https://travis-ci.org/JuliaGL/DevIL.jl
(Tests are successful, because they're still empty - builds are not.)
Here is an overview of DevIL's features: http://openil.sourceforge.net/features.php
It seems pretty capable and easy to integrate into FileIO.
Since I now allow for multiple IO libraries for the same formats, its also straight forward to have multiple libraries as a back up for some format.

Sorry for any misunderstandings. Everything is a little opaque to me and I don't have lots of time to do this :(

Best,
Simon

From the issues on Images.jl I conclude that this concerns: @timholy @lucasb-eyer @staticfloat @rsrock @jwmerrill

[SimonDanisch]

Whoops forgot to include the PR dealing with this:
#6

[SimonDanisch]

For DevIL its: JuliaGL/DevIL.jl#1

[lucasb-eyer]

I can only add useful info concerning two points:

1. MagickWandGenesis is a "wand-library init function" which you should always call once in the beginning of a process. See an example. We can see that it calls MagickCoreGenesis which sets up signal handlers and calls a boatload of other genesis functions, a lot of which initialize globals.
2. I think you have a typo about the env variables, because you have the same twice. This SO answer might be of use about the env-variables required for IM, also on OSX.

[SimonDanisch]

Thanks a lot!
This probably explains, why it segfaults in dump.c, as on windows ccall((:MagickWandGenesis,libwand), Void, ()) only gets called at precompilation.
Which is then wrong?

[lucasb-eyer]

Only at precompilation sounds wrong, yes.

[timholy]

Images has finally figured out how to do this correctly (took me long enough...):
https://github.com/timholy/Images.jl/blob/ecd35f185912489a94e57f22852d8be974b6a9dd/deps/build.jl#L18

It generates a deps.jl file which, on my system, is this:

# This is an auto-generated file; do not edit                                                                                        

# Pre-hooks                                                                                                                          

# Macro to load a library                                                                                                            
macro checked_lib(libname, path)                                                                                                     
    ((VERSION >= v"0.4.0-dev+3844" ? Base.Libdl.dlopen_e : Base.dlopen_e)(path) == C_NULL) && error("Unable to load \n\n$libname ($path)\n\nPlease re-run Pkg.build(package), and restart Julia.")                                                                        
    quote const $(esc(libname)) = $path end                                                                                          
end                                                                                                                                  

# Load dependencies
@checked_lib libwand "/usr/lib/x86_64-linux-gnu/libMagickWand.so"

# Load-hooks
ccall((:MagickWandGenesis,libwand), Void, ())

[timholy]

See also https://github.com/timholy/Images.jl/blob/ecd35f185912489a94e57f22852d8be974b6a9dd/src/ioformats/libmagickwand.jl#L26-L35. My versioninfo.jl file contains:

const libversion = v"6.7.7"

[SimonDanisch]

Isn't that exactly the issue, because MagickWandGenesis is only evaluated once while precompiling?
I tried the build.jl from Images master, which fails for me on windows...Debugging it right now.

[timholy]

Oh, crap. See JuliaImages/Images.jl@bf4b6bb
I'd better move it inside an __init__.

[timholy]

JuliaImages/Images.jl#366

[SimonDanisch]

I'm running into this:
JuliaImages/Images.jl#181
from what I read, the solution wasn't really solved in build.jl, but by manually installing imagemagick?
Fun fact, it worked for me the last couple of times, this just started recently. Not really sure what's going on, since I tried different versions of the build script.

[SimonDanisch]

Aha! ENV["MAGICK_CODER_MODULE_PATH"] needs to be set before ANYTHING. Which is not the case for all the build scripts I tried. I mean it looks like it, but still didn't work. I look into what needs to be done on the build script side.

[timholy]

Responding to your bullets at the top:

Depending on where included and what Julia version you have, init will be called at different points, which makes it hard to reason about

I don't understand this point; __init__ gets called on module initialization, right? It's kind of the last thing to run before loading the module is done.

onload hooks are not guaranteed to get executed

Right, they just get written to the deps.jl file. But if you include the deps.jl file into your source...

why and when exactly ccall((:MagickQueryConfigureOption, libwand), Ptr{UInt8}, (Ptr{UInt8},), "LIB_VERSION_NUMBER") segfaults on OSX wasn't really found out?!

Shouldn't happen anymore. These lines seem to solve the problem:
https://github.com/timholy/Images.jl/blob/ecd35f185912489a94e57f22852d8be974b6a9dd/deps/build.jl#L92-L95

I'm not sure why the version check is in a Module. Is this some hack for init ?

Not sure, but I bet I didn't want to risk borking an __init__ function in whatever code called include("build.jl").

I'm running into this: JuliaImages/Images.jl#181

Which aspect? It's a pretty multifaceted issue.

[timholy]

Yay, glad you figured it out!

[SimonDanisch]

onload hooks are not guaranteed to get executed

I meant, they're not guaranteed to be included into the deps.jl file.