VTK 7 (py2 and py3) #453

Korijn · 2016-04-25T11:26:46Z

We built VTK 7 successfully using this recipe for the full range of Python versions (2.6, 2.7, 3.3, 3.4, 3.5) on x64 windows and linux architectures, so I reckon it should work on x86 and OS X as well.

The main reason for writing it is that the VTK package in the official anaconda channels is lagging behind and doesn't support python 3.x.

Looking forward to improving this recipe together with you all.

conda-forge-linter · 2016-04-25T11:26:51Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/vtk) and found some lint.

Here's what I've got...

For recipes/vtk:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

pelson · 2016-04-25T20:02:22Z

Hi @Korijn! Thanks for submitting!

The main reason for writing it is that the VTK package in the official anaconda channels is lagging behind and doesn't support python 3.x.

Awesome. Glad conda-forge can help!

Looking forward to improving this recipe together with you all.

👍

The CI looks a bit scary when it is all red like that, but hopefully we can iterate on it. To break down the problems by OS:

Linux:

-- Found X11: /usr/lib64/libX11.so
CMake Error at Rendering/OpenGL2/CMakeLists.txt:177 (message):
  X11_Xt_LIB could not be found.  Required for VTK X lib.

I'm afraid I know very little about X11, but I notice that libxt isn't part of the manylinux spec. For now, you can add a yum_requirements.txt containing libXt-devel in your recipe directory to get the necessary build dependency (see https://github.com/conda-forge/staged-recipes/blob/master/scripts/run_docker_build.sh#L45 for how this has been implemented).

He is currently away, but @jakirkham may be able to help in deciding whether we need to package libxt within conda-forge, or whether we simply make it a requirement for using the conda-forge distribution.

OSX:

+ mkdir build
+ cd build
++ uname
+ '[' Darwin == Linux ']'
+ make -j2
make: *** No targets specified and no makefile found.  Stop.
Command failed: /bin/bash -x -e /Users/travis/build/conda-forge/staged-recipes/recipes/vtk/build.sh

Windows. Took 43min on Py27, 1 hr (and timed out) on py34 x86.

The timeouts are problematic - we have an hour to build on item of the matrix, and unfortunately that is a hard limit put onto us from AppVeyor. I don't know that there is anything we can do about that in terms of either a) what we build or b) asking for longer.

The win x64 (py27):

C:\Miniconda-x64\conda-bld\work\VTK-7.0.0>REM ===== end generated header ===== 
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:29 (project):
  No CMAKE_C_COMPILER could be found.

CMake Error at CMakeLists.txt:29 (project):
  No CMAKE_CXX_COMPILER could be found.

-- Configuring incomplete, errors occurred!
See also "C:/Miniconda-x64/conda-bld/work/VTK-7.0.0/build/CMakeFiles/CMakeOutput.log".
See also "C:/Miniconda-x64/conda-bld/work/VTK-7.0.0/build/CMakeFiles/CMakeError.log".

Microsoft (R) Visual C++ Express Edition Version 9.0.30729.1.
Copyright (C) Microsoft Corp 2007. All rights reserved.

The following files were specified on the command line:

    VTK.sln

These files could not be found and will not be loaded. 
Microsoft (R) Visual C++ Express Edition Version 9.0.30729.1.
Copyright (C) Microsoft Corp 2007. All rights reserved.

The following files were specified on the command line:

    VTK.sln

These files could not be found and will not be loaded. 
Microsoft (R) Visual C++ Express Edition Version 9.0.30729.1.
Copyright (C) Microsoft Corp 2007. All rights reserved.

The following files were specified on the command line:

    VTK.sln

These files could not be found and will not be loaded. 
Microsoft (R) Visual C++ Express Edition Version 9.0.30729.1.
Copyright (C) Microsoft Corp 2007. All rights reserved.

The following files were specified on the command line:

    VTK.sln

These files could not be found and will not be loaded.Command failed: C:\Windows\system32\cmd.exe /c call bld.bat

Hope the failures aren't demoralizing - this is a really cool recipe to package and I'm super pleased you've submitted a PR to fix the current state of VTK on conda.`

Korijn · 2016-04-25T22:30:36Z

I can address most of these problems. About the x64 2.7 build; do you have a custom identifier for the c compiler? This recipe is tested and working on a Windows 7 VM with VS2008 Pro SP1 which has the x64 compiler built-in.

msarahan · 2016-04-25T22:50:37Z

I think the tricky part is that we have AppVeyor here. It uses VS 2008
Express, which has no x64 support. Instead, we use the VS Compiler for
Python 2.7, which is the same compiler, but no SDKs. That's what Conda
1.20.1 fixed primarily - the search order. Unfortunately, it brought in
other issues.

On Mon, Apr 25, 2016 at 5:30 PM Korijn van Golen [email protected]
wrote:

I can address most of these problems. About the x64 2.7 build; do you have
a custom identifier for the c compiler? This recipe is tested and working
on a Windows 7 VM with VS2008 Pro SP1 which has the x64 compiler built-in.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#453 (comment)

jakirkham · 2016-04-25T22:53:32Z

So, I guess it is time to bring back the VS 2008 64-bit workaround ( #251 ). That didn't last long.

Unfortunately, it brought in other issues.

I take it you are aware of this issue ( #448 ) then. Did that move upstream? Do we have any idea of what is causing it.

msarahan · 2016-04-25T23:19:15Z

I can't make sense of it right away. https://ci.appveyor.com/project/conda-forge/xonsh-feedstock/build/1.0.14/job/827pntj23q0xxt0p#L202 seems to be one issue - Windows 7.1 SDK not having the 64-bit compiler, possibly. I'm pretty sure @patricksnape tested that, though.

The other issue appears to be the same: https://ci.appveyor.com/project/conda-forge/staged-recipes/build/1.0.1375/job/jq2rrc7uwjb8iehw#L213

This should be a quick fix once we figure out what the correct value should be. I can help push a quick release.

PS: @patricksnape it is better to set variables like:

set "VARNAME=value"

rather than set VARNAME="value"

https://ci.appveyor.com/project/conda-forge/xonsh-feedstock/build/1.0.14/job/827pntj23q0xxt0p#L198

Both treat spaces well, but the former doesn't have quotes in the end variable value, and is easier to deal with.

Sorry I didn't catch this on the PR.

jakirkham · 2016-04-25T23:27:14Z

Might need freeglut. Take a look at this comment ( #285 (comment) ).

jakirkham · 2016-04-26T00:27:47Z

To help preserve our sanity, I moved the weird Windows build issue into this issue ( conda/conda-build#895 ). IMHO we have at least confirmed there is a conda-build bug at this point. Maybe we can focus the discussion there. Tracking it over multiple PRs, issues, and feedstocks is getting a little out of hand.

jakirkham · 2016-04-26T00:29:43Z

Also, I'm wrong about needing that workaround. I forgot the new issue is restricted to Windows 64-bit Python 3.4. So, we shouldn't need the old VS 2008 64-bit workaround as the downgrade of conda-build is not needed on Python 2.7.

jakirkham · 2016-04-26T01:13:21Z

Ignore the above. Also, don't dig any deeper. I know the cause.

jakirkham · 2016-04-26T01:21:58Z

So, this PR ( #455 ) should fix things for you here at staged-recipes. Want to give it a try?

pelson · 2016-04-28T10:18:24Z

So, this PR ( #455 ) should fix things for you here at staged-recipes. Want to give it a try?

That is, just rebase this PR against upstream/master and get #455 pulled in.

conda-forge-linter · 2016-04-29T08:02:49Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/vtk) and found it was in an excellent condition.

Korijn · 2016-04-29T08:04:09Z

Changes:

Rebased against upstream/master
Added sha1 checksum
Dropped mesa requirement in favor of freeglut
Added yum_requirements.txt for libXi-devel
Added .gitattributes to guard newlines in build.sh

I'm not too sure what has to be done for OS X; I have no machine to test on.

jakirkham · 2016-04-29T11:36:32Z

recipes/vtk/bld.bat

+cmake --build . --target INSTALL --config %BUILD_CONFIG%
+if errorlevel 1 exit 1
+
+exit /b 0


Please add a terminal newline.

jakirkham · 2016-04-29T11:39:47Z

Is it possible to disable X11 support on Mac? If so, that's what I would do here.

Korijn · 2016-04-29T11:54:48Z

I'm clueless about X11 and also about Mac. There's probably a cmake flag to disable it.

jakirkham · 2016-04-29T12:04:42Z

There is. Just set this on Mac only -DVTK_USE_X:BOOL=OFF.

jakirkham · 2016-04-29T12:06:35Z

The thing with Mac is they used to have X11 support baked in, but they decided to drop it. Though one can still install it and Apple still supports this open source effort. It is just something we can't expect to find on the user's system. Maybe one day we can add a package to install it in a conda environment. Until then, disabling is the right way to go.

jakirkham · 2016-04-29T12:11:26Z

So, I don't think he mentioned it, but @msarahan had a recipe for this that was in the process of being added. Not to speak for him, but you seem motivated to get this in and I think we are happy to see someone pushing this forward so please continue (plus I'm sure @msarahan has tons on his plate and doesn't mind someone taking this off it 😉). Though you might be able to pick up a few useful things from his recipe, which can be found in this PR ( #285 ).

jakirkham · 2016-04-29T12:16:49Z

@stuarteberg, given your use of VTK, maybe you have some useful feedback here. Please feel free to share.

jakirkham · 2016-04-29T12:19:36Z

recipes/vtk/build.sh

+        -DPYTHON_LIBRARY:FILEPATH=$library_file_path
+fi
+
+make -j${CPU_COUNT}


Could we add ctest. Also, you may need this.

We have been a little cautious about using CPU_COUNT as the CIs don't always report this correctly. For instance, they may report the number of CPUs for the node you are on instead of noting how many of these you get. Could we try removing this? If the build time is too long, we can play with this more.

stuarteberg · 2016-04-29T12:31:48Z

given your use of VTK, maybe you have some useful feedback here.

I don't have much to offer. I use a pretty old version of VTK. Recipe is here:
https://github.com/ilastik/ilastik-build-conda/blob/master/vtk/build.sh

One possible suggestion: Our recipe sets the -DVTK_USE_SYSTEM_ZLIB/PNG/JPEG/TIFF/LIBXML flags, just to make sure VTK is using the same version of those libraries that the rest of our conda packages are using. Not sure if that's a big deal.

ivoflipse · 2016-06-23T14:37:26Z

recipes/vtk/bld.bat

+REM tell cmake where Python is
+set PYTHON_LIBRARY=%PREFIX%\libs\python%PY_VER:~0,1%%PY_VER:~2,1%.lib
+
+cmake .. -G"%GENERATOR_NAME%" ^


We could use NMake here too, if it fails on 2.7

Korijn · 2016-06-23T16:48:45Z

We've rebased again, processed the comments, and cleaned up using the things we learned from the u3d recipe. Adding tests will be challenging and probably cause timeouts on the builds... but we'll see!

ivoflipse · 2016-06-23T17:00:53Z

@thewtex or @dlonie Would any of you guys at Kitware mind having a look at our build scripts to see what we can do to get our VTK recipe working?

Korijn · 2016-06-24T10:08:46Z

The good news:

Windows and Linux build scripts work for Python 2k and 3k.
I've built and installed both locally and run some test scripts to start a renderer; works!
OS X seems to be a simple fix, we're currently pointing to the python library incorrectly.

The bad news:

Timeouts are a problem, even with all optional components disabled:
- Setting BUILD_TESTING:BOOL=ON triggers the downloading of external test data which increases build time with about 1 hour.
- The Windows build takes about 2,5 hours per configuration.
- The Linux build takes about 50 minutes per configuration.
- As we want to build for x86/x64, py2.7, py3.4 and py3.5 (10 configurations in total), there is no way we'll be able to stay under the timeout limits of 1 hour on Windows (AppVeyor), 2 hours on Linux (CircleCI) and 50 minutes on OS X (Travis CI).

I'm currently working under a tight schedule and need to get at least the Linux packages out there. I would like to propose the following:

skip Windows: not even 1 configuration will complete before the build timeout
skip OS X: we don't have access to OS X systems to work on the build script
restrict Linux to x64 py2.7 and x64 py.35: it should complete for these 2 important configurations before the build timeout

We should then get green light on the CI and be able to merge. We can then open a new PR on the feedstock where we'll work on:

the build timeout issues on CI so we can get all configurations out there, also for Windows
proper testing
possibly supporting features (e.g. Matplotlib) that e.g. Mayavi needs.

@jakirkham, thoughts?

ocefpaf · 2016-06-24T10:56:24Z

I'm currently working under a tight schedule and need to get at least the Linux packages out there. I would like to propose the following:

skip Windows: not even 1 configuration will complete before the build timeout

skip OS X: we don't have access to OS X systems to work on the build script

restrict Linux to x64 py2.7 and x64 py.35: it should complete for these 2 important configurations before the build timeout

@Korijn that is a sound strategy (I did something similar with gdal some time ago).

👍

Korijn · 2016-06-24T11:00:49Z

@ocefpaf cool. I commited the proposed changes, let's see if CI turns all-green.

ocefpaf · 2016-06-24T11:01:38Z

Fingers crossed!

Korijn · 2016-06-24T11:26:56Z

recipes/vtk/build.sh

+    -DCMAKE_BUILD_TYPE=$BUILD_CONFIG \
+    -DCMAKE_INSTALL_PREFIX:PATH="${PREFIX}" \
+    -DCMAKE_INSTALL_RPATH:PATH="${PREFIX}/lib" \
+    -DINSTALL_PKGCONFIG_DIR:PATH=$PKG_CONFIG_PATH \


INSTALL_PKGCONFIG_DIR is unused according to a CMake warning. will remove that in future PR.

Korijn · 2016-06-24T12:59:18Z

@ocefpaf can we merge? I'll open a PR on the feedstock directly afterwards.

ocefpaf · 2016-06-24T13:01:54Z

Thanks @Korijn! I know the pain to fight against the CI limitations. Thanks for leading this effort!

jakirkham · 2016-06-24T13:27:56Z

Timeouts are a problem, even with all optional components disabled:

Setting BUILD_TESTING:BOOL=ON triggers the downloading of external test data which increases build time with about 1 hour.

The Windows build takes about 2,5 hours per configuration.

The Linux build takes about 50 minutes per configuration.

As we want to build for x86/x64, py2.7, py3.4 and py3.5 (10 configurations in total), there is no way we'll be able to stay under the timeout limits of 1 hour on Windows (AppVeyor), 2 hours on Linux (CircleCI) and 50 minutes on OS X (Travis CI).

We had the same rough experience adding OpenCV, @Korijn. This sounds totally sensible to me. I'm glad this was added as is.

Also, good job getting this to build. 🎉

thewtex · 2016-06-24T14:55:25Z

CC: @demarle @cryos @vibraphone

Korijn · 2016-06-24T15:10:11Z

This work is continued here:

conda-forge/vtk-feedstock#1

Korijn mentioned this pull request Apr 25, 2016

add mayavi and vtk #285

Closed

jakirkham mentioned this pull request Apr 26, 2016

Restrict conda-build downgrade to Python 3.4 64-bit. #455

Merged

jakirkham reviewed Apr 29, 2016
View reviewed changes

EelcoHoogendoorn added 4 commits June 23, 2016 16:14

whitespace

aa972ca

removed gcc compiler spec

8ba601b

build cleanup

cb815ce

review comments

ea796ff

ivoflipse reviewed Jun 23, 2016
View reviewed changes

EelcoHoogendoorn and others added 3 commits June 23, 2016 16:48

case sensitive?

f1b22b2

Trying to use NMake instead

f0c23f2

Make install for osx and linux

0b0ce99

Korijn and others added 3 commits June 24, 2016 09:01

Harmonized linux and windows recipes

9c692d1

Corrected cmake option types

8566cd7

%PY_VER% -> ${PY_VER} for build.sh, duh

873c312

Korijn added 3 commits June 24, 2016 12:12

Skip windows and OS X

0097caa

Added skips

4871137

Added test script for manual testing of the package

8f50d1a

terminal newline

859f744

Korijn reviewed Jun 24, 2016
View reviewed changes

ocefpaf merged commit e8c8e9f into conda-forge:master Jun 24, 2016

Korijn deleted the vtk branch June 29, 2016 08:45

Korijn mentioned this pull request May 25, 2017

VTK 7.1.0 conda-forge is incorrectly reporting attribute error conda-forge/vtk-feedstock#29

Closed

VTK 7 (py2 and py3) #453

VTK 7 (py2 and py3) #453

Conversation

Korijn commented Apr 25, 2016

conda-forge-linter commented Apr 25, 2016

pelson commented Apr 25, 2016

Korijn commented Apr 25, 2016

msarahan commented Apr 25, 2016

jakirkham commented Apr 25, 2016

msarahan commented Apr 25, 2016 • edited Loading

jakirkham commented Apr 25, 2016

jakirkham commented Apr 26, 2016

jakirkham commented Apr 26, 2016

jakirkham commented Apr 26, 2016

jakirkham commented Apr 26, 2016

pelson commented Apr 28, 2016

conda-forge-linter commented Apr 29, 2016

Korijn commented Apr 29, 2016 • edited Loading

jakirkham Apr 29, 2016

Choose a reason for hiding this comment

jakirkham commented Apr 29, 2016

Korijn commented Apr 29, 2016

jakirkham commented Apr 29, 2016

jakirkham commented Apr 29, 2016

jakirkham commented Apr 29, 2016

jakirkham commented Apr 29, 2016

jakirkham Apr 29, 2016

Choose a reason for hiding this comment

jakirkham Apr 29, 2016

Choose a reason for hiding this comment

stuarteberg commented Apr 29, 2016

ivoflipse Jun 23, 2016

Choose a reason for hiding this comment

Korijn commented Jun 23, 2016

ivoflipse commented Jun 23, 2016

Korijn commented Jun 24, 2016 • edited Loading

ocefpaf commented Jun 24, 2016

Korijn commented Jun 24, 2016

ocefpaf commented Jun 24, 2016 • edited Loading

Korijn Jun 24, 2016

Choose a reason for hiding this comment

Korijn commented Jun 24, 2016

ocefpaf commented Jun 24, 2016

jakirkham commented Jun 24, 2016

thewtex commented Jun 24, 2016

Korijn commented Jun 24, 2016

msarahan commented Apr 25, 2016 •

edited

Loading

Korijn commented Apr 29, 2016 •

edited

Loading

Korijn commented Jun 24, 2016 •

edited

Loading

ocefpaf commented Jun 24, 2016 •

edited

Loading