Wheels for release do not build successfully

[kbonney]

Summary
(For users) How does this affect me?
When we released 1.3.0, the wheels did not include any EPANET binaries and would cause any functionality that depended on EPANET to fail (#471 #466 etc). This release has since been yanked, so that 1.2.0 is now the latest release available on pypi/conda. Installing WNTR from these sources should work as normal. Currently, the only impact to users is that the release of 1.3.0 (which includes the MSX capabilities) will be delayed until this issue is resolved. We apologize for any inconvenience.

What is happening?
When attempting a tagged release of WNTR, the release.yml workflow triggers and uses cibuildwheel to prepare generic wheels across OS and architectures. Currently, cibuildwheel fails to build wheels for MacOS (13 and latest) and linux (manylinux).

Why is this happening?
Since the last release of WNTR, we have updated the EPANET binaries to include the MSX extension. MSX depends on libomp and this appears to be the source of the issues on the Mac wheels and potentially the manylinux wheels as well. The binaries are built with an older version of libomp (14.X.X). I suspect that the old version is causing the build issues. The binaries are also built with a version of glibc which surpasses the minimum required by common manylinux standards (we have been using manylinux2014 in previous builds). This causes the manylinux build to fail.

How can it be fixed?
Changing the versions of libomp and glibc that the EPANET binaries are built with could solve the build issue for Mac and linux. We have tried to force install libomp 14 during wheel building to address this issue, but it does not resolve the issue. The forced installation does allow us to build native wheels for the GH action runners such as macOS-13 and macOS-latest in build_tests.yml.

For linux, some issues appear to be resolved by using a manylinux image which is compatible with the glibc version used by EPANET binaries. However, other issues on linux still persist and look similar to the errors that show up on the mac builds.

TODO

• Try to build EPANET/EPANET-MSX with glibc 2.17 or earlier. If this is not possible, we may be able to get away with using a different manylinux image but it will reduce the compatibility of our wheel.
• Try to build EPANET/EPANET-MSX with a later version of libomp. I am not sure what minimum version is needed, but we can start by trying the most recent version that EPANET can be built with.

Relevant Logs

MacOS wheel failure:

Log snippet

• delocate-wheel --require-archs arm64 -w /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/repaired_wheel -v /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/built_wheel/wntr-1.3.1rc0-cp39-cp39-macosx_14_0_arm64.whl
  ERROR:delocate.libsana:
  @rpath/libepanet2.dylib not found:
  Needed by: /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-x64/libepanetmsx.dylib
  Search path:

  ERROR:delocate.libsana:
  /usr/local/opt/libomp/lib/libomp.dylib not found:
  Needed by: /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-x64/libepanetmsx.dylib
  ERROR:delocate.libsana:
  @rpath/libepanet2.dylib not found:
  Needed by: /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-x64/libepanetmsx.dylib
  Search path:

  ERROR:delocate.libsana:@rpath/libepanet2.dylib not found, requested by /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-x64/libepanetmsx.dylib
  ERROR:delocate.libsana:
  /usr/local/opt/libomp/lib/libomp.dylib not found:
  Needed by: /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-x64/libepanetmsx.dylib
  ERROR:delocate.libsana:/usr/local/opt/libomp/lib/libomp.dylib not found, requested by /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-x64/libepanetmsx.dylib
  ERROR:delocate.libsana:
  @rpath/libepanet2.dylib not found:
  Needed by: /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-arm/libepanetmsx.dylib
  Search path:

  ERROR:delocate.libsana:
  @rpath/libepanet2.dylib not found:
  Needed by: /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-arm/libepanetmsx.dylib
  Search path:

  ERROR:delocate.libsana:@rpath/libepanet2.dylib not found, requested by /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/tmpg0ymcvsq/wheel/wntr/epanet/libepanet/darwin-arm/libepanetmsx.dylib
  Fixing: /private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/built_wheel/wntr-1.3.1rc0-cp39-cp39-macosx_14_0_arm64.whl
  Traceback (most recent call last):
  File "/private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/build/venv/bin/delocate-wheel", line 8, in
  sys.exit(main())
  File "/private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/cmd/delocate_wheel.py", line 116, in main
  copied = delocate_wheel(
  File "/private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/delocating.py", line 1057, in delocate_wheel
  copied_libs = delocate_path(
  File "/private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/delocating.py", line 511, in delocate_path
  lib_dict = tree_libs_from_directory(
  File "/private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/libsana.py", line 377, in tree_libs_from_directory
  return _tree_libs_from_libraries(
  File "/private/var/folders/2s/h6hvv9ps03xgz_krkkstvq_r0000gn/T/cibw-run-8ng6jzoo/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/libsana.py", line 320, in _tree_libs_from_libraries
  raise DelocationError("Could not find all dependencies.")
  delocate.libsana.DelocationError: Could not find all dependencies.

Linux wheel failure:

Log snippet

+ sh -c 'auditwheel show /tmp/cibuildwheel/built_wheel/wntr-1.3.1rc0-cp39-cp39-linux_x86_64.whl && auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/wntr-1.3.1rc0-cp39-cp39-linux_x86_64.whl'

wntr-1.3.1rc0-cp39-cp39-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libm.so.6 with versions
{'GLIBC_2.2.5', 'GLIBC_2.29'}, libc.so.6 with versions {'GLIBC_2.2.5',
'GLIBC_2.34', 'GLIBC_2.14', 'GLIBC_2.6', 'GLIBC_2.3', 'GLIBC_2.17',
'GLIBC_2.7', 'GLIBC_2.4'}, libgomp.so.1 with versions {'GOMP_4.0',
'GOMP_1.0', 'OMP_1.0'}, libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.3.4', 'GLIBC_2.2.5'},
libstdc++.so.6 with versions {'GLIBCXX_3.4', 'CXXABI_1.3'}

This constrains the platform tag to "manylinux_2_34_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.
INFO:auditwheel.main_repair:Repairing wntr-1.3.1rc0-cp39-cp39-linux_x86_64.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot repair "/tmp/cibuildwheel/built_wheel/wntr-1.3.1rc0-cp39-cp39-linux_x86_64.whl" to "manylinux2014_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

NOTE: Shared object (.so) files found in this project.

These files might be built against the wrong OS, causing problems with
auditwheel. If possible, run cibuildwheel in a clean checkout.

If you're using Cython and have previously done an in-place build,
remove those build files (*.so and *.c) before starting cibuildwheel.

setuptools uses the build/ folder to store its build cache. It
may be necessary to remove those build files (*.so and *.o) before
starting cibuildwheel.

Files that belong to a virtual environment are probably not an issue
unless you used a custom command telling cibuildwheel to activate it.

Files detected:
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanet2.so
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanet20.so
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanetmsx.so
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanet22.so

Linux wheel failure on updated manylinux image (quay.io/pypa/manylinux_2_34_x86_64:latest):

Log snippet

+ sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/wntr-1.3.1rc0-cp39-cp39-linux_x86_64.whl' INFO:auditwheel.main_repair:Repairing wntr-1.3.1rc0-cp39-cp39-linux_x86_64.whl Traceback (most recent call last): File "/usr/local/bin/auditwheel", line 8, in sys.exit(main()) ^^^^^^ File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.12/site-packages/auditwheel/main.py", line 54, in main return args.func(args, p) ^^^^^^^^^^^^^^^^^^ File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.12/site-packages/auditwheel/main_repair.py", line 178, in execute out_wheel = repair_wheel( ^^^^^^^^^^^^^ File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.12/site-packages/auditwheel/repair.py", line 82, in repair_wheel raise ValueError(msg)

ValueError: Cannot repair wheel, because required library "libepanet2.so" could not be located

NOTE: Shared object (.so) files found in this project.

These files might be built against the wrong OS, causing problems with
auditwheel. If possible, run cibuildwheel in a clean checkout.

If you're using Cython and have previously done an in-place build,
remove those build files (*.so and *.c) before starting cibuildwheel.

setuptools uses the build/ folder to store its build cache. It
may be necessary to remove those build files (*.so and *.o) before
starting cibuildwheel.

Files that belong to a virtual environment are probably not an issue
unless you used a custom command telling cibuildwheel to activate it.

Files detected:
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanet2.so
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanet20.so
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanetmsx.so
  /home/runner/work/WNTR/WNTR/wntr/epanet/libepanet/linux-x64/libepanet22.so

[kbonney]

MSX now should be built with the latest libomp, see USEPA/EPANETMSX#18.

[kbonney]

To get around the manylinux issues, we are currently releasing our linux wheels without repairing them. This means that they do not technically fulfill the manylinux standard. We have tested these wheels on Ubuntu 22, 24, and on Google Colab and they provide a successful installation of WNTR, despite being unrepaired. However, other linux architectures may not work with our current wheels.