Fix linking of shared libraries for local `pip install .`

[matthewfeickert]

So one thing that I've noticed is that the build instructions aren't fully transparent as it seems that a local pip install doesn't do a good job of actually building and installing a wheel as we can see in this demo that _fastjet_core/lib/python3.11/site-packages/_fastjet.so.0 is linked against the shared libraries in the source directory and not installed versions under site-packages!

$ docker run --rm -ti python:3.11 /bin/bash
root@305165ec2faa:/# python -m venv venv && . venv/bin/activate
(venv) root@305165ec2faa:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@305165ec2faa:/# apt-get update && apt-get install -y libboost-dev libmpfr-dev swig autoconf libtool
(venv) root@305165ec2faa:/# git clone --recursive https://github.com/scikit-hep/fastjet.git --branch fix/change-origin-par-expansion
(venv) root@305165ec2faa:/# cd fastjet/
(venv) root@305165ec2faa:/fastjet# python -m pip install --upgrade --verbose .
(venv) root@305165ec2faa:/fastjet# find /venv/lib/python3.11/site-packages/ -maxdepth 1 -iname "fastjet*"
/venv/lib/python3.11/site-packages/fastjet
/venv/lib/python3.11/site-packages/fastjet-3.4.2.0.dist-info
(venv) root@305165ec2faa:/fastjet# find /venv/lib/python3.11/site-packages/ -type f -iname "_fastjet.so.0"
/venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/_fastjet.so.0
(venv) root@305165ec2faa:/fastjet# ldd $(find /venv/lib/python3.11/site-packages/ -type f -iname "_fastjet.so.0")
	linux-vdso.so.1 (0x00007ffe17e7b000)
	libfastjet.so.0 => /fastjet/src/fastjet/_fastjet_core/lib/libfastjet.so.0 (0x00007f6e283a5000)
	libfastjettools.so.0 => /fastjet/src/fastjet/_fastjet_core/lib/libfastjettools.so.0 (0x00007f6e2835a000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f6e2813a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6e2805b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6e27e78000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6e27e58000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f6e27dd7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6e285eb000)
(venv) root@305165ec2faa:/fastjet# ls -l /fastjet/src/fastjet/_fastjet_core/lib/libfastjet.so.0
lrwxrwxrwx 1 root root 19 Feb 21 21:30 /fastjet/src/fastjet/_fastjet_core/lib/libfastjet.so.0 -> libfastjet.so.0.0.0
(venv) root@305165ec2faa:/fastjet# ls -l /fastjet/src/fastjet/_fastjet_core/lib/libfastjet.so.0.0.0  # this is the wrong file to link against
-rwxr-xr-x 1 root root 16193200 Feb 21 21:30 /fastjet/src/fastjet/_fastjet_core/lib/libfastjet.so.0.0.0
(venv) root@305165ec2faa:/fastjet# find /venv/ -type f -iname "libfastjet.so.0"
/venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/libfastjet.so.0
(venv) root@305165ec2faa:/fastjet# ls -l /venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/libfastjet.so.0  # should be linking against this
-rwxr-xr-x 1 root root 16193200 Feb 21 21:31 /venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/libfastjet.so.0

Though if we download a wheel from this PR which was built with cibuildwheel (https://github.com/scikit-hep/fastjet/actions/runs/7995766598/artifacts/1264455642) (here I'm going to copy it into the Docker container as resolving the full URL in advance is a bit annoying) we see that cibuildwheel is doing additional things that change the package layout as well (e.g. site-packages/fastjet.libs/ and linking to shared libraries in it):

(venv) root@305165ec2faa:/fastjet# deactivate
root@305165ec2faa:/fastjet# python -m venv /cibuildwheel-venv && . /cibuildwheel-venv/bin/activate
(cibuildwheel-venv) root@305165ec2faa:/fastjet# cd /tmp/
(cibuildwheel-venv) root@305165ec2faa:/tmp# unzip cibw-wheels-ubuntu-latest-311-auto64.zip
Archive:  cibw-wheels-ubuntu-latest-311-auto64.zip
  inflating: fastjet-3.4.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
(cibuildwheel-venv) root@305165ec2faa:/tmp# python -m pip --no-cache-dir install --upgrade ./fastjet-3.4.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
(cibuildwheel-venv) root@305165ec2faa:/tmp# find /cibuildwheel-venv/lib/python3.11/site-packages/ -maxdepth 1 -iname "fastjet*"  # we now have fastjet.libs too
/cibuildwheel-venv/lib/python3.11/site-packages/fastjet
/cibuildwheel-venv/lib/python3.11/site-packages/fastjet-3.4.2.0.dist-info
/cibuildwheel-venv/lib/python3.11/site-packages/fastjet.libs
(cibuildwheel-venv) root@305165ec2faa:/tmp# find /cibuildwheel-venv/lib/python3.11/site-packages/ -type f -iname "_fastjet.so.0"
/cibuildwheel-venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/_fastjet.so.0
(cibuildwheel-venv) root@305165ec2faa:/tmp# ldd $(find /cibuildwheel-venv/lib/python3.11/site-packages/ -type f -iname "_fastjet.so.0")
	linux-vdso.so.1 (0x00007fffe296a000)
	libfastjettools-ec77b8d8.so.0.0.0 => /cibuildwheel-venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/../../../../../fastjet.libs/libfastjettools-ec77b8d8.so.0.0.0 (0x00007fd49aab9000)
	libfastjet-8223762e.so.0.0.0 => /cibuildwheel-venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/../../../../../fastjet.libs/libfastjet-8223762e.so.0.0.0 (0x00007fd49a981000)
	libgmp-afec2dd4.so.10.2.0 => /cibuildwheel-venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/../../../../../fastjet.libs/libgmp-afec2dd4.so.10.2.0 (0x00007fd49a600000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd49a3e6000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd49a89a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd49a205000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd49a1e5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd49accb000)
(cibuildwheel-venv) root@305165ec2faa:/tmp#

This matters especially for conda-forge/staged-recipes#21052 as there things are built with python -m pip install . -vv and no cibuildwheel. So getting the linking correct is imporant.

@chrispap95 @lgray @jpivarski can you comment on this and where to go?

edit: Note that while the file structure is different than the cibuildwheel wheel, if you first use build to build a wheel and then install from it the linking is at least correct:

$ docker run --rm -ti python:3.11 /bin/bash
root@88ae83b3563a:/# python -m venv venv && . venv/bin/activate
(venv) root@88ae83b3563a:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@88ae83b3563a:/# apt-get update && apt-get install -y libboost-dev libmpfr-dev swig autoconf libtool
(venv) root@88ae83b3563a:/# git clone --recursive https://github.com/scikit-hep/fastjet.git --branch fix/change-origin-par-expansion
(venv) root@305165ec2faa:/# cd fastjet/
(venv) root@88ae83b3563a:/fastjet# python -m pip install build
(venv) root@88ae83b3563a:/fastjet# python -m build .
(venv) root@88ae83b3563a:/fastjet# python -m pip install --upgrade ./dist/fastjet-3.4.2.0-*.whl
(venv) root@88ae83b3563a:/fastjet# find /venv/lib/python3.11/site-packages/ -maxdepth 1 -iname "fastjet*"
/venv/lib/python3.11/site-packages/fastjet
/venv/lib/python3.11/site-packages/fastjet-3.4.2.0.dist-info
(venv) root@88ae83b3563a:/fastjet# find /venv/lib/python3.11/site-packages/ -type f -iname "_fastjet.so.0"
/venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/_fastjet.so.0
(venv) root@88ae83b3563a:/fastjet# ldd $(find /venv/lib/python3.11/site-packages/ -type f -iname "_fastjet.so.0")
	linux-vdso.so.1 (0x00007ffe9c51a000)
	libfastjet.so.0 => /venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/../../libfastjet.so.0 (0x00007fbe62379000)
	libfastjettools.so.0 => /venv/lib/python3.11/site-packages/fastjet/_fastjet_core/lib/python3.11/site-packages/../../libfastjettools.so.0 (0x00007fbe6232e000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fbe6210e000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbe6202f000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbe61e4c000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fbe61e2c000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fbe61dab000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbe625bf000)
(venv) root@88ae83b3563a:/fastjet#

Originally posted by @matthewfeickert in #277 (comment)