Rewrite configure, adding --with-so support

[TravisCardwell]

See well-typed/hs-bindgen#1923 for details.

The --with-so argument allows users to specify the path to a shared library file, used to determine extra-libraries and extra-lib-dirs on Linux systems. It is used to work around Cabal linking issues.

When --with-so is specified, extra-lib-dirs might be configured with two library directories: the directory of the specified file and the library directory determined using clang-config or LLVM_PATH (when different). This necessitates different logic for calculating LDFLAGS, so the organization of the script is changed to enable this. This reorganization puts related things together, IMHO making the script easier to understand/maintain. All internal variables are renamed to use a (wt_) prefix, as is documented as best practice when using autoconf, since substitutions are now defined close to where the variables are defined.

Note that extra-lib-dirs is now dynamic. It is determined by users when using --with-so and defaults to clang when --with-so is not used. We need to update the Nix implementation to work with this.

I found the clang_isBeforeInTranslationUnit check was broken. This commit reverts to using AC_CONFIG_HEADERS to fix it.

---

This PR also moves clang-bootstrap into separate project.

When in the same project, Cabal runs the configure script for both the library and clang-bootstrap, in parallel, even when clang-bootstrap is not built! This results in duplicate configure output. This can be avoided by moving clang-bootstrap into a separate project. This also allows us to remove the dev flag.

[TravisCardwell]

Note that this PR does not compare LLVM/Clang versions. Here are the versions involved:

• We determine the include directory and library directory for a specific version of LLVM/Clang using either llvm-config (via LLVM_CONFIG or PATH) or LLVM_PATH.
  • When using llvm-config, we call llvm-config --version to check that we can run llvm-config, but we do not do anything with that version.
  • When using LLVM_PATH, we do not know the version. One option is to run ${LLVM_PATH}/bin/clang${EXEEXT} --version.
• We build a program to get the LLVM/Clang version for the linked library, and we use this as the "compile time version" in the library. When --with-so is used, we link to a specified library (via the SONAME).
• Users may configure ghc-options: -optc=-DCLANG_VERSION=21.1 that specifies a MAJOR, MAJOR.MINOR, or MAJOR.MINOR.PATCH version, which we check at compile time using Template Haskell. Note that the configure script is not able to see this macro.

Another complication is that versions retrieved from llvm-config, clang, and the linked library are all version strings such as Ubuntu clang version 19.1.1 (1ubuntu1~24.04.2), not just the version number string. We cannot simply compare such strings in the configure script, even using AX_COMPARE_VERSION.

[TravisCardwell]

@edsko asked me to document configuration in the libclang-bindings.cabal file. Here is what I wrote, trying to include details without being too verbose... When reviewing this PR, perhaps we can discuss if this looks good, or if we should (also?) put such documentation elsewhere. Note that libclang-bindings does not have a manual like hs-bindgen, and such documentation will of course be in the hs-bindgen manual.

---

Configuring libclang-bindings

This package is configured using an autoconf configure script, which determines how the libclang shared library is linked.

In general, llvm-config is used to determine the include and library directories. If you have more than one version LLVM/Clang installed, you may configure your PATH to ensure that the llvm-config for the version that you want to use is found.

If llvm-config is not available, you may instead configure the LLVM_PATH environment variable to determine the include (LLVM_PATH/include) and library (LLVM_PATH/lib) directories.

The clang library is linked by default. On Linux, this library is resolved to a shared library named libclang.so, and the SONAME of that shared library determines the versioned filename that is actually linked. The configured library directory ensures that the correct version is linked. Haskell packages that (transitively) depend on libclang-bindings are also linked to the library. In some cases, Cabal links to the wrong version. To work around this issue, the configure script has a --with-so option. For example, a symbolic link named libclang_21_1_8.so to shared library /usr/lib/llvm21/lib/libclang.so.21.1.8 provides a versioned library name (libclang_21_1_8) that ensures that the correct version is linked. Note that the symbolic link must have a name with a lib prefix and .so suffix. A package that (transitively) depends on libclang-bindings can configure this in a cabal.project.local file as follows:

package libclang-bindings
  configure-options: --with-so=/absolute/path/to/libclang_21_1_8.so

Note that this symbolic link must be persistent, the path is cached in the Cabal store and used whenever a package that (transitively) depends on libclang-bindings is (re)built.

If you upgrade LLVM/Clang, you must reconfigure and rebuild all packages that depend on it. When using the --with-so option, you must also create a new symbolic link to the new shared library. Note that configure-options are included in the Cabal store hash for the package, so changing the --with-so configuration is sufficient to let Cabal know that a new build is needed.

[TravisCardwell]

I added release process documentation: rendered.

Feel free to adjust as needed.

BTW, we used package name clang for our configure script for some reason, perhaps leftover from before the package was renamed. I went ahead and fixed this, so that libclang-bindings is used consistently.

The only thing that is still inconsistent is the name of the repository: libclang. IMHO, we should have renamed that before our alpha release! Perhaps we should go ahead and fix this now anyway, before we start using Hackage. We could keep the repo, with a deprecation notice, and delete it after the first release. (Once we start using Hackage, hs-bindgen will no longer reference the repo directly.)

[TravisCardwell]

This is not an issue on Windows, as Windows DLL versions are not managed like they are on Linux. We always link to LIBCLANG.DLL, and it is up to the user to configure PATH so that the correct DLL is loaded at runtime.

I am not sure about macOS. Perhaps it is possible to have this issue when the following conditions are met:

• LLVM/Clang installed as part of the XCode toolchain has libraries in the system library path
• Another dependency (such as zlib) has libraries in the system library path
• The user wants to use a different version of LLVM/Clang that is installed somewhere else (via manual installation, Homebrew, etc.)
• The Cabal dependency resolver happens to order the other dependency library directories before the LLVM/Clang library directory

IMHO we should worry about this only if it comes up. In that case, I can easily add a (separate!) --with-dylib option. Note that .dylib symbolic link names are formatted differently than on Linux: the version is put before the .dylib extension.

• Linux: libclang.so.22.1.4
• macOS: libclang.22.1.4.dylib

[edsko]

@travis and I workshopped the docs a bit; it's going to leave in a README.md in the repo (which will also be part of the cabal file).

Configuring libclang-bindings

Using llvm-config

This package is configured using an autoconf configure script, which determines how the libclang shared library is linked.

In general, llvm-config is used to determine the include and library directories. If you have more than one version LLVM/Clang installed, you can configure your PATH to ensure that the llvm-config for the version that you want to use is found.

Using LLVM_PATH

If llvm-config is not available, you can instead configure the LLVM_PATH environment variable to determine the include (LLVM_PATH/include) and library (LLVM_PATH/lib) directories.

Note

On all systems we are aware of, llvm-config is available. Windows users should llvm as installed by MSYS, probably the default version that is bundled with ghc. The version that is installed with vscode, or the versions installed by official llvm releases, are not suitable (msvc is not supported by ghc).

Workout for Cabal problem when multiple versions are available

Background

By default, the library name that we link against is clang. The resolution from such a name to a specific file on your system is complicated. On Linux, the linker will first search its library path for libclang.so; the linker option that modifies this path is -L; ultimately, the various methods listed above for configurating or choosing an llvm version result in different choices of -L flags for the linker.

Typically, libclang.so is a symlink to a versioned file such as /usr/lib/libclang.so.21.1.8. This file has an embedded string called the SONAME, which instructs the linker to link against a different file instead; in the case of clang specifically, this SONAME is set up to omit the patch number:

FILE                                     SYMLINK TO           SONAME
/usr/lib/llvm-21/lib/libclang.so         libclang.so.21.1.8   --
/usr/lib/llvm-21/lib/libclang.so.21.1.8  --                   libclang.so.21.1
/usr/lib/llvm-21/lib/libclang.so.21.1    libclang.so.21.1.8   --

It is this third file (libclang.so.21.1) that we actually end up linking against; the resulting library will then continue to work even if the patch is upgraded (to 21.1.9, say). This is also (one reason) why at runtime the loader will need to resolve libclang.so.21.1 again to a specific file location.

Cabal problem

A problem arises when cabal later attempts to link to libclang.so again; for example, it will link any Haskell packages that (directly or indirectly) depend on libclang-bindings themselves also to libclang.so. This can be problematic, because this means that the linker must resolve libclang.so to a filename again, and the linker flags that affect this resolution can be affected by any (Haskell) package, which can result link the linker now finding a different .so file than it it did before. The result is that we are now linking against multiple different versions of libclang (one through libclang-bindings, another through hs-bindgen).

For example, suppose llvm-config gave us -L/usr/lib/llvm-21/lib, and the linker resolved libclang.so to libclang.so.21.1, as per the example above. It's possible that another Haskell package would result in -L/usr/lib passed to the linker, which might resolve as follows:

                             SYMLINK TO           SONAME
/usr/lib/libclang.so         libclang.so.22.1.4   --
/usr/lib/libclang.so.22.1.4  --                   libclang.so.22.1
/usr/lib/libclang.so.22.1    libclang.so.22.1.4   --

This could result in an executable linking against both libclang.so.21.1 and libclang.so.22.1.

Workaround

Since the linker flags arising from all Haskell packages that are linked together are combined into one single list, it can be really difficult (or indeed impossible) to influence this linker resolution behaviour, especially because we have no influence over the order of the flags in the list.

The workaround is to introduce a symlink to the desired version, which can unambiguously be resolved independent from any linker flags (by ensuring that the symlink name is unique). To support this workflow, the libclang-bindings configure script has a --with-so option. For example, you could create a directory ~/.cabal/sys-libs containing

                                                      SYMLINK TO 
/home/me/.cabal/sys-libs/libclang_workaround_21_1.so  /usr/lib/llvm-21/lib/libclang.so.21.1

Configuring using --with-so /home/me/.cabal/sys-libs/libclang_workaround_21_1.so will result in the linker looking for clang_workaround_21_1, which exists in exactly one location, and hence no danger of confusion (provided that there is only a single libclang.so.21.1 in the library path).

Note

This symbolic link must be persistent, the path is cached in the Cabal store and used whenever a package that (transitively) depends on libclang-bindings is (re)built.

Transitive dependencies

A package that (transitively) depends on libclang-bindings can configure this in a cabal.project.local file as follows:

package libclang-bindings
  configure-options: --with-so /home/me/.cabal/sys-libs/libclang_workaround_21_1.so

Upgrading LLVM/Clang

If you upgrade LLVM/Clang (other than patch releases), you must reconfigure and rebuild all packages that depend on it. When using the --with-so option, you must also create a new symbolic link to the new shared library. Note that configure-options are included in the Cabal store hash for the package, so changing the --with-so configuration is sufficient to let Cabal know that a new build is needed.