Skip to content

build: unify static/dynamic linking for conda/package manager compatibility#668

Merged
sfchen merged 4 commits into
OpenGene:masterfrom
KimYannn:fix/makefile-conda-compat
Mar 19, 2026
Merged

build: unify static/dynamic linking for conda/package manager compatibility#668
sfchen merged 4 commits into
OpenGene:masterfrom
KimYannn:fix/makefile-conda-compat

Conversation

@KimYannn
Copy link
Copy Markdown
Member

Summary

  • Replace platform-specific linking logic (Linux -static vs macOS .a fallback) with a single unified approach
  • If .a static archives exist in LIBRARY_DIRS, link statically (unchanged behavior)
  • If only shared libs are available, fallback to -l dynamic linking (new)

Motivation

The current Linux Makefile uses -static which requires a static glibc. This breaks builds in package managers like conda/bioconda where only shared system libraries are available (see bioconda/bioconda-recipes#63664).

Behavior

Scenario Before After
User builds with .a libs installed Static link Static link (unchanged)
make LIBRARY_DIRS=/path/to/static/libs Static link Static link (unchanged)
conda/apt (shared libs only) Fails (-static needs static glibc) Dynamic link (works)
macOS with Homebrew Static preferred, fallback dynamic Same (unchanged)

Test plan

  • Build on Linux with static libs installed (same as before)
  • Build on macOS (same as before)
  • Build in conda environment (now works without LD_FLAGS override)

🤖 Generated with Claude Code

@KimYannn KimYannn force-pushed the fix/makefile-conda-compat branch from 04649a1 to 21cffeb Compare March 19, 2026 12:25
1. Add pkg-config auto-detection for libhwy, libisal, libdeflate.
   When installed via package managers (conda, brew, apt), their
   include/lib paths are automatically picked up without needing
   to pass INCLUDE_DIRS/LIBRARY_DIRS manually.

2. Unify linking: on Linux, keep -static when all .a archives exist
   (default). When some .a are missing (conda/apt shared-only),
   fallback to .a + -l dynamic. macOS unchanged.

With this change, bioconda build.sh can simply call:
  make CXX=$CXX -j$CPU_COUNT
  make install PREFIX=$PREFIX

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@KimYannn KimYannn force-pushed the fix/makefile-conda-compat branch from 21cffeb to 39e3295 Compare March 19, 2026 14:26
KimYannn and others added 2 commits March 19, 2026 22:28
Remove bundled igzip_lib.h and libdeflate.h from src/. Use only
angle-bracket includes that resolve via pkg-config or -I flags:
  - <isa-l/igzip_lib.h>  (was __has_include fallback to bundled)
  - <libdeflate.h>        (was __has_include fallback to bundled)
  - <hwy/foreach_target.h> (was quote include)
  - <hwy/highway.h>        (was quote include)

Libraries must now be installed system-wide or discoverable via
pkg-config / INCLUDE_DIRS before building.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Install isa-l and highway to /usr/local so pkg-config finds them
automatically. No more manual INCLUDE_DIRS/LIBRARY_DIRS in make.
Also bump isa-l to 2.31.1.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@KimYannn
Copy link
Copy Markdown
Member Author

@sfchen Hi, this PR is motivated by the bioconda packaging effort for fastp 1.2.0 (see bioconda/bioconda-recipes#63676 and #63664).

Problem

The current Makefile uses -static on Linux, which requires a static glibc — unavailable in conda and most package manager environments. This causes the bioconda build to fail.

What this PR does

  1. Unify static/dynamic linking: on Linux, if all .a archives are found, the fully static binary is built as before. If some .a are missing (e.g. conda/apt where only shared libs exist), it gracefully falls back to linking .a where available + -l dynamic for the rest. macOS behavior is unchanged.

  2. Auto-detect deps via pkg-config: the Makefile now queries pkg-config for libhwy, libisal, and libdeflate paths. When libraries are installed via conda, brew, or apt, their include/lib paths are picked up automatically — no need to pass INCLUDE_DIRS/LIBRARY_DIRS manually.

  3. Remove bundled headers: src/igzip_lib.h and src/libdeflate.h are removed. All three libraries now use standard angle-bracket includes (<isa-l/igzip_lib.h>, <libdeflate.h>, <hwy/...>) resolved via pkg-config or -I flags.

Behavior matrix

Scenario Before After
User builds with static libs installed Static binary ✅ Static binary ✅ (unchanged)
make LIBRARY_DIRS=/path/to/libs Static binary ✅ Static binary ✅ (unchanged)
conda / apt (shared libs only) Fails Dynamic link ✅
macOS + Homebrew Static preferred Same (unchanged)
bioconda build.sh Needs LD_FLAGS override Just make

With this change, the bioconda recipe simplifies to:

c src/readpool.cpp -o obj/readpool.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/seprocessor.cpp -o obj/seprocessor.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/unittest.cpp -o obj/unittest.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/threadconfig.cpp -o obj/threadconfig.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/umiprocessor.cpp -o obj/umiprocessor.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/stats.cpp -o obj/stats.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/simd.cpp -o obj/simd.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/read.cpp -o obj/read.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/processor.cpp -o obj/processor.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/polyx.cpp -o obj/polyx.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/peprocessor.cpp -o obj/peprocessor.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/overlapanalysis.cpp -o obj/overlapanalysis.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/options.cpp -o obj/options.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/nucleotidetree.cpp -o obj/nucleotidetree.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/main.cpp -o obj/main.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/jsonreporter.cpp -o obj/jsonreporter.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/htmlreporter.cpp -o obj/htmlreporter.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/filterresult.cpp -o obj/filterresult.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/filter.cpp -o obj/filter.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/fastqreader.cpp -o obj/fastqreader.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/evaluator.cpp -o obj/evaluator.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/duplicate.cpp -o obj/duplicate.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/basecorrector.cpp -o obj/basecorrector.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/adaptertrimmer.cpp -o obj/adaptertrimmer.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/writer.cpp -o obj/writer.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
c src/writerthread.cpp -o obj/writerthread.o -std=c++11 -pthread -g -O3 -MD -MP -I. -I./inc -O3 -std=c++14
install fastp /bin/fastp

No vendored sources, no patches, no workarounds.

apt's libdeflate-dev only provides shared lib. Build from source
with -DLIBDEFLATE_BUILD_STATIC_LIB=ON so all three deps (isa-l,
libdeflate, highway) are statically linked on Linux.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sfchen sfchen merged commit 491d3cb into OpenGene:master Mar 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants