Skip to content

Commit fd7189c

Browse files
Satish Kumarmeta-codesync[bot]
Satish Kumar
authored and
meta-codesync[bot]
committed
Build dependencies from source for Python extension builds
Summary: Python extension wheels (.whl) that link against C++ libraries require those libraries to be compiled in a specific way. System packages installed via apt/yum lack this compilation. # The Problem Python extensions are shared libraries (.so). Static C++ libraries linked into them must be compiled with "position-independent code" (-fPIC flag). System packages skip this flag because it has a small performance cost and normal use doesn't need it. # The Solution Create manifest variants (named `*-python`) that skip system packages and build everything from source with the required flags. # Why Not Modify Existing Manifests? That would force all OSS builds to compile from source, even when system packages would work fine. The `-python` variants keep fast system package installs for other builds while Python extension builds get correct compilation. The fbthrift-python manifest pulls in these dependency variants to build thrift-python wheels for OSS distribution. # Workflow File (`getdeps_python_linux.yml`) `getdeps.py generate-github-actions` generates this workflow. Hand modifications add pre-build steps that install auditwheel and patchelf, which bundle shared libraries into the wheel. Hand modifications mean the generated comment no longer applies. The manifest route for these dependencies (python-auditwheel, python-patchelf) became a rabbit hole - auditwheel requires specific Python packaging infrastructure and patchelf has its own build dependencies. Pip shell commands proved simpler and more maintainable than manifest chains. Reviewed By: martindemello Differential Revision: D90833579 fbshipit-source-id: e7510d665f22ac2a869aaed63e240a09449b9f39
1 parent ade49be commit fd7189c

15 files changed

Lines changed: 686 additions & 0 deletions

‎build/fbcode_builder/manifests/boost-python‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
[manifest]
2+
name = boost-python
3+
4+
[download.not(os=windows)]
5+
url = https://archives.boost.io/release/1.83.0/source/boost_1_83_0.tar.gz
6+
sha256 = c0685b68dd44cc46574cce86c4e17c0f611b15e195be9848dfd0769a0a207628
7+
8+
[download.os=windows]
9+
url = https://archives.boost.io/release/1.83.0/source/boost_1_83_0.zip
10+
sha256 = c86bd9d9eef795b4b0d3802279419fde5221922805b073b9bd822edecb1ca28e
11+
12+
[preinstalled.env]
13+
# Here we list the acceptable versions that cmake needs a hint to find
14+
BOOST_ROOT_1_69_0
15+
BOOST_ROOT_1_83_0
16+
17+
[homebrew]
18+
boost
19+
# Boost cmake detection on homebrew adds this as requirement: https://github.com/Homebrew/homebrew-core/issues/67427#issuecomment-754187345
20+
icu4c
21+
22+
[pps]
23+
boost
24+
25+
[rpms.all(distro=centos_stream,distro_vers=8)]
26+
boost169
27+
boost169-math
28+
boost169-test
29+
boost169-fiber
30+
boost169-graph
31+
boost169-log
32+
boost169-openmpi
33+
boost169-timer
34+
boost169-chrono
35+
boost169-locale
36+
boost169-thread
37+
boost169-atomic
38+
boost169-random
39+
boost169-static
40+
boost169-contract
41+
boost169-date-time
42+
boost169-iostreams
43+
boost169-container
44+
boost169-coroutine
45+
boost169-filesystem
46+
boost169-system
47+
boost169-stacktrace
48+
boost169-regex
49+
boost169-devel
50+
boost169-context
51+
boost169-python3-devel
52+
boost169-type_erasure
53+
boost169-wave
54+
boost169-python3
55+
boost169-serialization
56+
boost169-program-options
57+
58+
[rpms.distro=fedora]
59+
boost-devel
60+
boost-static
61+
62+
[build]
63+
builder = boost
64+
job_weight_mib = 512
65+
patchfile = boost_1_83_0.patch
66+
67+
[build.not(os=linux)]
68+
builder = nop
69+
70+
[b2.args]
71+
--with-atomic
72+
--with-chrono
73+
--with-container
74+
--with-context
75+
--with-contract
76+
--with-coroutine
77+
--with-date_time
78+
--with-exception
79+
--with-fiber
80+
--with-filesystem
81+
--with-graph
82+
--with-graph_parallel
83+
--with-iostreams
84+
--with-locale
85+
--with-log
86+
--with-math
87+
--with-mpi
88+
--with-program_options
89+
--with-python
90+
--with-random
91+
--with-regex
92+
--with-serialization
93+
--with-stacktrace
94+
--with-system
95+
--with-test
96+
--with-thread
97+
--with-timer
98+
--with-type_erasure
99+
100+
[bootstrap.args.os=darwin]
101+
# Not really gcc, but CI puts a broken clang in the PATH, and saying gcc
102+
# here selects the correct one from Xcode.
103+
--with-toolset=gcc
104+
105+
[b2.args.os=linux]
106+
# RHEL hardened gcc is not compatible with PCH
107+
# https://bugzilla.redhat.com/show_bug.cgi?id=1806545
108+
pch=off
109+
# Python extensions need -fPIC for static library linking into shared objects
110+
cxxflags="-fPIC"
111+
112+
[b2.args.os=darwin]
113+
toolset=clang
114+
# Since Xcode 15.3 std::piecewise_construct is only visible in C++17 and later modes
115+
cxxflags="-DBOOST_UNORDERED_HAVE_PIECEWISE_CONSTRUCT=0"
116+
117+
[b2.args.all(os=windows,fb=on)]
118+
toolset=msvc-14.3

‎build/fbcode_builder/manifests/double-conversion-python‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
[manifest]
2+
name = double-conversion-python
3+
4+
[download]
5+
url = https://github.com/google/double-conversion/archive/v3.1.4.tar.gz
6+
sha256 = 95004b65e43fefc6100f337a25da27bb99b9ef8d4071a36a33b5e83eb1f82021
7+
8+
[homebrew]
9+
double-conversion
10+
11+
[debs]
12+
libdouble-conversion-dev
13+
14+
[rpms]
15+
double-conversion
16+
double-conversion-devel
17+
18+
[pps]
19+
double-conversion
20+
21+
[build]
22+
builder = cmake
23+
subdir = double-conversion-3.1.4
24+
25+
[build.not(os=linux)]
26+
builder = nop
27+
28+
[cmake.defines]
29+
CMAKE_POSITION_INDEPENDENT_CODE=ON

‎build/fbcode_builder/manifests/fbthrift-python‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
[manifest]
2+
name = fbthrift-python
3+
fbsource_path = xplat/thrift
4+
shipit_project = fbthrift
5+
shipit_fbcode_builder = true
6+
7+
[git]
8+
repo_url = https://github.com/facebook/fbthrift.git
9+
10+
[cargo]
11+
cargo_config_file = source/thrift/lib/rust/.cargo/config.toml
12+
13+
[crate.pathmap]
14+
fbthrift = thrift/lib/rust
15+
16+
[build]
17+
builder = cmake
18+
job_weight_mib = 2048
19+
20+
[build.not(os=linux)]
21+
builder = nop
22+
23+
[cmake.defines.all(not(os=windows),test=on)]
24+
enable_tests=ON
25+
26+
[cmake.defines.any(os=windows,test=off)]
27+
enable_tests=OFF
28+
29+
[cmake.defines.os=linux]
30+
thrift_python=ON
31+
enable_tests=ON
32+
33+
[dependencies]
34+
fizz-python
35+
fmt-python
36+
folly-python
37+
googletest
38+
libsodium
39+
wangle-python
40+
zstd-python
41+
mvfst-python
42+
xxhash
43+
# Thrift also depends on openssl but since the latter requires a platform-
44+
# specific configuration we rely on the folly manifest to provide this
45+
# dependency to avoid duplication.
46+
47+
[dependencies.os=linux]
48+
libaio
49+
libevent-python
50+
proxygen-python
51+
52+
[shipit.pathmap]
53+
xplat/thrift/public_tld = .
54+
xplat/thrift = thrift
55+
56+
[shipit.strip]
57+
^xplat/thrift/thrift-config\.h$
58+
^xplat/thrift/perf/canary.py$
59+
^xplat/thrift/perf/loadtest.py$
60+
^xplat/thrift/.castle/.*

‎build/fbcode_builder/manifests/fizz-python‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
[manifest]
2+
name = fizz-python
3+
fbsource_path = fbcode/fizz
4+
shipit_project = fizz
5+
shipit_fbcode_builder = true
6+
7+
[git]
8+
repo_url = https://github.com/facebookincubator/fizz.git
9+
10+
[build]
11+
builder = cmake
12+
subdir = fizz
13+
14+
[build.not(os=linux)]
15+
builder = nop
16+
17+
[cmake.defines]
18+
BUILD_EXAMPLES = OFF
19+
20+
[cmake.defines.os=linux]
21+
CMAKE_POSITION_INDEPENDENT_CODE = ON
22+
BUILD_SHARED_LIBS = ON
23+
24+
[cmake.defines.test=on]
25+
BUILD_TESTS = ON
26+
27+
[cmake.defines.all(os=windows, test=on)]
28+
BUILD_TESTS = OFF
29+
30+
[cmake.defines.test=off]
31+
BUILD_TESTS = OFF
32+
33+
[dependencies]
34+
folly-python
35+
liboqs
36+
libsodium
37+
zlib-python
38+
zstd-python
39+
40+
[dependencies.all(test=on, not(os=windows))]
41+
googletest
42+
43+
[shipit.pathmap]
44+
fbcode/fizz/public_tld = .
45+
fbcode/fizz = fizz

‎build/fbcode_builder/manifests/fmt-python‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
[manifest]
2+
name = fmt-python
3+
4+
[download]
5+
url = https://github.com/fmtlib/fmt/archive/refs/tags/12.1.0.tar.gz
6+
sha256 = ea7de4299689e12b6dddd392f9896f08fb0777ac7168897a244a6d6085043fea
7+
8+
[build]
9+
builder = cmake
10+
subdir = fmt-12.1.0
11+
12+
[build.not(os=linux)]
13+
builder = nop
14+
15+
[cmake.defines]
16+
FMT_TEST = OFF
17+
FMT_DOC = OFF
18+
# Build as shared library so Python extensions can find fmt symbols at runtime
19+
# (fmt uses -fvisibility=hidden, so static linking leaves symbols unexported)
20+
BUILD_SHARED_LIBS = ON
21+
22+
[homebrew]
23+
fmt
24+
25+
[rpms.distro=fedora]
26+
fmt-devel

‎build/fbcode_builder/manifests/folly-python‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
[manifest]
2+
name = folly-python
3+
fbsource_path = fbcode/folly
4+
shipit_project = folly
5+
shipit_fbcode_builder = true
6+
7+
[git]
8+
repo_url = https://github.com/facebook/folly.git
9+
10+
[build]
11+
builder = cmake
12+
job_weight_mib = 1024
13+
14+
[build.not(os=linux)]
15+
builder = nop
16+
17+
[dependencies]
18+
gflags
19+
glog
20+
googletest
21+
boost-python
22+
libdwarf-python
23+
libevent-python
24+
libsodium
25+
double-conversion-python
26+
fast_float
27+
fmt-python
28+
lz4-python
29+
snappy
30+
zstd-python
31+
# no openssl or zlib in the linux case, why?
32+
# these are usually installed on the system
33+
# and are the easiest system deps to pull in.
34+
# In the future we want to be able to express
35+
# that a system dep is sufficient in the manifest
36+
# for eg: openssl and zlib, but for now we don't
37+
# have it.
38+
39+
# macOS doesn't expose the openssl api so we need
40+
# to build our own.
41+
[dependencies.os=darwin]
42+
openssl
43+
44+
# Windows has neither openssl nor zlib, so we get
45+
# to provide both
46+
[dependencies.os=windows]
47+
openssl
48+
zlib
49+
50+
[dependencies.os=linux]
51+
libaio
52+
libiberty-python
53+
libunwind
54+
55+
# xz depends on autoconf which does not build on
56+
# Windows
57+
[dependencies.not(os=windows)]
58+
xz
59+
60+
[shipit.pathmap]
61+
fbcode/folly/public_tld = .
62+
fbcode/folly = folly
63+
64+
[shipit.strip]
65+
^fbcode/folly/folly-config\.h$
66+
^fbcode/folly/public_tld/build/facebook_.*
67+
68+
[cmake.defines.os=linux]
69+
PYTHON_EXTENSIONS=ON
70+
BUILD_SHARED_LIBS=ON
71+
72+
[cmake.defines.not(os=windows)]
73+
BOOST_LINK_STATIC=ON
74+
75+
[cmake.defines.os=freebsd]
76+
LIBDWARF_FOUND=NO
77+
78+
[cmake.defines.test=on]
79+
BUILD_TESTS=ON
80+
BUILD_BENCHMARKS=OFF
81+
82+
[cmake.defines.test=off]
83+
BUILD_TESTS=OFF
84+
BUILD_BENCHMARKS=OFF

‎build/fbcode_builder/manifests/libdwarf-python‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
[manifest]
2+
name = libdwarf-python
3+
4+
[rpms]
5+
libdwarf-devel
6+
libdwarf
7+
8+
[debs]
9+
libdwarf-dev
10+
11+
[homebrew]
12+
dwarfutils
13+
14+
[download]
15+
url = https://www.prevanders.net/libdwarf-0.9.2.tar.xz
16+
sha256 = 22b66d06831a76f6a062126cdcad3fcc58540b89a1acb23c99f8861f50999ec3
17+
18+
[build]
19+
builder = cmake
20+
subdir = libdwarf-0.9.2
21+
22+
[build.not(os=linux)]
23+
builder = nop
24+
25+
[cmake.defines]
26+
CMAKE_POSITION_INDEPENDENT_CODE=ON

0 commit comments

Comments
 (0)