Skip to content

Commit 4fe3d63

Browse files
authored
Fix wheel naming on macOS for single-arch rebuilds (DataDog#20699)
While working on DataDog/datadog-agent#38582, I realized `arm64` shared libraries could surprisingly be pulled into `x86_64` DMGs, for instance: ```json { "path": "datadog-agent-7.69.0-devel.git.364.9cd4057.pipeline.70042062-1.dmg/datadog-agent-7.69.0-devel.git.364.9cd4057.pipeline.70042062-1.pkg/datadog-agent-core.pkg Contents/Payload/opt/datadog-agent/embedded/lib/python3.12/site-packages/confluent_kafka/.dylibs/liblmdb.so", "digestAlgorithm": "SHA-256", "cdhash": "5c30429cac77cd7a04f798957cd6c4ea876555d4", "arch": "arm64" } ``` (https://gitlab.ddbuild.io/DataDog/datadog-agent/-/jobs/1020563452) It would be perfectly fine if that shared library would embed multiple architectures in addition to the `x86_64` target, but it's clearly not the case for `confluent_kafka` that we rebuild and repackage as a single-architecture wheel. The selection of the wheel to pin in the lock file is based on a filter that accepts `universal2` for both architectures: ```py 'macos-x86_64': 'macosx.*_(x86_64|intel|universal2)', 'macos-aarch64': 'macosx.*_(aarch64|arm64|universal2)', ``` (https://github.com/DataDog/integrations-core/blob/80e9e635ef25806f1501ebe0d2d37740f180bd69/.builders/lock.py#L25-L26) A problem is that available wheels are treated in the order returned by the cloud storage API (https://agent-int-packages.datadoghq.com), which means the latest iterated over for a given version will be considered in the final selection, independently from other factors like the build timestamp or the macOS version: ```py build_number = int(build[0]) if build else -1 candidates[build_number] = blob ``` (https://github.com/DataDog/integrations-core/blob/80e9e635ef25806f1501ebe0d2d37740f180bd69/.builders/lock.py#L112-L113) Practically speaking, which one of the following wheels will be picked relies on the order in which they're returned by the cloud storage API: - `confluent_kafka-2.8.0-20250707163239-cp312-cp312-macosx_10_12_universal2.whl` - `confluent_kafka-2.8.0-20250707163256-cp312-cp312-macosx_11_0_universal2.whl` - `confluent_kafka-2.8.0-20250707171022-cp312-cp312-macosx_10_12_universal2.whl` - `confluent_kafka-2.8.0-20250707171037-cp312-cp312-macosx_11_0_universal2.whl` The present change aims at decreasing the probability `arm64`-only shared libraries get transitively depended on by `x86_64` targets and the other way around. In the above example, this would limit the wheels being considered to applicable architectures - for `x86_64`: - `confluent_kafka-2.8.0-20250707163239-cp312-cp312-macosx_10_12_x86_64.whl` - `confluent_kafka-2.8.0-20250707171022-cp312-cp312-macosx_10_12_x86_64.whl` ... and, for `arm64`: - `confluent_kafka-2.8.0-20250707163256-cp312-cp312-macosx_11_0_arm64.whl` - `confluent_kafka-2.8.0-20250707171037-cp312-cp312-macosx_11_0_arm64.whl`
1 parent be5c0b2 commit 4fe3d63

1 file changed

Lines changed: 9 additions & 2 deletions

File tree

.builders/scripts/repair_wheels.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,8 @@ def repair_windows(source_dir: str, built_dir: str, external_dir: str) -> None:
228228

229229
def repair_darwin(source_dir: str, built_dir: str, external_dir: str) -> None:
230230
from delocate import delocate_wheel
231+
from packaging.version import Version
232+
231233
exclusions = [re.compile(s) for s in [
232234
# pymqi
233235
r'pymqe\.cpython-\d+-darwin\.so',
@@ -277,12 +279,17 @@ def copy_filt_func(libname):
277279
shutil.move(wheel, Path(built_dir) / dest)
278280
continue
279281

282+
# Platform dependent wheels: rename with single arch and verify target macOS version
283+
single_arch = os.uname().machine
284+
dest = str(wheel_name._replace(platform_tag=wheel_name.platform_tag.replace('universal2', single_arch)))
280285
copied_libs = delocate_wheel(
281286
str(wheel),
282-
os.path.join(built_dir, wheel.name),
287+
os.path.join(built_dir, dest),
283288
copy_filt_func=copy_filt_func,
289+
# require_archs=[single_arch], TODO(regis): address multi-arch confluent_kafka/cimpl.cpython-312-darwin.so
290+
require_target_macos_version=Version(os.environ["MACOSX_DEPLOYMENT_TARGET"]),
284291
)
285-
print('Repaired wheel')
292+
print(f'Repaired wheel to {dest}')
286293
if copied_libs:
287294
print('Libraries copied into the wheel:')
288295
print('\n'.join(copied_libs))

0 commit comments

Comments
 (0)