Skip to content

Commit 4565a24

Browse files
build: Add a python script to test imports during the build (AcademySoftwareFoundation#1057)
### Add a python script to test imports during the build ### Linked issues n/a ### Summarize your change. Add build-time validation of Python package imports and clarify Python test purposes. ### Describe the reason for the change. When there are issues with package imports (ABI incompatibilities, missing dependencies, etc.), we often find them out at runtime. This change adds `test_python_imports.py `to detect import failures during the build. Additionally, renamed `test_python.py` to `test_python_distribution.py` to better distinguish between: - Import testing (validates pip-installed packages) - Distribution testing (validates relocatability and environment setup) ### Describe what you have tested and on which operating system. MacOS ### Add a list of changes, and note any that might need special attention during the review. ### If possible, provide screenshots. --------- Signed-off-by: Cédrik Fuoco <cedrik.fuoco@autodesk.com>
1 parent 3bebfb6 commit 4565a24

4 files changed

Lines changed: 248 additions & 12 deletions

File tree

cmake/dependencies/python3.cmake

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -50,14 +50,6 @@ SET(_python3_download_hash
5050
"${RV_DEPS_PYTHON_DOWNLOAD_HASH}"
5151
)
5252

53-
SET(_opentimelineio_download_url
54-
"https://github.com/AcademySoftwareFoundation/OpenTimelineIO"
55-
)
56-
57-
SET(_opentimelineio_git_tag
58-
"v${_opentimelineio_version}"
59-
)
60-
6153
SET(_pyside_archive_url
6254
"${RV_DEPS_PYSIDE_ARCHIVE_URL}"
6355
)
@@ -504,21 +496,40 @@ ADD_CUSTOM_COMMAND(
504496
DEPENDS ${_python3_target} ${${_python3_target}-build-deps-flag} ${_requirements_output_file} ${_requirements_input_file}
505497
)
506498

507-
# Test the Python distribution after requirements are installed
499+
# Test Python package imports after requirements are installed. This validates that all pip-installed packages (numpy, opentimelineio, OpenGL, cryptography,
500+
# etc.) can be imported successfully and tests for ABI compatibility issues. Runs in-place using the built Python executable.
501+
SET(${_python3_target}-imports-test-flag
502+
${_install_dir}/${_python3_target}-imports-test-flag
503+
)
504+
505+
SET(_test_python_imports_script
506+
"${PROJECT_SOURCE_DIR}/src/build/test_python_imports.py"
507+
)
508+
509+
ADD_CUSTOM_COMMAND(
510+
COMMENT "Testing Python package imports (build-time validation)"
511+
OUTPUT ${${_python3_target}-imports-test-flag}
512+
COMMAND "${_python3_executable}" "${_test_python_imports_script}"
513+
COMMAND cmake -E touch ${${_python3_target}-imports-test-flag}
514+
DEPENDS ${${_python3_target}-requirements-flag} ${_test_python_imports_script}
515+
)
516+
517+
# Test the Python distribution's relocatability and environment setup. This moves Python to a temporary location to ensure it works when relocated and validates
518+
# SSL_CERT_FILE setup via sitecustomize.py. Uses system Python to orchestrate the test (since the built Python gets moved during testing).
508519
SET(${_python3_target}-test-flag
509520
${_install_dir}/${_python3_target}-test-flag
510521
)
511522

512523
SET(_test_python_script
513-
"${PROJECT_SOURCE_DIR}/src/build/test_python.py"
524+
"${PROJECT_SOURCE_DIR}/src/build/test_python_distribution.py"
514525
)
515526

516527
ADD_CUSTOM_COMMAND(
517-
COMMENT "Testing Python distribution"
528+
COMMENT "Testing Python distribution relocatability and environment"
518529
OUTPUT ${${_python3_target}-test-flag}
519530
COMMAND python3 "${_test_python_script}" --python-home "${_install_dir}" --variant "${CMAKE_BUILD_TYPE}"
520531
COMMAND cmake -E touch ${${_python3_target}-test-flag}
521-
DEPENDS ${${_python3_target}-requirements-flag} ${_test_python_script}
532+
DEPENDS ${${_python3_target}-imports-test-flag} ${_test_python_script}
522533
)
523534

524535
IF(RV_TARGET_WINDOWS
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
#!/bin/bash
2+
#
3+
# Copyright (C) 2026 Autodesk, Inc. All Rights Reserved.
4+
#
5+
# SPDX-License-Identifier: Apache-2.0
6+
#
7+
# strip_provenance_and_link.sh — macOS 26+ provenance-safe linker wrapper
8+
#
9+
# On macOS 26+, com.apple.provenance is kernel-enforced and irremovable. When multiple
10+
# processes write signed Mach-O files to the same directory concurrently (parallel linker
11+
# invocations, linker + file-copy custom commands, etc.), the kernel blocks the open()
12+
# with EPERM, causing "ld: open() failed, errno=1".
13+
#
14+
# Fix: link to a private temporary directory (no contention), then atomically move the
15+
# output to the real destination. Each link step gets its own temp dir, so there is zero
16+
# concurrent write contention regardless of Ninja parallelism.
17+
#
18+
# The -install_name / @rpath embedded in the binary comes from the -install_name flag,
19+
# not from the -o path, so moving the file afterward is safe.
20+
#
21+
# Usage: strip_provenance_and_link.sh <linker> [args...] -o <output> [more args...]
22+
23+
# Find the -o argument and rewrite it to a temp directory.
24+
args=("$@")
25+
real_output=""
26+
27+
for ((i=0; i<${#args[@]}; i++)); do
28+
if [[ "${args[$i]}" == "-o" ]]; then
29+
j=$((i+1))
30+
real_output="${args[$j]}"
31+
fname=$(basename "$real_output")
32+
33+
# Create a private temp dir for this link step.
34+
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/rv_link.XXXXXX")
35+
trap "rm -rf '$tmpdir'" EXIT
36+
37+
# Rewrite -o to point to the temp dir.
38+
args[$j]="${tmpdir}/${fname}"
39+
break
40+
fi
41+
done
42+
43+
# Debug: verify this script version is actually running.
44+
echo "PROVENANCE_WRAPPER_V3: real_output=${real_output} tmpdir=${tmpdir} fname=${fname}" >&2
45+
46+
# Run the linker, outputting to the private temp dir.
47+
"${args[@]}"
48+
status=$?
49+
50+
if [[ $status -eq 0 && -n "$real_output" ]]; then
51+
# Atomically move the output to the real destination.
52+
# mv within the same filesystem is atomic; across filesystems it copies+deletes.
53+
mv -f "${tmpdir}/${fname}" "${real_output}"
54+
mv_status=$?
55+
if [[ $mv_status -ne 0 ]]; then
56+
echo "PROVENANCE_WRAPPER_V3: mv failed (status=$mv_status) from ${tmpdir}/${fname} to ${real_output}" >&2
57+
status=$mv_status
58+
fi
59+
fi
60+
61+
exit $status

src/build/test_python_imports.py

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
#!/usr/bin/env python3
2+
# -*- coding: utf-8 -*-
3+
4+
# *****************************************************************************
5+
# Copyright 2025 Autodesk, Inc. All rights reserved.
6+
#
7+
# SPDX-License-Identifier: Apache-2.0
8+
#
9+
# *****************************************************************************
10+
11+
"""
12+
Test script to validate all Python package imports at build time.
13+
This catches issues like missing dependencies, ABI incompatibilities, and
14+
configuration problems (like OpenSSL legacy provider) before running the build.
15+
16+
Package list is automatically generated from requirements.txt.in to prevent
17+
manual synchronization errors.
18+
"""
19+
20+
import os
21+
import re
22+
import sys
23+
24+
# Packages to skip from requirements.txt.in (e.g., build dependencies that don't need import testing).
25+
SKIP_IMPORTS = [
26+
"pip",
27+
"setuptools",
28+
"wheel",
29+
]
30+
31+
# Additional imports to test beyond what's in requirements.txt.in.
32+
# These test deeper functionality (e.g., sub-modules that verify OpenSSL legacy provider support).
33+
ADDITIONAL_IMPORTS = [
34+
"cryptography.fernet",
35+
"cryptography.hazmat.bindings._rust",
36+
]
37+
38+
39+
def parse_requirements(file_path):
40+
"""Parse requirements.txt.in and return list of package names.
41+
42+
Extracts package names from lines like:
43+
- numpy==@_numpy_version@
44+
- pip==24.0
45+
46+
Skips comments and empty lines.
47+
"""
48+
packages = []
49+
with open(file_path, encoding="utf-8") as f:
50+
for line in f:
51+
# Remove inline comments and whitespace
52+
line = line.split("#", 1)[0].strip()
53+
# Skip comments and empty lines
54+
if not line or line.startswith("#"):
55+
continue
56+
# Handle VCS/URL form: name @ git+...
57+
if " @ " in line:
58+
packages.append(line.split(" @ ", 1)[0].strip())
59+
continue
60+
# Extract package name (before ==, <, >, etc.)
61+
match = re.match(r"^([A-Za-z0-9_-]+)", line)
62+
if match:
63+
packages.append(match.group(1))
64+
return packages
65+
66+
67+
def try_import(package_name):
68+
"""Try importing package, with fallback to stripped 'Py' prefix.
69+
70+
Handles cases where pip package name differs from import name:
71+
- PyOpenGL -> OpenGL
72+
- PyOpenGL_accelerate -> OpenGL_accelerate
73+
74+
Raises ImportError if both attempts fail.
75+
"""
76+
try:
77+
return __import__(package_name)
78+
except ImportError:
79+
if package_name.startswith("Py"):
80+
# Try without "Py" prefix
81+
return __import__(package_name[2:])
82+
raise
83+
84+
85+
def test_imports():
86+
"""Test that all required packages can be imported."""
87+
# No PATH manipulation needed.
88+
89+
# Get requirements.txt.in from same directory as this script
90+
script_dir = os.path.dirname(os.path.abspath(__file__))
91+
requirements_file = os.path.join(script_dir, "requirements.txt.in")
92+
93+
if not os.path.exists(requirements_file):
94+
print(f"ERROR: Could not find {requirements_file}")
95+
return 1
96+
97+
# Parse package list from requirements.txt.in
98+
packages = parse_requirements(requirements_file)
99+
100+
# Build list of imports to test (packages from requirements.txt + additional imports)
101+
imports_to_test = []
102+
skipped_packages = []
103+
for package in packages:
104+
# Skip build dependencies and other packages that don't need import testing
105+
if package not in SKIP_IMPORTS:
106+
imports_to_test.append((package, "from requirements.txt"))
107+
else:
108+
skipped_packages.append(package)
109+
110+
# Add any additional imports (e.g., sub-modules for deeper testing)
111+
for additional_import in ADDITIONAL_IMPORTS:
112+
imports_to_test.append((additional_import, "additional import"))
113+
114+
failed_imports = []
115+
successful_imports = []
116+
117+
print("=" * 80)
118+
print("Testing Python package imports at build time")
119+
print(f"Python: {sys.version}")
120+
print(f"Platform: {sys.platform}")
121+
print(f"Executable: {sys.executable}")
122+
print(f"Source: {requirements_file}")
123+
if skipped_packages:
124+
print(f"Skipping: {', '.join(skipped_packages)}")
125+
126+
print("=" * 80)
127+
print()
128+
129+
for module_name, description in imports_to_test:
130+
try:
131+
print(f"Testing {module_name:45} ({description})...", end=" ")
132+
try_import(module_name)
133+
print("OK")
134+
successful_imports.append(module_name)
135+
except Exception as e:
136+
print("FAILED")
137+
print(f" Error: {type(e).__name__}: {e}")
138+
failed_imports.append((module_name, e))
139+
140+
# Keep failure output minimal; detailed diagnostics removed.
141+
142+
print()
143+
print("=" * 80)
144+
print(f"Results: {len(successful_imports)} passed, {len(failed_imports)} failed")
145+
print("=" * 80)
146+
147+
if failed_imports:
148+
print()
149+
print("FAILED IMPORTS:")
150+
for module_name, error in failed_imports:
151+
print(f" - {module_name}: {type(error).__name__}: {error}")
152+
print()
153+
print("Build-time import test FAILED!")
154+
print("One or more required Python packages could not be imported.")
155+
print()
156+
return 1
157+
else:
158+
print()
159+
print("All Python package imports successful!")
160+
return 0
161+
162+
163+
if __name__ == "__main__":
164+
sys.exit(test_imports())

0 commit comments

Comments
 (0)