Skip to content

Commit ec86e94

Browse files
DarylHawkinsAMDclaude
DarylHawkinsAMD
and
claude
committed
[CK DSL] Embed python-build-standalone interpreter (drop system Python)
Provision a pinned, self-contained CPython from astral's python-build-standalone (PBS) for the embedded interpreter instead of depending on the host system Python. This removes the fragile /usr pin and the `pip install --user pybind11` prerequisite, and is the first step toward a relocatable, self-contained installable plugin. - cmake/CkDslPython.cmake (new): ck_dsl_provider_provision_python() honors -DCKDSL_PYTHON_DIST_DIR (air-gap / pre-stage) else FetchContents the pinned PBS install_only tarball with sha256 verification, then steers find_package(Python3) at the provisioned prefix. Asset/sha are pinned per platform (Linux x86_64 verified; aarch64 + Windows wired). - CMakeLists.txt: replace the /usr Python pin and the pip pybind11 probe with the provisioning include; pybind11 falls back to a pinned FetchContent when no CMake config is found. - EmbeddedInterpreter: pin PyConfig.home + executable to the bundled prefix so the bundled stdlib loads deterministically under the existing isolated config; executable path is platform-aware. - Bake the bundled prefix as kCkDslPythonHome in the generated paths header. - Tests: add provenance (stdlib resolves from the bundled prefix) and ctypes->comgr load checks. PBS statically links the runtime C-extension tail (_ctypes/_hashlib/ _bz2/_lzma) into libpython, so the hardened plugin loads them through the single libpython dependency with no separate-extension symbol issues. Verified on Linux/gfx1151: configure + build green, unit 62/62, integration 10/10 conv (real compile via PBS + comgr on-device). Windows: provisioning validation and the interpreter executable path are platform-aware but NOT yet verified on Windows; the bundled-DLL runtime search (no $ORIGIN on Windows) and install staging are still to come. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 85119c5 commit ec86e94

6 files changed

Lines changed: 302 additions & 34 deletions

File tree

‎dnn-providers/ck-dsl-provider/CMakeLists.txt‎

Lines changed: 22 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -52,46 +52,35 @@ list(APPEND CMAKE_PREFIX_PATH ${ROCM_PATH}/llvm ${ROCM_PATH} ${ROCM_PATH}/hip)
5252

5353
find_package(hip REQUIRED)
5454

55-
# --- Embedded Python (libpython + pybind11) ----------------------------------
56-
# I-2 wires an embedded CPython interpreter into the plugin .so so that
57-
# the provider can drive the CK DSL compile pipeline at runtime (plan
58-
# §3.2, Option E + pybind11). The plumbing below mirrors the working
59-
# spike at WIP/pybind11_rtld_local_spike/.
60-
#
61-
# Pin to system Python so a uv-managed Python in ~/.local does not
62-
# hijack the discovery (which crashes at runtime with "failed to get
63-
# the Python codec of the filesystem encoding" because uv's stdlib
64-
# lives at an interpreter-internal path). See PREP_FINDINGS P-3.
65-
if(NOT DEFINED Python3_ROOT_DIR)
66-
set(Python3_ROOT_DIR "/usr")
67-
endif()
68-
if(NOT DEFINED Python3_FIND_STRATEGY)
69-
set(Python3_FIND_STRATEGY "LOCATION")
70-
endif()
55+
# --- Embedded Python (python-build-standalone + pybind11) --------------------
56+
# The plugin embeds a self-contained CPython so the provider can drive
57+
# the CK DSL compile pipeline at runtime with no dependency on the host
58+
# system Python. The interpreter is provisioned from astral's
59+
# python-build-standalone (a pinned, relocatable prefix), discovered via
60+
# find_package below, and bundled beside the plugin at install time.
61+
# See cmake/CkDslPython.cmake. Provisioning honors -DCKDSL_PYTHON_DIST_DIR
62+
# for air-gapped / pre-staged builds, else downloads the pinned tarball.
63+
include(${CMAKE_CURRENT_LIST_DIR}/cmake/CkDslPython.cmake)
64+
ck_dsl_provider_provision_python()
65+
7166
find_package(Python3 COMPONENTS Interpreter Development REQUIRED)
7267
message(STATUS "CK DSL provider Python3: ${Python3_EXECUTABLE} (${Python3_VERSION})")
7368

74-
# pybind11: prefer the standard CMake config; if that fails, fall back
75-
# to asking the pinned interpreter where its cmake dir is (this is the
76-
# common case for `pip install --user pybind11`, which lands the
77-
# cmake config under ~/.local/lib/.../pybind11/share/cmake/pybind11).
69+
# pybind11: prefer an installed CMake config; otherwise fetch a pinned
70+
# release. pybind11 is header-only for the embed use case and resolves
71+
# the provisioned interpreter through Python3_ROOT_DIR (set by the
72+
# provisioning step above), so no host `pip install` is required.
7873
find_package(pybind11 CONFIG QUIET)
7974
if(NOT pybind11_FOUND)
80-
execute_process(
81-
COMMAND ${Python3_EXECUTABLE} -m pybind11 --cmakedir
82-
OUTPUT_VARIABLE _pybind11_cmakedir
83-
OUTPUT_STRIP_TRAILING_WHITESPACE
84-
RESULT_VARIABLE _pybind11_probe_result
75+
include(FetchContent)
76+
set(PYBIND11_FINDPYTHON ON CACHE INTERNAL "")
77+
fetchcontent_declare(
78+
pybind11 URL https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.tar.gz
79+
DOWNLOAD_EXTRACT_TIMESTAMP TRUE
8580
)
86-
if(NOT _pybind11_probe_result EQUAL 0)
87-
message(FATAL_ERROR
88-
"pybind11 not found via find_package and `${Python3_EXECUTABLE} -m pybind11 --cmakedir` failed. "
89-
"Install with: ${Python3_EXECUTABLE} -m pip install --user pybind11")
90-
endif()
91-
set(pybind11_DIR "${_pybind11_cmakedir}" CACHE PATH "pybind11 CMake config dir (auto-detected)" FORCE)
92-
find_package(pybind11 CONFIG REQUIRED)
81+
fetchcontent_makeavailable(pybind11)
9382
endif()
94-
message(STATUS "CK DSL provider pybind11: ${pybind11_INCLUDE_DIR} (${pybind11_VERSION})")
83+
message(STATUS "CK DSL provider pybind11: ${pybind11_VERSION}")
9584

9685
# hipdnn SDK dependencies - use targets directly if available (superbuild),
9786
# otherwise find installed packages (true standalone).

‎dnn-providers/ck-dsl-provider/cmake/CkDslProviderPaths.cmake‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,11 +75,23 @@ function(ck_dsl_provider_resolve_python_paths)
7575
"${_providerPyDir}/ck_dsl_provider/__init__.py.")
7676
endif()
7777

78+
# The bundled python-build-standalone prefix (build-tree absolute).
79+
# Provisioned by cmake/CkDslPython.cmake into the CKDSL_PYTHON_PREFIX
80+
# cache var before this function runs.
81+
if(NOT CKDSL_PYTHON_PREFIX)
82+
message(FATAL_ERROR
83+
"CK DSL provider: CKDSL_PYTHON_PREFIX is not set. "
84+
"ck_dsl_provider_provision_python() must run before "
85+
"ck_dsl_provider_resolve_python_paths().")
86+
endif()
87+
7888
set(CK_DSL_PYTHON_PACKAGE_PATH "${_resolvedCkDslDir}" PARENT_SCOPE)
7989
set(CK_DSL_PROVIDER_PYTHON_PACKAGE_PATH "${_providerPyDir}" PARENT_SCOPE)
90+
set(CK_DSL_PYTHON_HOME "${CKDSL_PYTHON_PREFIX}" PARENT_SCOPE)
8091

8192
message(STATUS
8293
"CK DSL provider Python paths: "
8394
"ck_dsl=${_resolvedCkDslDir}, "
84-
"ck_dsl_provider=${_providerPyDir}")
95+
"ck_dsl_provider=${_providerPyDir}, "
96+
"python_home=${CKDSL_PYTHON_PREFIX}")
8597
endfunction()

‎dnn-providers/ck-dsl-provider/cmake/CkDslPython.cmake‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Copyright © Advanced Micro Devices, Inc., or its affiliates.
2+
# SPDX-License-Identifier: MIT
3+
4+
# Provision a self-contained CPython for the embedded interpreter from
5+
# astral's python-build-standalone (PBS), replacing any dependency on
6+
# the host system Python.
7+
#
8+
# The PBS `install_only` distribution ships a relocatable prefix with a
9+
# shared libpython, the full stdlib, and headers. Critically, it links
10+
# the runtime C-extensions the CK DSL compile path needs
11+
# (`_ctypes` -> comgr, `_hashlib`, `_bz2`, `_lzma`) statically into
12+
# libpython itself, so the hardened plugin (hidden visibility +
13+
# --exclude-libs=ALL) can load them through the single libpython
14+
# DT_NEEDED dependency -- no separate extension .so resolution.
15+
#
16+
# Provisioning honors a caller-supplied distribution first
17+
# (CKDSL_PYTHON_DIST_DIR), so air-gapped CI / TheRock can pre-stage the
18+
# prefix; otherwise the pinned tarball is downloaded and checksum-
19+
# verified via FetchContent.
20+
#
21+
# Outputs (cache):
22+
# CKDSL_PYTHON_PREFIX absolute path of the CPython prefix
23+
# (the dir containing bin/ lib/ include/)
24+
# CKDSL_PYTHON_VERSION the CPython MAJOR.MINOR (e.g. "3.12")
25+
#
26+
# Side effects: sets Python3_ROOT_DIR + discovery hints in the parent
27+
# scope so a subsequent find_package(Python3 ... Development) resolves
28+
# the provisioned prefix rather than any host interpreter.
29+
30+
# --- Pinned distribution -----------------------------------------------------
31+
# python-build-standalone release tag and the install_only assets. Bump
32+
# the tag + all three checksums together; verify against the release's
33+
# SHA256SUMS (PBS does not publish per-asset .sha256 sidecars).
34+
set(_CKDSL_PYTHON_PBS_TAG "20260602")
35+
set(_CKDSL_PYTHON_PBS_VERSION "3.12.13")
36+
set(_CKDSL_PYTHON_PBS_BASE
37+
"https://github.com/astral-sh/python-build-standalone/releases/download/${_CKDSL_PYTHON_PBS_TAG}"
38+
)
39+
40+
# Per-platform install_only asset + sha256 (Linux x86_64 verified; the
41+
# others are wired for parity and selected by host, but only the Linux
42+
# x86_64 path is validated today).
43+
set(_CKDSL_PYTHON_ASSET_linux_x86_64
44+
"cpython-${_CKDSL_PYTHON_PBS_VERSION}+${_CKDSL_PYTHON_PBS_TAG}-x86_64-unknown-linux-gnu-install_only.tar.gz")
45+
set(_CKDSL_PYTHON_SHA_linux_x86_64
46+
"9be5c21b78dbc371e739bc7faf3b007b8e607335f780bdd2e0dd44a6e3580d76")
47+
48+
set(_CKDSL_PYTHON_ASSET_linux_aarch64
49+
"cpython-${_CKDSL_PYTHON_PBS_VERSION}+${_CKDSL_PYTHON_PBS_TAG}-aarch64-unknown-linux-gnu-install_only.tar.gz")
50+
set(_CKDSL_PYTHON_SHA_linux_aarch64
51+
"f0c9ea0022b2dfdf0a4733e962ba8cc883c45d26df26116b9802b658240a25d7")
52+
53+
set(_CKDSL_PYTHON_ASSET_windows_x86_64
54+
"cpython-${_CKDSL_PYTHON_PBS_VERSION}+${_CKDSL_PYTHON_PBS_TAG}-x86_64-pc-windows-msvc-install_only.tar.gz")
55+
set(_CKDSL_PYTHON_SHA_windows_x86_64
56+
"f89539b0a6f1d48c655f97b7e77f3f8738bbe9d7a32c9306d0d20335dc8ae0fb")
57+
58+
# Snapshot this .cmake file's dir at include time (CMAKE_CURRENT_LIST_DIR
59+
# inside a function body reflects the caller, not the definition site).
60+
set(_ckDslPythonCmakeDir "${CMAKE_CURRENT_LIST_DIR}")
61+
62+
# Validate that a directory looks like a usable CPython prefix: the
63+
# shared library, the public header, and the interpreter binary must all
64+
# exist. The python-build-standalone layout differs by platform:
65+
# Linux: bin/python<ver>, lib/libpython<ver>.so, include/python<ver>/
66+
# Windows: python.exe, python3.dll (+ python<ver>.dll), include/ (flat)
67+
# Sets ${outVar} TRUE/FALSE in the caller's scope.
68+
function(_ck_dsl_python_validate_prefix prefix version outVar)
69+
set(_ok TRUE)
70+
if(WIN32)
71+
set(_lib "${prefix}/python3.dll")
72+
set(_bin "${prefix}/python.exe")
73+
set(_inc "${prefix}/include/Python.h")
74+
else()
75+
set(_lib "${prefix}/lib/libpython${version}.so")
76+
set(_bin "${prefix}/bin/python${version}")
77+
set(_inc "${prefix}/include/python${version}/Python.h")
78+
endif()
79+
foreach(_p "${_lib}" "${_bin}" "${_inc}")
80+
if(NOT EXISTS "${_p}")
81+
message(STATUS "CK DSL provider: python prefix missing ${_p}")
82+
set(_ok FALSE)
83+
endif()
84+
endforeach()
85+
set(${outVar} ${_ok} PARENT_SCOPE)
86+
endfunction()
87+
88+
# Resolve the pinned asset + sha for the host platform.
89+
function(_ck_dsl_python_select_asset assetOut shaOut)
90+
if(WIN32 AND CMAKE_SIZEOF_VOID_P EQUAL 8)
91+
set(_key "windows_x86_64")
92+
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|arm64")
93+
set(_key "linux_aarch64")
94+
else()
95+
set(_key "linux_x86_64")
96+
endif()
97+
if(NOT DEFINED _CKDSL_PYTHON_ASSET_${_key})
98+
message(FATAL_ERROR
99+
"CK DSL provider: no pinned python-build-standalone asset for this "
100+
"platform (key '${_key}'). Provide one via -DCKDSL_PYTHON_DIST_DIR.")
101+
endif()
102+
set(${assetOut} "${_CKDSL_PYTHON_ASSET_${_key}}" PARENT_SCOPE)
103+
set(${shaOut} "${_CKDSL_PYTHON_SHA_${_key}}" PARENT_SCOPE)
104+
endfunction()
105+
106+
# Provision CPython and steer find_package(Python3) at it.
107+
function(ck_dsl_provider_provision_python)
108+
set(_version "${_CKDSL_PYTHON_PBS_VERSION}")
109+
string(REGEX MATCH "^([0-9]+\\.[0-9]+)" _shortVersion "${_version}")
110+
111+
# 1. Caller-supplied distribution wins (air-gap / pre-stage).
112+
if(DEFINED CKDSL_PYTHON_DIST_DIR AND CKDSL_PYTHON_DIST_DIR)
113+
get_filename_component(_prefix "${CKDSL_PYTHON_DIST_DIR}" ABSOLUTE)
114+
_ck_dsl_python_validate_prefix("${_prefix}" "${_shortVersion}" _valid)
115+
if(NOT _valid)
116+
message(FATAL_ERROR
117+
"CK DSL provider: CKDSL_PYTHON_DIST_DIR='${_prefix}' is not a "
118+
"usable CPython ${_shortVersion} prefix (expected bin/, "
119+
"lib/libpython${_shortVersion}.so, include/python${_shortVersion}).")
120+
endif()
121+
message(STATUS "CK DSL provider: using supplied Python prefix ${_prefix}")
122+
else()
123+
# 2. Download + checksum-verify the pinned PBS tarball.
124+
_ck_dsl_python_select_asset(_asset _sha)
125+
include(FetchContent)
126+
message(STATUS
127+
"CK DSL provider: fetching python-build-standalone ${_version} "
128+
"(${_asset})")
129+
fetchcontent_declare(
130+
ckdsl_cpython
131+
URL "${_CKDSL_PYTHON_PBS_BASE}/${_asset}"
132+
URL_HASH SHA256=${_sha}
133+
DOWNLOAD_EXTRACT_TIMESTAMP TRUE
134+
)
135+
# The archive has no CMakeLists, so MakeAvailable only populates
136+
# (download + extract); it never tries to add_subdirectory.
137+
fetchcontent_makeavailable(ckdsl_cpython)
138+
# PBS archives have a single top-level "python/" dir, which
139+
# FetchContent strips on extraction -- so bin/ lib/ include/ land
140+
# directly in SOURCE_DIR, which is therefore the prefix.
141+
set(_prefix "${ckdsl_cpython_SOURCE_DIR}")
142+
_ck_dsl_python_validate_prefix("${_prefix}" "${_shortVersion}" _valid)
143+
if(NOT _valid)
144+
message(FATAL_ERROR
145+
"CK DSL provider: fetched python-build-standalone archive did not "
146+
"unpack to the expected prefix at ${_prefix}.")
147+
endif()
148+
endif()
149+
150+
set(CKDSL_PYTHON_PREFIX "${_prefix}" CACHE PATH
151+
"CPython prefix used by the ck-dsl-provider embedded interpreter" FORCE)
152+
set(CKDSL_PYTHON_VERSION "${_shortVersion}" CACHE STRING
153+
"CPython MAJOR.MINOR for the ck-dsl-provider embedded interpreter" FORCE)
154+
155+
# Steer find_package(Python3 ... Development) at the provisioned
156+
# prefix. LOCATION strategy + NEVER virtualenv stop a host
157+
# interpreter (incl. a uv-managed one in ~/.local) from hijacking
158+
# discovery -- the original failure the system-Python pin guarded.
159+
set(Python3_ROOT_DIR "${_prefix}" PARENT_SCOPE)
160+
set(Python3_FIND_STRATEGY "LOCATION" PARENT_SCOPE)
161+
set(Python3_FIND_VIRTUALENV "NEVER" PARENT_SCOPE)
162+
163+
message(STATUS
164+
"CK DSL provider: provisioned CPython ${_version} at ${_prefix}")
165+
endfunction()

‎dnn-providers/ck-dsl-provider/src/python/EmbeddedInterpreter.cpp‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@
99
#include <mutex>
1010
#include <string>
1111

12+
#include "ckdsl_provider_paths.h"
13+
1214
namespace py = pybind11;
1315

1416
namespace ck_dsl_provider {
@@ -18,6 +20,44 @@ namespace {
1820
std::once_flag _initFlag;
1921
std::atomic<unsigned> _initializationCount{0};
2022

23+
/// Resolve the CPython prefix the embedded interpreter should run from.
24+
///
25+
/// The interpreter is a self-contained python-build-standalone prefix
26+
/// bundled with the provider. In a dev/build tree this is the baked
27+
/// absolute path; an installed plugin will resolve it relative to its
28+
/// own .so (added in the install-staging step). Returning the baked
29+
/// path here keeps the build-tree path working today.
30+
std::string resolvePythonHome() {
31+
return std::string(kCkDslPythonHome);
32+
}
33+
34+
/// The interpreter executable within a python-build-standalone prefix.
35+
/// Layout differs by platform: Linux keeps it under bin/ (python3 is a
36+
/// version-agnostic symlink); Windows places python.exe at the prefix
37+
/// root. Used only for sys.executable provenance.
38+
/// NOTE: the Windows branch is written but not yet verified on Windows.
39+
std::string pythonExecutable(const std::string& home) {
40+
#ifdef _WIN32
41+
return home + "/python.exe";
42+
#else
43+
return home + "/bin/python3";
44+
#endif
45+
}
46+
47+
/// Set an isolated-config string field, raising on failure.
48+
void setConfigString(PyConfig& config, wchar_t** field, const std::string& value,
49+
const char* what) {
50+
PyStatus status = PyConfig_SetBytesString(&config, field, value.c_str());
51+
if (PyStatus_Exception(status) != 0) {
52+
std::string msg = std::string("EmbeddedInterpreter: failed to set ") + what;
53+
if (status.err_msg != nullptr) {
54+
msg += std::string(": ") + status.err_msg;
55+
}
56+
PyConfig_Clear(&config);
57+
throw hipdnn_plugin_sdk::HipdnnPluginException(HIPDNN_PLUGIN_STATUS_INTERNAL_ERROR, msg);
58+
}
59+
}
60+
2161
void doInitialize() {
2262
// Guard against another in-process embedder having initialised
2363
// CPython before this plugin loaded. If the interpreter is already
@@ -34,6 +74,17 @@ void doInitialize() {
3474
// directory to sys.path)
3575
// The provider's required search paths are injected explicitly
3676
// inside CompileServiceBridge::ctor after init.
77+
78+
// Pin home + executable to the bundled python-build-standalone
79+
// prefix so the bundled stdlib loads deterministically rather
80+
// than from whatever interpreter happens to host the process.
81+
// `home` drives the stdlib search; `executable` gives a correct
82+
// sys.executable pointing at the bundled interpreter.
83+
const std::string pythonHome = resolvePythonHome();
84+
setConfigString(config, &config.home, pythonHome, "PyConfig.home");
85+
setConfigString(config, &config.executable, pythonExecutable(pythonHome),
86+
"PyConfig.executable");
87+
3788
PyStatus status = Py_InitializeFromConfig(&config);
3889
PyConfig_Clear(&config);
3990
if (PyStatus_Exception(status) != 0) {

‎dnn-providers/ck-dsl-provider/templates/ckdsl_provider_paths.h.in‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,15 @@ inline constexpr std::string_view kCkDslPythonPackagePath =
2121
inline constexpr std::string_view kCkDslProviderPythonPackagePath =
2222
"@CK_DSL_PROVIDER_PYTHON_PACKAGE_PATH@";
2323

24+
// Absolute path of the bundled python-build-standalone prefix (the dir
25+
// containing bin/ lib/ include/) provisioned by cmake/CkDslPython.cmake.
26+
// The embedded interpreter sets PyConfig.home / executable from this so
27+
// the bundled stdlib loads deterministically, independent of any host
28+
// Python. This is the build-tree path; an installed plugin resolves the
29+
// prefix relative to its own .so (see EmbeddedInterpreter).
30+
inline constexpr std::string_view kCkDslPythonHome =
31+
"@CK_DSL_PYTHON_HOME@";
32+
2433
} // namespace ck_dsl_provider
2534

2635
#endif // CK_DSL_PROVIDER_PATHS_H

0 commit comments

Comments
 (0)