Point hipconfig's ROCM_PATH towards dev packages if develop packages are installed (#2870)

PR based on #2428. 
## Motivation

We'd like `hipconfig --rocmpath` to point towards `_rocm_sdk_devel` if
the `rocm[devel]` packages are installed. This is useful for code that
depends on `hipconfig --rocmpath` to resolve paths for ROCm files. Aimed
as a fix for #1880 and other
PyTorch issues that require the develop packages.
<!-- Explain the purpose of this PR and the goals it aims to achieve.
-->

## Technical Details

- The PR at #2428 successfully
directed `hipconfig --rocmpath` towards the `_rocm_sdk_devel` packages
on Windows but was running into some issues on Linux as we were still
using symlinks.
- Added a change so Linux also uses hardlinks to avoid this issue.
<!-- Explain the changes along with any relevant GitHub links. -->

## Test Plan
Linux:

- The unit test `testCLIUsesDevelRootPath` passes (`rocm-sdk test`
passes)
- Tested by installing without the devel packages and with the devel
packages
```
pip install --index-url https://rocm.devreleases.amd.com/v2-staging/gfx1151/ "rocm[libraries]==7.11.0.dev0+5d4dfe88517275760144e58af2e70af3dff5d2bc"
hipconfig --rocmpath
# /home/rocm/darren/hipconfig-test/.venv/lib/python3.12/site-packages/_rocm_sdk_core
pip install --index-url https://rocm.devreleases.amd.com/v2-staging/gfx1151/ "rocm[libraries,devel]==7.11.0.dev0+5d4dfe88517275760144e58af2e70af3dff5d2bc"
hipconfig --rocmpath
# /home/rocm/darren/hipconfig-test/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
```

<!-- Explain any relevant testing done to verify this PR. -->

## Test Result

On Linux, the unit test passes and without the develop packages
`hipconfig --rocmpath` correctly points at the core packages. When the
develop packages are installed, it correctly expands and points tot he
develop packages.
<!-- Briefly summarize test outcomes. -->

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

---------

Co-authored-by: Scott Todd <scott.todd0@gmail.com>
Co-authored-by: darren-amd <darren-amd@users.noreply.github.com>

Author: darren-amd

build_tools/packaging/python/templates/rocm-sdk-core/src/rocm_sdk_core/_cli.py

@@ -4,6 +4,7 @@
 """Trampoline for console scripts."""
 
 import importlib
+import importlib.util
 import os
 import platform
 import sys
@@ -12,17 +13,95 @@
 from ._dist_info import ALL_PACKAGES
 
 CORE_PACKAGE = ALL_PACKAGES["core"]
-PLATFORM_NAME = CORE_PACKAGE.get_py_package_name()
-PLATFORM_MODULE = importlib.import_module(PLATFORM_NAME)
-# NOTE: dependent on there being an __init__.py in the platform package.
-PLATFORM_PATH = Path(PLATFORM_MODULE.__file__).parent
+CORE_PY_PACKAGE_NAME = CORE_PACKAGE.get_py_package_name()
+
+
+def _get_core_module_path():
+    # NOTE: dependent on there being an __init__.py in the core package.
+    core_module = importlib.import_module(CORE_PY_PACKAGE_NAME)
+    return Path(core_module.__file__).parent
+
+
+DEVEL_PACKAGE = ALL_PACKAGES["devel"]
+DEVEL_PURE_PY_PACKAGE_NAME = DEVEL_PACKAGE.pure_py_package_name
+DEVEL_PY_PACKAGE_NAME = DEVEL_PACKAGE.get_py_package_name()
+
+
+def _has_devel_module():
+    return importlib.util.find_spec(DEVEL_PURE_PY_PACKAGE_NAME) is not None
+
+
+def _is_devel_module_expanded():
+    return importlib.util.find_spec(DEVEL_PY_PACKAGE_NAME) is not None
+
+
+def _expand_devel_module():
+    import subprocess
+
+    try:
+        subprocess.check_call([sys.executable, "-m", "rocm_sdk", "init", "--quiet"])
+    except subprocess.CalledProcessError:
+        print(
+            "ERROR: Failed to expand rocm[devel] package. "
+            "Try running `rocm-sdk init` manually for details.",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+
+def _get_devel_module_path():
+    # NOTE: dependent on there being an __init__.py in the devel package.
+    try:
+        devel_module = importlib.import_module(DEVEL_PY_PACKAGE_NAME)
+    except ImportError:
+        print(
+            "WARNING: Failed to import devel module, falling back to core.",
+            file=sys.stderr,
+        )
+        return _get_core_module_path()
+    if devel_module.__file__ is None:
+        print(
+            "WARNING: Devel module has no __file__, falling back to core.",
+            file=sys.stderr,
+        )
+        return _get_core_module_path()
+    return Path(devel_module.__file__).parent
+
+
+def _get_module_path(expand_devel: bool) -> Path:
+    """Gets the module path, either from 'core' or 'devel'.
+
+    If the 'devel' package IS NOT installed then 'core' is used, ignoring the input of `expand_devel`.
+    If the 'devel' package IS installed AND already expanded then it is used.
+    If the 'devel' package IS installed AND NOT already expanded then either
+      A) System information tools like amd-smi can choose to run more quickly
+         with 'core' by skipping the (compute-intensive) 'devel' expansion.
+         These tools should pass `expand_devel=False`.
+      B) Other tools that benefit from the extra files in the 'devel' package
+         will expand expand it by passing `expand_devel=True`.
+    """
+    if _has_devel_module():
+        if _is_devel_module_expanded():
+            return _get_devel_module_path()
+        elif expand_devel:
+            _expand_devel_module()
+            return _get_devel_module_path()
+        else:
+            # Passthrough. Fallback to core module.
+            pass
+
+    return _get_core_module_path()
+
 
 is_windows = platform.system() == "Windows"
 exe_suffix = ".exe" if is_windows else ""
 
 
-def _exec(relpath: str):
-    full_path = PLATFORM_PATH / (relpath + exe_suffix)
+def _exec(relpath: str, expand_devel=True):
+    # Default is True because most CLI tools are compiler/build tools that
+    # need the devel files. System info tools (amd-smi, rocminfo, etc.)
+    # override with expand_devel=False to avoid the expansion cost.
+    full_path = _get_module_path(expand_devel) / (relpath + exe_suffix)
     os.execv(full_path, [str(full_path)] + sys.argv[1:])
 
 
@@ -51,7 +130,7 @@ def amdlld():
 
 
 def amd_smi():
-    _exec("bin/amd-smi")
+    _exec("bin/amd-smi", expand_devel=False)
 
 
 def hipcc():
@@ -71,23 +150,23 @@ def hipify_perl():
 
 
 def hipInfo():
-    _exec("bin/hipInfo")
+    _exec("bin/hipInfo", expand_devel=False)
 
 
 def offload_arch():
     _exec("lib/llvm/bin/offload-arch")
 
 
 def rocm_agent_enumerator():
-    _exec("bin/rocm_agent_enumerator")
+    _exec("bin/rocm_agent_enumerator", expand_devel=False)
 
 
 def rocm_info():
-    _exec("bin/rocminfo")
+    _exec("bin/rocminfo", expand_devel=False)
 
 
 def rocm_smi():
-    _exec("bin/rocm-smi")
+    _exec("bin/rocm-smi", expand_devel=False)
 
 
 def roccoremerge():

build_tools/packaging/python/templates/rocm/src/rocm_sdk/__main__.py

@@ -45,9 +45,10 @@ def _do_init(args: argparse.Namespace):
         # if contents for development were not yet unpacked.
         root_path = _devel.get_devel_root()
     except ModuleNotFoundError as e:
-        print(f"ERROR: {e}", file=sys.stderr)
+        print(f"ERROR running init: {e}", file=sys.stderr)
         sys.exit(1)
-    print(f"Devel contents expanded to '{root_path}'")
+    if not args.quiet:
+        print(f"Devel contents expanded to '{root_path}'")
 
 
 def _do_test(args: argparse.Namespace):
@@ -147,6 +148,11 @@ def main(argv: list[str] | None = None):
     init_p = sub_p.add_parser(
         "init", help="Expand devel contents to initialize rocm[devel]"
     )
+    init_p.add_argument(
+        "--quiet",
+        action="store_true",
+        help="Run without producing output",
+    )
     init_p.set_defaults(func=_do_init)
 
     args = p.parse_args(argv)

build_tools/packaging/python/templates/rocm/src/rocm_sdk/_devel.py

@@ -198,17 +198,25 @@ def _clean_dir(dir: Path):
                             #   hash (empty)
                             #   size (empty)
                             record_file.write(f"{ti.name},,\n")
-                        if _is_windows() and ti.issym():
-                            # Convert symlinks into hardlinks on Windows.
-                            # This saves disk space while improving compatibility
-                            # on systems without as robust symlink support.
+                        if ti.issym():
+                            # Convert file symlinks into hardlinks on all platforms.
+                            # This saves disk space while improving compatibility.
+                            # On Windows: symlinks require admin privileges.
+                            # On Linux: native binaries that use readlink(/proc/self/exe)
+                            #   to determine their location will resolve symlinks and
+                            #   report the wrong path (e.g., _rocm_sdk_core instead of
+                            #   _rocm_sdk_devel). Hardlinks avoid this issue.
                             # As needed, we could also generate tarfiles with
                             # copies instead of symlinks, at the cost of disk space.
-                            # Creating symlinks on Windows also requires admin privileges.
                             parent_path.mkdir(parents=True, exist_ok=True)
                             symlink_target = ti.linkname
                             hardlink_target = dest_path.parent / symlink_target
-                            dest_path.hardlink_to(hardlink_target)
+                            # Only create hardlinks for files, not directories
+                            if hardlink_target.is_file():
+                                dest_path.hardlink_to(hardlink_target)
+                            else:
+                                # For directory symlinks, extract as normal
+                                tf.extract(ti, path=site_lib_path)
                         else:
                             tf.extract(ti, path=site_lib_path)
                     elif ti.isdir():

build_tools/packaging/python/templates/rocm/src/rocm_sdk/tests/devel_test.py

@@ -66,6 +66,30 @@ def testCLIPathRoot(self):
         bin_path = path / "bin"
         self.assertTrue(bin_path.exists(), msg=f"Expected bin path {bin_path} to exist")
 
+    def testCLIUsesDevelRootPath(self):
+        root_path_output = (
+            utils.run_command(
+                [sys.executable, "-m", "rocm_sdk", "path", "--root"], capture=True
+            )
+            .decode()
+            .strip()
+        )
+        root_path = Path(root_path_output)
+
+        # CLI scripts by default run from _rocm_sdk_core.
+        # When the devel package is installed they should run from _rocm_sdk_devel.
+        rocmpath_output = (
+            utils.run_command(["hipconfig", "--rocmpath"], capture=True)
+            .decode()
+            .strip()
+        )
+        rocmpath = Path(rocmpath_output)
+        self.assertEqual(
+            root_path,
+            rocmpath,
+            msg=f"Expected `hipconfig --rocmpath` to return {root_path}, not {rocmpath}",
+        )
+
     @unittest.skipIf(
         platform.system() == "Windows", "root LLVM symlink only exists on Linux"
     )

docs/packaging/python_packaging.md

@@ -25,10 +25,12 @@ We generate three types of packages:
   a relative binary).
 - Devel package: The `rocm-sdk-devel` package is the catch-all for everything.
   For any file already populated in a runtime package, it will include it as
-  a relative symlink (also rewriting shared library soname links as needed).
-  Since symlinks and non-standard attributes cannot be included in a wheel file,
-  the platform contents are stored in a `_devel.tar` or `_devel.tar.xz` file.
-  The installed package is extended in response to requesting a path to it
+  a relative symlink in the tarball. During extraction, file symlinks are
+  converted to hardlinks to improve compatibility, while directory symlinks
+  remain as symlinks. Shared library soname links are also rewritten as needed.
+  Since symlinks/hardlinks and non-standard attributes cannot be included in a
+  wheel file, the platform contents are stored in a `_devel.tar` or `_devel.tar.xz`
+  file. The installed package is extended in response to requesting a path to it
   via the `rocm-sdk` tool.
 
 Runtime packages can either be target neutral or target specific. Target specific