Skip to content

Latest commit

 

History

History
137 lines (92 loc) · 10.1 KB

development_guide.md

File metadata and controls

137 lines (92 loc) · 10.1 KB

TheRock Development Guide

While TheRock started life as a super-project for doing combined builds and releases of individual ROCm components, it has increasingly been used as an entrypoint for developing ROCm components as well, since it provides a stable, single entrypoint to iterate on any ROCm sub-project. This guide collects notes and procedures aimed at ROCm developers who wish to use TheRock as a development environment.

While there is much overlap between using TheRock as a development environment and as a CI/release platform, this document is geared at exposing features and techniques specifically targeted at making ROCm developers more productive. Since development features and approaches are built on an as-needed basis, please consider this a working document that presents approaches that have worked for core developers.

IDE Support

Once configured, the project outputs a combined compile_commands.json for all configured project components. This means that if opened in IDEs such as VSCode, with an appropriate C++ development extension, code assistance should be available project wide. Since the project is quite large, this can add a significant overhead to your development machine, and we are still gathering experience on the best way to optimize this powerful feature.

Single Component Development

When working on a single component, TheRock can be configured to build only a desired subset, allowing rapid iteration once the initial configure/build is done. See the Feature Flags in the main project README or the corresponding documentation in the top-level CMakeLists.txt for help in selecting a subset. As an example, this section will use the HIPIFY component to demonstrate the basic flow.

Initial build

First, build TheRock as usual but requesting only a certain component. This is done with CMake arguments like:

-DTHEROCK_ENABLE_ALL=OFF -DTHEROCK_ENABLE_HIPIFY=ON
# Note that if modifying an existing build, you may also want to add
# -DTHEROCK_RESET_FEATURES=ON in order to reset any existing enable/disable
# flags to the new settings.

TODO: Add CMake preset files for useful configurations in order to make this more ergonomic.

Build directory layout

The hipify sources are in the compiler/hipify directory and the outer build target and CMake flags are contained in the compiler/CMakeLists.txt file. Once the initial configure is done, you will see a corresponding directory in your build directory. At the time of writing, this directory looks something like this:

$ ls -lh
total 8.0K
drwxrwxr-x 1 stella stella  272 Mar 11 19:38 build
drwxrwxr-x 1 stella stella   26 Mar 11 19:38 dist
-rw-rw-r-- 1 stella stella 3.8K Mar 11 14:56 _init.cmake
drwxrwxr-x 1 stella stella   20 Mar 11 19:38 stage
drwxrwxr-x 1 stella stella   94 Mar 11 19:38 stamp
-rw-rw-r-- 1 stella stella  465 Mar 11 14:56 _toolchain.cmake

These entries are explained here:

  • build: The CMake build for the component. You can cd into this directory and do anything you want once the project is initially built.
  • stage: Standalone install directory for the component. Each component is installed in isolation into its dedicated stage directory without any of its dependencies. This is kept as a pristine install tree and is never intermixed with other components. As such, the contents may not actually work (since they will be missing runtime deps, etc) but keeping this isolated aids in packaging, caching, pre-builds, etc.
  • dist: The stage tree is combined with the stage tree of all of its transitive runtime dependencies, resulting in the minimum surface-area directory in which the installed component should be able to be used. Note that hard-links are used between the stage and dist directories, so maintaining these copies should be efficient. Be careful about changes propagating, though.
  • stamp: This directory contains "stamp" files which help the super-project know when a component needs to be reconfigured or rebuilt. Deleting the stamp files is one way to get around issues where the super-project is not detecting sub-project changes and is therefore not incrementally building.
  • _init.cmake: This file is generated by the super-project and CMake will include it upon encountering the first project command in the component. This includes a variety of customizations to the build, enabling it to properly find dependencies, set up program/linker paths, etc. It also triggers inclusion of the pre_hook_${component}.cmake and post_hook_${component}.cmake files, which are used to further customize individual projects as part of the overall build.
  • _toolchain.cmake: This file is generated by the super-project and set as the CMAKE_TOOLCHAIN_FILE for the component. This contains overrides for CMake settings that influence the compiler toolchain. Different components will have different toolchain settings (i.e. HIP projects will reference the in-tree built HIP compiler, whereas non-HIP projects will configure the host toolchain).

Option 1: Work directly on the project

For developers who just want to use TheRock to bootstrap their development activity, after performing an initial build, they can just go into the build directory for their component and issue ninja and cmake commands as if it was any normal CMake project. The super-project will set everything up to defaults and the developer can make any local changes desired and iterate.

If ever needing to reset and start over (using hipify as the component name), just ninja hipify+expunge followed by ninja hipify. You can also choose to just run the configure or build stage with ninja hipify+configure or ninja hipify+build as desired.

If you blow away the entire component build directory, you may need to reconfigure the super-project (i.e. cmake . in the super-project build directory).

Option 2: Drive from the super-project

If not doing deep development on the component, it is often effective to just use the targets exposed for each component at in the super-project and never touch the component build directory.

This can be done (using hipify as the example component name) with commands like this:

ninja hipify+expunge && ninja hipify

If only changing source files and not wanting to blow away the configured build directory, the expunge step is not needed: that basically deletes all intermediates and forces a reconfigure/rebuild and is the most effective way to "start over" at a component level.

Source Management

Every component is a submodule (not strictly true: there are some small compatibility components that are directly included in TheRock repo). Use normal git tools for adding remotes, switching branches, applying patches, etc.

An outer level execution of ./build_tools/fetch_sources.py will reset all submodule state and re-apply any local patches. If this results in loss of local commits, they can typically be found in the component's git reflog. It should not be possibly to truly lose uncommitted work, but re-fetching sources will effectively git reset --hard.

Developer CMake Flags

CMake System Flags

The following CMake flags are mirrored to component projects (see THEROCK_DEFAULT_CMAKE_VARS in therock_subproject.cmake):

  • CMAKE_BUILD_TYPE
  • CMAKE_C_VISIBILITY_PRESET
  • CMAKE_CXX_VISIBILITY_PRESET
  • CMAKE_PROGRAM_PATH
  • CMAKE_PLATFORM_NO_VERSIONED_SONAME
  • Python3_EXECUTABLE
  • Python3_FIND_VIRTUALENV
  • THEROCK_SOURCE_DIR
  • THEROCK_BINARY_DIR
  • ROCM_SYMLINK_LIBS

In addition, if a component is using the host toolchain, the following are mirrored:

  • CMAKE_C_COMPILER
  • CMAKE_CXX_COMPILER
  • CMAKE_LINKER
  • CMAKE_C_COMPILER_LAUNCHER
  • CMAKE_CXX_COMPILER_LAUNCHER
  • CMAKE_C_FLAGS
  • CMAKE_CXX_FLAGS
  • CMAKE_EXE_LINKER_FLAGS
  • CMAKE_SHARED_LINKER_FLAGS
  • CMAKE_MSVC_DEBUG_INFORMATION_FORMAT
  • CMAKE_INSTALL_LIBDIR

If building with the in-tree HIP compiler, CMAKE_C_COMPILER, CMAKE_CXX_COMPILER, CMAKE_LINKER, AMDGPU_TARGETS, GPU_TARGETSandCMAKE_CXX_FLAGS_INIT` will be set accordingly.

Additional Developer Ergonomic Flags

We add developer ergonomic flags as needed in order to support project-wide development activities. This currently includes:

  • THEROCK_VERBOSE: If true, enables much more verbose CMake output. This is especially useful for diagnosing dependency and component project setup issues.
  • THEROCK_BUNDLE_SYSDEPS: Defaults to true on Linux, resulting in the use of in-tree built dependencies for most system packages. You can build with actual system packages vs locally built versions by disabling this.

In addition, the following environment variables effect the project:

  • THEROCK_INTERACTIVE: Sets up all build actions for USES_TERMINAL. This disables all background building and causes all configure/build steps to stream to the console. This can be useful for debugging tricky issues, but it is usually more convenien to just look at the log files under build/logs/. (TODO: Change this to a CMake cache variable as it is somewhat unwieldy as an environment variable).

Additional Build Targets

Top-level targets:

  • therock-archives: Generates .tar.xz archives for each artifact in the build. These can be used for uploading to CI systems, bootstrapping later builds, or packaging.
  • therock-dist: Builds all components and materializes the build/dist/rocm unified tree from all artifacts. This is setup for ALL, so usually just building with no arguments is sufficient.

Per-component targets:

Each component build consists of stages: configure > build > stage > dist. These can be invoked directly. The component target just builds the dist stage, which includes everything.

  • component+build: Runs the component's build stage.
  • component+configure: Runs the component's configure stage.
  • component+expunge: Completely removes all intermediate files for the component.
  • component+stage: Performs component installation to the stage/ directory.
  • component+dist: Assembles the component's dist/ directory from its stage/ directory and the stage/ directories of all of its runtime dependencies.