Cannot build MKLCPU backend with AdaptiveCpp

[nilsfriess]

Summary

Building the MKLCPU backend with AdaptiveCpp doesn't work.

Version

> git rev-parse HEAD
ca1ecc28e86ac82280f3be586bc0d56ef83a0ab3

Environment

• HW you use: Some AMD CPU
• oneMKL 2025.1 (have same issue with 2024.2)
• Ubuntu 24.04
• AdaptiveCpp built from my own fork (it's built from a very recent AdaptiveCpp commit, it additionally defines sycl::span which seems to be required somewhere in oneMKL or oneMATH, don't remember exactly where): nilsfriess/OpenSYCL@79379c5. Additionally, the fork also has a small modification in AdaptiveCpp's compiler driver to ignore the -fsycl flag (of course this is a hack and should just be fixed in oneMath).

Steps to reproduce

Clone oneMath and apply the following diff:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2826d1c..3930905 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -296,7 +296,7 @@ else()
       message(STATUS "Looking for hipSYCL")
       find_package(hipSYCL CONFIG REQUIRED)
       set(USE_ADD_SYCL_TO_TARGET_INTEGRATION true)
-      set (CMAKE_CXX_STANDARD 17)
+      set (CMAKE_CXX_STANDARD 20)
       add_library(ONEMATH::SYCL::SYCL INTERFACE IMPORTED)
     elseif(ONEMATH_SYCL_IMPLEMENTATION STREQUAL "dpc++")
       message(STATUS "Looking for dpc++")

This is just to make sure that the sycl::span implementation mentioned above (which essentially just does using sycl::span = std::span) works. Then build with

cmake .. -DCMAKE_CXX_COMPILER=acpp \ 
         -DONEMATH_SYCL_IMPLEMENTATION=hipsycl \ 
         -DENABLE_MKLCPU_BACKEND=True \ 
         -DBUILD_FUNCTIONAL_TESTS=False \ 
         -DBUILD_EXAMPLES=True \
         -DCMAKE_INSTALL_PREFIX=./install \
         -DTARGET_DOMAINS=blas \ 
         -DHIPSYCL_TARGETS=generic \
         -DENABLE_MKLGPU_BACKEND=False

Observed behavior

Configuring works, building fails: CMake output:

-- CMAKE_BUILD_TYPE: Release
-- TARGET_DOMAINS: blas
-- MKL_VERSION: 2024.2.0
-- MKL_ROOT: /opt/intel/oneapi/mkl/2024.2
-- MKL_SYCL_ARCH: intel64
-- MKL_ARCH: intel64
-- MKL_SYCL_LINK: dynamic
-- MKL_LINK: dynamic
-- MKL_SYCL_INTERFACE_FULL: intel_ilp64
-- MKL_INTERFACE_FULL: intel_ilp64
-- MKL_SYCL_THREADING: tbb_thread
-- MKL_THREADING: tbb_thread
-- MKL_MPI: intelmpi
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_scalapack_ilp64.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_cdft_core.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_intel_ilp64.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_tbb_thread.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_core.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_blacs_intelmpi_ilp64.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_blas.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_lapack.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_dft.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_sparse.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_data_fitting.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_rng.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_stats.so
-- Found /opt/intel/oneapi/mkl/2024.2/lib/libmkl_sycl_vm.so
-- Looking for hipSYCL
CMake Deprecation Warning at /home/friessn/Projects/AdaptiveCpp/build/install/lib/cmake/hipSYCL/hipsycl-config.cmake:27 (message):
  find_package(hipSYCL) is deprecated.  Use find_package(AdaptiveCpp)
  instead.  For this, you may have to set
  -DAdaptiveCpp_DIR=/install/prefix/lib/cmake/AdaptiveCpp.  Additionally,
  replace -DHIPSYCL_* cmake arguments with -DACPP_*
Call Stack (most recent call first):
  CMakeLists.txt:297 (find_package)


-- ONEAPI_DEVICE_SELECTOR will be set to the following value(s): [opencl:cpu] for run-time dispatching examples
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/friessn/Projects/oneMath/build

Output after running make:

make.log

It seems (?) that the issue is somehow related to conversions from bfloat16 but honestly I'm not too sure what's going on.

Expected behavior

It should build.

[nilsfriess]

Made some progress with these changes:

diff --git a/include/oneapi/math/types.hpp b/include/oneapi/math/types.hpp
index e30d346..db0e08c 100644
--- a/include/oneapi/math/types.hpp
+++ b/include/oneapi/math/types.hpp
@@ -28,6 +28,9 @@

 #ifndef SYCL_IMPLEMENTATION_INTEL
 #include "oneapi/math/bfloat16.hpp"
+namespace oneapi {
+  using bfloat16 = math::bfloat16;
+}
 #endif

 namespace oneapi {
diff --git a/src/blas/backends/mkl_common/mkl_blas_backend.hpp b/src/blas/backends/mkl_common/mkl_blas_backend.hpp
index 4298b35..b247268 100644
--- a/src/blas/backends/mkl_common/mkl_blas_backend.hpp
+++ b/src/blas/backends/mkl_common/mkl_blas_backend.hpp
@@ -19,6 +19,8 @@

 #pragma once

+#include "oneapi/math/types.hpp"
+
 #include <complex>

 // Intel(R) oneMKL headers
@@ -26,7 +28,6 @@
 #include <mkl/blas.hpp>

 #include "common_onemkl_conversion.hpp"
-#include "oneapi/math/types.hpp"

 namespace oneapi {
 namespace math {

Everything then builds. But running the example then gives:

./bin/example_blas_gemm_usm

########################################################################
# General Matrix-Matrix Multiplication using Unified Shared Memory Example:
#
# C = alpha * A * B + beta * C
#
# where A, B and C are general dense matrices and alpha, beta are
# floating point type precision scalars.
#
# Using apis:
#   gemm
#
# Using single precision (float) data type
#
# Device will be selected during runtime.
# The environment variable ONEAPI_DEVICE_SELECTOR can be used to specify
# available devices
#
########################################################################

Running BLAS GEMM USM example on CPU device.
Device name is: AdaptiveCpp OpenMP host device
Running with single precision real data type:
./bin/example_blas_gemm_usm: symbol lookup error: /home/friessn/Projects/oneMath/build/lib/libonemath_blas_mklcpu.so: undefined symbol: _ZN6oneapi3mkl4blas12column_major4gemmERN7hipsycl4sycl5queueENS0_9transposeES7_lllNS0_16value_or_pointerIfEEPKflSB_lS9_PflNS1_12compute_modeERKSt6vectorINS4_5eventESaISF_EE

[nilsfriess]

Just realised that this might be a duplicate of related to #567.

[nilsfriess]

Hi @Rbiessy, pinging you because you added the 'help wanted' label. As stated above, I think this issue is related to #567, but that issue hasn't gotten any comments since 9 months. I would be happy to provide more details or help fixing this, but I don't really know where to even begin.

[Rbiessy]

Hey @nilsfriess, yes sure that would be nice.
So for the issue related to -fsycl the fix suggested in the issue linked is probably good enough. For the issue regarding the undefined references it is a bit more difficult to debug. I would suggest trying again with the tip of oneMath and oneAPI 2025.1. If that's not enough the best approach is probably to reduce the issue at the maximum: I wonder if it can be reproduced outside of oneMath with a single file and using the same compiler flags.

[nilsfriess]

Here's my test file. No oneMath, just MKL:

#include <iostream>

#include <sycl/sycl.hpp>
#include <oneapi/mkl.hpp>

int main() {
  sycl::queue q{};

  const int N = 1024;
  auto *a = sycl::malloc_shared<float>(N, q);

  std::iota(a, a + N, 0);

  auto *res = sycl::malloc_shared<float>(1, q);
  oneapi::mkl::blas::column_major::asum(q, N, a, 1, res);
  q.wait();

  std::cout << "Result: " << *res << "\n";
  
  sycl::free(a, q);
}

I run source /opt/intel/oneapi/mkl/2025.1/env/vars.sh and then

> acpp -std=c++20 -o main main.cc -DMKL_ILP64  -I"${MKLROOT}/include"  -L${MKLROOT}/lib -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl

This is what the link line advisor gives me when I choose C/C++ as programming language. This results in a linker error:

main.cc:(.text+0x17f): undefined reference to `oneapi::mkl::blas::column_major::asum(hipsycl::sycl::queue&, long, float const*, long, float*, std::vector<hipsycl::sycl::event, std::allocator<hipsycl::sycl::event> > const&)'

The link line advisor doesn't allow me to choose a "SYCL domain library" with C/C++ as the language so I instead chose DPC++ and set BLAS as the "SYCL domain library". Then I get the following compile command:

> acpp -std=c++20 -o main main.cc -DMKL_ILP64  -I"${MKLROOT}/include" -L${MKLROOT}/lib -lmkl_sycl_blas -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lsycl -lpthread -lm -ldl

(Of course the link line advisor then also tells me to add the -fsycl flag, which I ignored). I get the same linker error.

[Rbiessy]

That's good to know.

Thinking about this linker error, I don't think we can use the MKL libraries with AdaptiveCpp given that they are shipped compiled with DPC++.
As far as I know there is nothing we can do about this in oneMath other than failing earlier in CMake to report that DPC++ is required to use the mklcpu or mklgpu backends.

[nilsfriess]

But then I don't understand how the MKLCPU backend has ever worked with AdaptiveCpp? And I'm pretty sure it has, at least there are a few PRs (e.g. #191) that explicitly fix issues of the MKLCPU backend in combination with hipSYCL (=AdaptiveCpp).

[Rbiessy]

I'm not sure either tbh. The fact that the linker is looking for a symbol with hipsycl::sycl::queue makes me think that the header compiled with AdaptiveCpp declares a symbol with this namespace for some reason but none of the mkl .so can provide this symbol since it was compiled with DPC++ (i.e. a symbol with sycl::queue).
Maybe this used to work only with older versions of AdaptiveCpp? @illuhad would know best.

[illuhad]

Maybe this used to work only with older versions of AdaptiveCpp? @illuhad would know best.

The only thing I can say here that we haven't changed anything here on the AdaptiveCpp side. Our queue has always been defined as hipsycl::sycl::queue. So if MKL has a SYCL interface that is compiled with DPC++, then I can't see how this ever worked. AdaptiveCpp and DPC++ runtimes have never been ABI compatible.

I've never been aware of the AdaptiveCpp CPU backend working with MKL, but maybe I just missed that.

Question: Which programming models does MKL support?

• If there's e.g. also an OpenCL interface, then we could leverage that with AdaptiveCpp, and then at least get its OpenCL CPU backend working
• If there's a native C++ interface (i.e. without enqueuing anything), we could call that from inside a single_task kernel in the OpenMP backend since AdaptiveCpp supports more or less arbitrary C++ in there.

[nilsfriess]

I've never been aware of the AdaptiveCpp CPU backend working with MKL, but maybe I just missed that.

I guess you're right and it never worked. Seems like the entry in the support table in the README was accidentally added in #144.

[Rbiessy]

Question: Which programming models does MKL support?

If there's e.g. also an OpenCL interface, then we could leverage that with AdaptiveCpp, and then at least get its OpenCL CPU backend working
If there's a native C++ interface (i.e. without enqueuing anything), we could call that from inside a single_task kernel in the OpenMP backend since AdaptiveCpp supports more or less arbitrary C++ in there.

MKL has a C and Fortran interface for CPU only and SYCL for CPU and GPU (see here for reference). I think the short answer is oneMath can't support the MKL backends for SYCL implementations other than DPC++.