oneAPI 2022.2.0 Hangs in CI

[ax3l]

Since the update 1 week ago from 2022.1.0 to 2022.2.0, most CI runs using either ICX (host) or DPC++ (device) compiles hang.

It looks like this is from the linking part: #3421 (comment)
The same problem appears in the AMReX CI.

Open this issue for triage and tracking.

cc @rscohn2

[ax3l]

@WeiqunZhang commented for AMReX:

have seen it takes half hour to compile a single function, whereas linking is really fast.
That was in embedded boundaries (EB).

So likely not the linker but some complexity in the compiler / optimizer exploded?

[ax3l]

Search for origin of that DPC++ does not like to compile fast anymore in AMReX-Codes/amrex#2969

[WeiqunZhang]

Here is how to reproduce the compile time issue.

$ git clone -b reflux_compile_time https://github.com/WeiqunZhang/amrex
$ cd amrex/Tests/LinearSolvers/EBTensor
$ make -j8

The function that's very slow to compile is amrex/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_reflux.cpp. For the original version, it took about half an hour to compile that file. I have reduced that function to the minimum for demonstration. Now it takes about 15 minutes. If I comment out either line 91 or 96, the compile time is much shorter. What's interesting is it's the same function template used in the two places. One in a device kernel, the other in a cpu kernel. https://github.com/WeiqunZhang/amrex/blob/049fbc7a9ac2a933e07cf666c487ee0bd81cc206/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_reflux.cpp#L88

[ax3l]

We will apply a work-around from AMReX via #3444
We can now not compile CPU and DPC++ GPU kernels at the same time.

For demonstrating the compiler regression in DPC++, please use @WeiqunZhang 's work-around posted above and an AMReX version prior to this PR: AMReX-Codes/amrex#2969

[ax3l]

Even with the update in #3444 merged, it still looks like oneAPI builds forever