AMDGPU downgrades Waterlily

[SimonDanisch]

I wanted to try out Waterlily on an AMD gpu with AMDGPU.jl, but adding AMDGPU to the project downgrades Waterlily significantly:

Any help to run this on my 7900xtx would be appreciated :)

[b-fg]

Hi Simon, thanks for reporting this. What ROCm version do you have installed in your system? I was recently able to run WaterLily on an AMD GPU, but because the ROCm version in that system was quite low (5.1.1) I had to force AMDGPU.jl to use its 0.4.15 version. Maybe something similar to this is happening?
Also, we could force WaterLily v1.1 to compile only upstream of a certain AMDGPU.jl version.
And it would also be helpful to see the full log of what packages changed versions when using AMDGPU (if there were more than just WaterLily) - maybe there is an incompatibility with another package.

[SimonDanisch]

That wasn't the reason, I just updated all my drivers including rocm (5.7 + 6.1). I also don't think Pkg can even install different AMDGPU version based on available drivers, if I'm not mistaken.

You can actually reproduce it with this minimal setup (Julia 1.10.4):

]activate --temp
]add AMDGPU@0.9.6 WaterLily@1.1.0

[b-fg]

What happens when you try to not specify the version, ie

]activate --temp
]add AMDGPU WaterLily

This works fine on my system, even though it installs AMDGPU v0.6.1. So maybe we should update our version compatibilities.

[b-fg]

Can you please try my temporary fix in the fix_compat branch and report back?

]
activate --temp
add WaterLily#fix_compat
add AMDGPU@0.9.6

[SimonDanisch]

Yes, this seems to work :)

[b-fg]

Good! I will create the PR and (re)activate some of the compatibilities. If that still works, then we can merge it :)

[SimonDanisch]

It does not work though :(
I'm getting lots of:

Reason: unsupported call through a literal pointer (call to jl_gc_run_pending_finalizers)
Reason: unsupported call to an unknown function (call to ijl_pop_handler)

I guess some kernel isn't setup correctly, although I would expect CUDA.jl to also caugh on something like jl_gc_run_pending_finalizers ....
Do you have an example that you have tested with AMDGPU that runs fine?

[b-fg]

You can try the following MWE

using WaterLily
using AMDGPU

function tgv(p, backend; Re=1600, T=Float32)
    L = 2^p; U = 1; κ=π/L; ν = 1/(κ*Re)
    function uλ(i,xyz)
        x,y,z = @. xyz/L*π                # scaled coordinates
        i==1 && return -U*sin(x)*cos(y)*cos(z) # u_x
        i==2 && return  U*cos(x)*sin(y)*cos(z) # u_y
        return 0.                              # u_z
    end
    Simulation((L, L, L), (0, 0, 0), 1/κ; U=U, uλ=uλ, ν=ν, T=T, mem=backend)
end

function main()
    sim = tgv(5, ROCArray)
    sim_step!(sim)
end

main()

I can correctly run this using ROCm/5.1.1, WaterLily.jl/1.1, AMDGPU.jl/0.4.15 on a Radeon Instinct MI50 32GB.

[SimonDanisch]

That actually kills the julia session 😓
AMDGPU@0.9.6, WaterLily v1.1.0 and ROCm/6.1 on a 7900xtx.
The AMDGPU test seem to be getting much further on these new versions than any before!

[b-fg]

That's unfortunate. This week I will get access to LUMI, which I presume has a more updated ROCm version. I will look into it and let you know how that goes.

[weymouth]

The PR doesn't actually fix this issue.