Skip to content

Conversation

@victorapm
Copy link
Contributor

Add support to hypre solvers running on GPUs (NVIDIA, AMD and Intel)
GPU support can be controlled via input file with the variable HYPRE_DEVICE_RUN. In case hypre is not compiled with GPU support, the solvers will run on CPU
FDS+hypre build via CMake is ready, however the Makefile build needs work

@rmcdermo
Copy link
Contributor

@victorapm Thank you! Let me have a chance to test these cases on our cluster and then we can merge.

@rmcdermo rmcdermo self-assigned this Jan 28, 2025
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Victor for this PR! In this file and line
&PRES CHECK_POISSON=T, SOLVER='UGLMAT HYPRE', HYPRE_DEVICE_RUN=T, MAX_PRESSURE_ITERATIONS=1 /

Do we really need to set HYPRE_DEVICE_RUN? In cons.f90 it is already set to true, and if there is no DEVICE it will default to the CPU solver. I'm thinking we might not need to define HYPRE_DEVICE_RUN and have this as an input parameter, unless you can think of a reason to have it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Marcos, I agree it's unnecessary setting the variable in this case. The reason I put it there is to give an example of where this variable should be set in case someone builds hypre with GPU support but still wants to run it on CPUs. Then, in this case, HYPRE_DEVICE_RUN should be set to false. But, feel free to change any logic here as this is your code :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks victor, we'll test it here. The other question is, what HYPRE version do we need to work with the new routines added here (i.e. HYPRE_IJVECTORINITIALIZE_V2)?

Copy link
Contributor

@marcosvanella marcosvanella left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question that came up in terms of the coarsening/relaxation schemes defined in imkl.f90:

HYPRE_PRECOND_COARSENINGTYPE
HYPRE_PRECOND_SETRELAXTYPE

For CPUs we had 10 and 8 and I see in your PR you changed them to 8 and 18. Should we have different values if HYPRE is being run in GPU or CPU?

@rmcdermo
Copy link
Contributor

@victorapm This is not compiling with hypre 2.32.0, which looks to me like the current release. I get an error in undefined reference to hypre_ijmatrixinitialize_v2_'`.

Let me suggest that if we need a development branch of hypre that this should live in Marcos's FireX branch.

@rmcdermo rmcdermo closed this Jan 28, 2025
# Currently we need to refer directly to a commit which contains a
# patch to handle HYPRE_FMANGLE
GIT_TAG 6d01417697aa84d118eb1e1eb91315df42575be5 # v${HYPRE_GIT_VERSION}
GIT_TAG 63331f19c72b2ec363824907c0575a6edf7bbf3c # From: 01/27/25 # v${HYPRE_GIT_VERSION}
Copy link
Contributor Author

@victorapm victorapm Jan 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marcosvanella @rmcdermo this is the minimum required version of hypre for this to work. This commit hash is located at hypre's master branch and will be added in the upcoming release

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@victorapm OK, understood. Thanks! We can add to FDS master branch once there is a new hypre release.

@victorapm
Copy link
Contributor Author

For CPUs we had 10 and 8 and I see in your PR you changed them to 8 and 18. Should we have different values if HYPRE is being run in GPU or CPU?

@marcosvanella Yes, we could have different defaults for CPU and GPU builds. Even though the new default leads to very similar performance as the old one (in terms of execution time)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants