MOOSE with HIP

[ljeft]

Check these boxes if you have followed the posting rules.

• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (avoid screenshots if possible, triple back quotes before/after pasted text, etc)

Issue or question about MOOSE

I have successfully installed MOOSE once before through CONDA, I want to install MOOSE with HIP support now, I have installed petsc with HIP support now. But don't know how to make MOOSE use this petsc directly. At the same time, the petsc I installed is less STRUMPACK than the petsc MOOSE downloaded by default. I wonder how this will affect my subsequent work? Or is the STRUMPACK a must for MOOSE?

Diagnostics for MOOSE installation issues

No response

[GiudGiud]

Hello

strumpack is not necessary for MOOSE
You can specify the PETSC_DIR then follow instructions for HPC installs of moose starting at libmesh
https://mooseframework.inl.gov/getting_started/installation/hpc_install_moose.html
Guillaume

[roystgnr]

Does grep PETSC_USE /opt/petsc-3.20.3/*/include/petscconf.h show a 64BIT_INDICES entry?

[ljeft]

Yes, it's

root@4f98d93f0617:/opt# grep PETSC_USE /opt/petsc-3.20.3/*/include/petscconf.h
#define PETSC_USE_64BIT_INDICES 1
#define PETSC_USE_AVX512_KERNELS 1
#define PETSC_USE_BACKWARD_LOOP 1
#define PETSC_USE_CTABLE 1
#define PETSC_USE_DMLANDAU_2D 1
#define PETSC_USE_INFO 1
#define PETSC_USE_ISATTY 1
#define PETSC_USE_LOG 1
#define PETSC_USE_MALLOC_COALESCED 1
#define PETSC_USE_PROC_FOR_SIZE 1
#define PETSC_USE_REAL_DOUBLE 1
#define PETSC_USE_SHARED_LIBRARIES 1
#define PETSC_USE_SINGLE_LIBRARY 1
#define PETSC_USE_SOCKET_VIEWER 1
#define PETSC_USE_VISIBILITY_C 1
#define PETSC_USE_VISIBILITY_CXX 1

I just realized that both times I downloaded PETSc I followed the Quick Start Tutorial, so I didn't set any PETSC_ARCH.

[lindsayad]

Have you tried running the update libmesh script after setting your PETSC_ARCH? Looks like it should be export PETSC_ARCH=arch-linux-c-opt

[ljeft]

Yes I tried to update PETSC_ARCH this is where I run cd ~/projects/moose/test /run_tests -j 6 gives an error, why do I have to add -vec_type cuda ?

transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Working Directory: /opt/projects/moose/test/tests/transfers/multiapp_variable_value_sample_transfer
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Running command: /opt/projects/moose/test/moose_test-opt -i parent.i --error --error-override
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Framework Information:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: MOOSE Version:           unknown
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: LibMesh Version:         
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: PETSc Version:           3.21.0
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: SLEPc Version:           3.21.0
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Current Time:            Mon Jul 29 10:06:33 2024
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Executable Timestamp:    Mon Jul 29 09:46:49 2024
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Parallelism:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Num Processors:          1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Num Threads:             1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Mesh: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Parallel Type:           replicated
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Mesh Dimension:          2
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Spatial Dimension:       2
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Nodes:                   144
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Elems:                   121
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Num Subdomains:          1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Nonlinear System:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Num DOFs:                144
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Num Local DOFs:          144
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Variables:               "u" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Finite Element Types:    "LAGRANGE" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Approximation Orders:    "FIRST" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Execution Information:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Executioner:             Transient
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   TimeStepper:             ConstantDT
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   TimeIntegrator:          ImplicitEuler
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   Solver Mode:             Preconditioned JFNK
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:   PETSc Preconditioner:    hypre boomeramg 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: Parallelism:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Num Processors:          1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Num Threads:             1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: Mesh: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Parallel Type:           replicated
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Mesh Dimension:          2
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Spatial Dimension:       2
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Nodes:                   121
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Elems:                   100
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Num Subdomains:          1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: Nonlinear System:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Num DOFs:                121
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Num Local DOFs:          121
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Variables:               "u" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Finite Element Types:    "LAGRANGE" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Approximation Orders:    "FIRST" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: Auxiliary System:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Num DOFs:                121
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Num Local DOFs:          121
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Variables:               "from_parent" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Finite Element Types:    "LAGRANGE" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Approximation Orders:    "FIRST" 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: Execution Information:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Executioner:             Transient
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   TimeStepper:             ConstantDT
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   TimeIntegrator:          ImplicitEuler
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   Solver Mode:             Preconditioned JFNK
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0:   PETSc Preconditioner:    hypre boomeramg 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub0: Time Step 0, time = 0
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub1: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: sub1: Time Step 0, time = 0
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Time Step 0, time = 0
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Time Step 1, time = 1, dt = 1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:  0 Nonlinear |R| = 3.240370e+00
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test:       0 Linear |R| = 3.240370e+00
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Launch params (512, 1, 1) are larger than launch bounds (256) for kernel _Z31hypre_BoomerAMGCreateS_rowcountRPviddPdPiS2_S1_S2_S2_S2_S2_iS2_S2_S2_S2_ please add __launch_bounds__ to kernel define or use --gpu-max-threads-per-block recompile program ! 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: Invalid argument
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to enable PETSc device support, for example, in some cases, -vec_type cuda
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR:   Option left: name:--error (no value) source: command line
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR:   Option left: name:--error-override (no value) source: command line
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR:   Option left: name:-i value: parent.i source: command line
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR:   Option left: name:-ksp_converged_reason value: ::failed source: code
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR:   Option left: name:-snes_converged_reason value: ::failed source: code
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: Petsc Release Version 3.21.0, unknown 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: /opt/projects/moose/test/moose_test-opt on a arch-linux-c-opt named e9a45d4c9858 by root Mon Jul 29 10:06:33 2024
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --with-hip=1 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-superlu_dist=1 --with-hipc=/opt/dtk-23.10/bin/hipcc --with-hip-arch=gfx906 --download-fblaslapack=/opt/petsc/v3.4.2-p3.tar.gz --download-metis=/opt/petsc/v5.1.0-p11.tar.gz --download-parmetis=/opt/petsc/v4.0.3-p9.tar.gz --download-ptscotch=/opt/petsc/scotch-v7.0.3.tar.gz --download-bison=/opt/petsc/bison-3.8.2.tar.gz --download-superlu_dist=/opt/petsc/v8.2.1.tar.gz --download-scalapack=/opt/petsc/scalapack --download-mumps=/opt/petsc//MUMPS_5.6.2.tar.gz --download-hypre=/opt/petsc/ee74c20e7a84e4e48eec142c6bb6ff2a75db72f1.tar.gz --download-slepc=/opt/petsc/slepc-v3.21.0.tar.gz
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #1 VecGetArrayForHYPRE() at /opt/petsc/src/vec/vec/impls/hypre/vhyp.c:94
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #2 VecHYPRE_IJVectorPushVecRead() at /opt/petsc/src/vec/vec/impls/hypre/vhyp.c:137
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #3 PCApply_HYPRE() at /opt/petsc/src/ksp/pc/impls/hypre/hypre.c:438
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #4 PCApply() at /opt/petsc/src/ksp/pc/interface/precon.c:497
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #5 PCApplyBAorAB() at /opt/petsc/src/ksp/pc/interface/precon.c:767
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #6 KSP_PCApplyBAorAB() at /opt/petsc/include/petsc/private/kspimpl.h:469
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #7 KSPGMRESCycle() at /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c:146
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #8 KSPSolve_GMRES() at /opt/petsc/src/ksp/ksp/impls/gmres/gmres.c:227
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #9 KSPSolve_Private() at /opt/petsc/src/ksp/ksp/interface/itfunc.c:905
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #10 KSPSolve() at /opt/petsc/src/ksp/ksp/interface/itfunc.c:1078
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #11 SNESSolve_NEWTONLS() at /opt/petsc/src/snes/impls/ls/ls.c:220
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: [0]PETSC ERROR: #12 SNESSolve() at /opt/petsc/src/snes/interface/snes.c:4738
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: libMesh terminating:
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: HYPRE_MEMORY_DEVICE expects a device vector. You need to enable PETSc device support, for example, in some cases, -vec_type cuda
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: --------------------------------------------------------------------------
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: with errorcode 1.
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: You may or may not see output from other processes, depending on
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: exactly when Open MPI kills them.
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: --------------------------------------------------------------------------
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: ################################################################################
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: Tester failed, reason: ERRMSG
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test: 
transfers/multiapp_variable_value_sample_transfer.to_multiapp_test ........................... FAILED (ERRMSG)

[lindsayad]

Because with the way PETSc's wrapper of hypre work, hypre expects your vector to be on the device. So any inputs that use hypre must specify the vec type as the error message indicates

[ljeft]

This is the result of me running the MOOSE case. Is this error due to the hip I used?

root@e9a45d4c9858:/opt/projects/moose/examples/ex01_inputfile# mpiexec -n 6 --allow-run-as-root ./ex01-opt -i ex01.i -vec_type hip -mat_type aijcusparse 
[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1



*** Info ***
'execute_on' parameter specified in [Outputs] block is ignored for object 'checkpoint'.
Define this object in its own sub-block of [Outputs] to modify its execution schedule.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------


*** ERROR ***
Unknown Mat type given: aijcusparse



*** ERROR ***
Unknown Mat type given: aijcusparse



*** ERROR ***
Unknown Mat type given: aijcusparse



*** ERROR ***
Unknown Mat type given: aijcusparse



*** ERROR ***
Unknown Mat type given: aijcusparse

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: https://petsc.org/release/install/install/#external-packages
[0]PETSC ERROR: Unknown Mat type given: aijcusparse
[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
[0]PETSC ERROR:   Option left: name:-i value: ex01.i source: command line
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.21.0, unknown 
[0]PETSC ERROR: ./ex01-opt on a arch-linux-c-opt named e9a45d4c9858 by root Tue Jul 30 09:38:21 2024
[0]PETSC ERROR: Configure options --with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --with-hip=1 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-superlu_dist=1 --with-hipc=/opt/dtk-23.10/bin/hipcc --with-hip-arch=gfx906 --download-fblaslapack=/opt/petsc/v3.4.2-p3.tar.gz --download-metis=/opt/petsc/v5.1.0-p11.tar.gz --download-parmetis=/opt/petsc/v4.0.3-p9.tar.gz --download-ptscotch=/opt/petsc/scotch-v7.0.3.tar.gz --download-bison=/opt/petsc/bison-3.8.2.tar.gz --download-superlu_dist=/opt/petsc/v8.2.1.tar.gz --download-scalapack=/opt/petsc/scalapack --download-mumps=/opt/petsc//MUMPS_5.6.2.tar.gz --download-hypre=/opt/petsc/ee74c20e7a84e4e48eec142c6bb6ff2a75db72f1.tar.gz --download-slepc=/opt/petsc/slepc-v3.21.0.tar.gz
[0]PETSC ERROR: #1 MatSetType() at /opt/petsc/src/mat/interface/matreg.c:144
[0]PETSC ERROR: #2 MatSetFromOptions() at /opt/petsc/src/mat/utils/gcreate.c:300


*** ERROR ***
Unknown Mat type given: aijcusparse

                                                                               [  0.16 s] [  221 MB]
[e9a45d4c9858:23417] 5 more processes have sent help message help-mpi-api.txt / mpi-abort
[e9a45d4c9858:23417] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
root@e9a45d4c9858:/opt/projects/moose/examples/ex01_inputfile# mpiexec -n 6 --allow-run-as-root ./ex01-opt -i ex01.i -vec_type hip -mat_type aijcusparse 

[GiudGiud]

@lindsayad shouldnt we see -lcuda or something in the linked libraries?

[lindsayad]

I don't think so for an AMD GPU with HIP. Can you run with -snes_view? Also you need to specify Executioner/solve_type = NEWTON. I don't recall why, but these GPU implementations do not work with matrix-free Jacobians. As a demo on my machine with cuda for the simple_diffusion.i input:

lindad@pop-os:~/projects/moose/test/tests/kernels/simple_diffusion(kokkos)$ ../../../moose_test-devel -i simple_diffusion.i -vec_type cuda -mat_type aijcusparse

 0 Nonlinear |R| = 3.082207e+00
      0 Linear |R| = 3.082207e+00
      1 Linear |R| = 3.082207e+00
  Linear solve did not converge due to DIVERGED_BREAKDOWN iterations 1
Nonlinear solve did not converge due to DIVERGED_LOCAL_MIN iterations 0
 Solve Did NOT Converge!

lindad@pop-os:~/projects/moose/test/tests/kernels/simple_diffusion(kokkos)$ ../../../moose_test-devel -i simple_diffusion.i -vec_type cuda -mat_type aijcusparse Executioner/solve_type=NEWTON

 0 Nonlinear |R| = 3.082207e+00
      0 Linear |R| = 3.082207e+00
      1 Linear |R| = 4.617657e-01
      2 Linear |R| = 2.103940e-01
      3 Linear |R| = 8.673169e-02
      4 Linear |R| = 1.563373e-02
      5 Linear |R| = 2.597863e-03
      6 Linear |R| = 8.230456e-04
      7 Linear |R| = 1.890142e-04
      8 Linear |R| = 2.560477e-05
 1 Nonlinear |R| = 2.560477e-05
      0 Linear |R| = 2.560477e-05
      1 Linear |R| = 5.688495e-06
      2 Linear |R| = 1.086579e-06
      3 Linear |R| = 3.885360e-07
      4 Linear |R| = 1.164039e-07
      5 Linear |R| = 4.452560e-08
      6 Linear |R| = 2.011179e-08
      7 Linear |R| = 2.891492e-09
      8 Linear |R| = 2.379732e-10
 2 Nonlinear |R| = 2.379733e-10  

[ljeft]

This is the result of running with -snes_view,and I modify Executioner/solve_type = NEWTON in ex01.i

root@e9a45d4c9858:/opt/projects/moose/examples/ex01_inputfile# make -j 8
root@e9a45d4c9858:/opt/projects/moose/examples/ex01_inputfile# mpiexec -n 6 --allow-run-as-root ./ex01-opt -i ex01.i -vec_type hip -snes_view
[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1



*** Info ***
'execute_on' parameter specified in [Outputs] block is ignored for object 'checkpoint'.
Define this object in its own sub-block of [Outputs] to modify its execution schedule.
                                                                               [  0.15 s] [  228 MB]
Framework Information:
MOOSE Version:           unknown
LibMesh Version:         
PETSc Version:           3.21.0
SLEPc Version:           3.21.0
Current Time:            Thu Aug  1 20:24:23 2024
Executable Timestamp:    Mon Jul 29 16:33:43 2024

Parallelism:
  Num Processors:          6
  Num Threads:             1

Mesh: 
  Parallel Type:           replicated
  Mesh Dimension:          3
  Spatial Dimension:       3
  Nodes:                   
    Total:                 3774
    Local:                 711
    Min/Max/Avg:           543/711/629
  Elems:                   
    Total:                 2476
    Local:                 412
    Min/Max/Avg:           412/414/412
  Num Subdomains:          1
  Num Partitions:          6
  Partitioner:             metis

Nonlinear System:
  Num DOFs:                3774
  Num Local DOFs:          711
  Variables:               "diffused" 
  Finite Element Types:    "LAGRANGE" 
  Approximation Orders:    "FIRST" 

Execution Information:
  Executioner:             Steady
  Solver Mode:             NEWTON
  MOOSE Preconditioner:    SMP (auto)

LEGACY MODES ENABLED:
 This application uses the legacy material output option: material properties are output only on TIMESTEP_END, not INITIAL. To remove this message, set 'use_legacy_material_output' to false in this application. If there are gold output files that contain material property output for which output occurs on INITIAL, then these will generate diffs due to zero values being stored, and these tests should be re-golded.

LEGACY MODES ENABLED:
 This application uses the legacy initial residual evaluation behavior. The legacy behavior performs an often times redundant residual evaluation before the solution modifying objects are executed prior to the initial (0th nonlinear iteration) residual evaluation. The new behavior skips that redundant residual evaluation unless the parameter Executioner/use_pre_smo_residual is set to true. To remove this message and enable the new behavior, set the parameter 'use_legacy_initial_residual_evaluation_behavior' to false in *App.C. Some tests that rely on the side effects of the legacy behavior may fail/diff and should be re-golded.

Currently Executing
    Computing Pre-SMO Residual
      Computing Residual..                                                               [ 18.42 s] [ 3817 MB]
Pre-SMO residual: 22.3159
    Finished Computing Pre-SMO Residual                                                  [ 18.42 s] [ 3817 MB]
 0 Nonlinear |R| = 6.105359e+00
      0 Linear |R| = 6.105359e+00
      1 Linear |R| = 9.974021e-01
      2 Linear |R| = 4.553251e-01
      3 Linear |R| = 2.267631e-01
      4 Linear |R| = 1.491978e-01
      5 Linear |R| = 1.100377e-01
      6 Linear |R| = 7.298555e-02
      7 Linear |R| = 5.521537e-02
      8 Linear |R| = 4.391126e-02
      9 Linear |R| = 3.575172e-02
     10 Linear |R| = 2.973446e-02
     11 Linear |R| = 2.281436e-02
     12 Linear |R| = 1.551603e-02
     13 Linear |R| = 7.965008e-03
     14 Linear |R| = 3.390029e-03
     15 Linear |R| = 2.212221e-03
     16 Linear |R| = 1.609138e-03
     17 Linear |R| = 1.157976e-03
     18 Linear |R| = 9.072908e-04
     19 Linear |R| = 6.776024e-04
     20 Linear |R| = 4.972971e-04
     21 Linear |R| = 3.304996e-04
     22 Linear |R| = 2.118354e-04
     23 Linear |R| = 1.469469e-04
     24 Linear |R| = 9.480257e-05
     25 Linear |R| = 6.049671e-05
 1 Nonlinear |R| = 6.049671e-05
      0 Linear |R| = 6.049671e-05
      1 Linear |R| = 4.491059e-05
      2 Linear |R| = 2.915180e-05
      3 Linear |R| = 1.878568e-05
      4 Linear |R| = 1.180170e-05
      5 Linear |R| = 7.971916e-06
      6 Linear |R| = 5.603588e-06
      7 Linear |R| = 4.117062e-06
      8 Linear |R| = 3.160187e-06
      9 Linear |R| = 2.320609e-06
     10 Linear |R| = 1.682351e-06
     11 Linear |R| = 1.170422e-06
     12 Linear |R| = 7.183455e-07
     13 Linear |R| = 4.165589e-07
     14 Linear |R| = 2.283220e-07
     15 Linear |R| = 1.551421e-07
     16 Linear |R| = 1.224430e-07
     17 Linear |R| = 9.953858e-08
     18 Linear |R| = 8.027675e-08
     19 Linear |R| = 6.429647e-08
     20 Linear |R| = 5.203135e-08
     21 Linear |R| = 4.170252e-08
     22 Linear |R| = 3.032067e-08
     23 Linear |R| = 1.939533e-08
     24 Linear |R| = 9.604536e-09
     25 Linear |R| = 5.720933e-09
     26 Linear |R| = 3.587562e-09
     27 Linear |R| = 2.277113e-09
     28 Linear |R| = 1.581914e-09
     29 Linear |R| = 1.044613e-09
     30 Linear |R| = 7.488340e-10
     31 Linear |R| = 6.160169e-10
     32 Linear |R| = 4.874183e-10
 2 Nonlinear |R| = 4.874183e-10
SNES Object: 6 MPI processes
  type: newtonls
  maximum iterations=50, maximum function evaluations=10000
  tolerances: relative=1e-08, absolute=1e-50, solution=0.
  total number of linear solver iterations=57
  total number of function evaluations=3
  norm schedule ALWAYS
  SNESLineSearch Object: 6 MPI processes
    type: bt
      interpolation: cubic
      alpha=1.000000e-04
    maxstep=1.000000e+08, minlambda=1.000000e-12
    tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
    maximum iterations=40
  KSP Object: 6 MPI processes
    type: gmres
      restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
      happy breakdown tolerance 1e-30
    maximum iterations=10000, initial guess is zero
    tolerances: relative=1e-05, absolute=1e-50, divergence=1e+100
    right preconditioning
    using UNPRECONDITIONED norm type for convergence test
  PC Object: 6 MPI processes
    type: bjacobi
      number of blocks = 6
      Local solver information for first block is in the following KSP and PC objects on rank 0:
      Use -ksp_view ::ascii_info_detail to display information for all blocks
      KSP Object: (sub_) 1 MPI process
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
        left preconditioning
        using NONE norm type for convergence test
      PC Object: (sub_) 1 MPI process
        type: ilu
          out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          matrix ordering: natural
          factor fill ratio given 1., needed 1.
            Factored matrix follows:
              Mat Object: (sub_) 1 MPI process
                type: seqaij
                rows=711, cols=711
                package used to perform factorization: petsc
                total: nonzeros=13657, allocated nonzeros=13657
                  not using I-node routines
        linear system matrix = precond matrix:
        Mat Object: (sub_) 1 MPI process
          type: seqaij
          rows=711, cols=711
          total: nonzeros=13657, allocated nonzeros=14630
          total number of mallocs used during MatSetValues calls=0
            not using I-node routines
    linear system matrix = precond matrix:
    Mat Object: () 6 MPI processes
      type: mpiaij
      rows=3774, cols=3774
      total: nonzeros=78400, allocated nonzeros=81774
      total number of mallocs used during MatSetValues calls=0
        not using I-node (on process 0) routines
 Solve Converged!
  Finished Solving                                                                       [ 19.54 s] [ 4019 MB]
[e9a45d4c9858:01081] Read -1, expected 5328, errno = 14
[e9a45d4c9858:01081] Read -1, expected 4728, errno = 14
[e9a45d4c9858:01081] Read -1, expected 4824, errno = 14
[e9a45d4c9858:01081] Read -1, expected 4344, errno = 14
[e9a45d4c9858:01081] Read -1, expected 5280, errno = 14
Finished Executing                                                                       [ 19.60 s] [ 4019 MB]

[lindsayad]

I don't know what's happening with the errno but it looks like most things are working. You should still try passing -mat_type aijhipsparse because right now the matrix type showing up in -snes_view is standard aij

[ljeft]

It looks like it worked

root@e9a45d4c9858:/opt/projects/moose/examples/ex01_inputfile# mpiexec -n 6 --allow-run-as-root ./ex01-opt -i ex01.i -vec_type hip -snes_view -mat_type aijhipsparse
[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1

[LOG_ERROR] ROCR-Runtime_5.2_docker/src/core/runtime/interrupt_signal.cpp:135: Create events pool failed. return value: 0x1



*** Info ***
'execute_on' parameter specified in [Outputs] block is ignored for object 'checkpoint'.
Define this object in its own sub-block of [Outputs] to modify its execution schedule.
                                                                               [  0.15 s] [  224 MB]
Framework Information:
MOOSE Version:           unknown
LibMesh Version:         
PETSc Version:           3.21.0
SLEPc Version:           3.21.0
Current Time:            Thu Aug  1 23:57:45 2024
Executable Timestamp:    Mon Jul 29 16:33:43 2024

Parallelism:
  Num Processors:          6
  Num Threads:             1

Mesh: 
  Parallel Type:           replicated
  Mesh Dimension:          3
  Spatial Dimension:       3
  Nodes:                   
    Total:                 3774
    Local:                 711
    Min/Max/Avg:           543/711/629
  Elems:                   
    Total:                 2476
    Local:                 412
    Min/Max/Avg:           412/414/412
  Num Subdomains:          1
  Num Partitions:          6
  Partitioner:             metis

Nonlinear System:
  Num DOFs:                3774
  Num Local DOFs:          711
  Variables:               "diffused" 
  Finite Element Types:    "LAGRANGE" 
  Approximation Orders:    "FIRST" 

Execution Information:
  Executioner:             Steady
  Solver Mode:             NEWTON
  MOOSE Preconditioner:    SMP (auto)

LEGACY MODES ENABLED:
 This application uses the legacy material output option: material properties are output only on TIMESTEP_END, not INITIAL. To remove this message, set 'use_legacy_material_output' to false in this application. If there are gold output files that contain material property output for which output occurs on INITIAL, then these will generate diffs due to zero values being stored, and these tests should be re-golded.

LEGACY MODES ENABLED:
 This application uses the legacy initial residual evaluation behavior. The legacy behavior performs an often times redundant residual evaluation before the solution modifying objects are executed prior to the initial (0th nonlinear iteration) residual evaluation. The new behavior skips that redundant residual evaluation unless the parameter Executioner/use_pre_smo_residual is set to true. To remove this message and enable the new behavior, set the parameter 'use_legacy_initial_residual_evaluation_behavior' to false in *App.C. Some tests that rely on the side effects of the legacy behavior may fail/diff and should be re-golded.

Currently Executing
    Computing Pre-SMO Residual
      Computing Residual..                                                               [ 18.54 s] [ 3813 MB]
Pre-SMO residual: 22.3159
    Finished Computing Pre-SMO Residual                                                  [ 18.54 s] [ 3813 MB]
 0 Nonlinear |R| = 6.105359e+00
      0 Linear |R| = 6.105359e+00
      1 Linear |R| = 9.974021e-01
      2 Linear |R| = 4.553251e-01
      3 Linear |R| = 2.267631e-01
      4 Linear |R| = 1.491978e-01
      5 Linear |R| = 1.100377e-01
      6 Linear |R| = 7.298555e-02
      7 Linear |R| = 5.521537e-02
      8 Linear |R| = 4.391126e-02
      9 Linear |R| = 3.575172e-02
     10 Linear |R| = 2.973446e-02
     11 Linear |R| = 2.281436e-02
     12 Linear |R| = 1.551603e-02
     13 Linear |R| = 7.965008e-03
     14 Linear |R| = 3.390029e-03
     15 Linear |R| = 2.212221e-03
     16 Linear |R| = 1.609138e-03
     17 Linear |R| = 1.157976e-03
     18 Linear |R| = 9.072908e-04
     19 Linear |R| = 6.776024e-04
     20 Linear |R| = 4.972971e-04
     21 Linear |R| = 3.304996e-04
     22 Linear |R| = 2.118354e-04
     23 Linear |R| = 1.469469e-04
     24 Linear |R| = 9.480257e-05
     25 Linear |R| = 6.049671e-05
 1 Nonlinear |R| = 6.049671e-05
      0 Linear |R| = 6.049671e-05
      1 Linear |R| = 4.491059e-05
      2 Linear |R| = 2.915180e-05
      3 Linear |R| = 1.878568e-05
      4 Linear |R| = 1.180170e-05
      5 Linear |R| = 7.971916e-06
      6 Linear |R| = 5.603588e-06
      7 Linear |R| = 4.117062e-06
      8 Linear |R| = 3.160187e-06
      9 Linear |R| = 2.320609e-06
     10 Linear |R| = 1.682351e-06
     11 Linear |R| = 1.170422e-06
     12 Linear |R| = 7.183455e-07
     13 Linear |R| = 4.165589e-07
     14 Linear |R| = 2.283220e-07
     15 Linear |R| = 1.551421e-07
     16 Linear |R| = 1.224430e-07
     17 Linear |R| = 9.953858e-08
     18 Linear |R| = 8.027675e-08
     19 Linear |R| = 6.429647e-08
     20 Linear |R| = 5.203135e-08
     21 Linear |R| = 4.170252e-08
     22 Linear |R| = 3.032067e-08
     23 Linear |R| = 1.939533e-08
     24 Linear |R| = 9.604536e-09
     25 Linear |R| = 5.720933e-09
     26 Linear |R| = 3.587562e-09
     27 Linear |R| = 2.277113e-09
     28 Linear |R| = 1.581914e-09
     29 Linear |R| = 1.044613e-09
     30 Linear |R| = 7.488340e-10
     31 Linear |R| = 6.160169e-10
     32 Linear |R| = 4.874183e-10
 2 Nonlinear |R| = 4.874183e-10
SNES Object: 6 MPI processes
  type: newtonls
  maximum iterations=50, maximum function evaluations=10000
  tolerances: relative=1e-08, absolute=1e-50, solution=0.
  total number of linear solver iterations=57
  total number of function evaluations=3
  norm schedule ALWAYS
  SNESLineSearch Object: 6 MPI processes
    type: bt
      interpolation: cubic
      alpha=1.000000e-04
    maxstep=1.000000e+08, minlambda=1.000000e-12
    tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
    maximum iterations=40
  KSP Object: 6 MPI processes
    type: gmres
      restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
      happy breakdown tolerance 1e-30
    maximum iterations=10000, initial guess is zero
    tolerances: relative=1e-05, absolute=1e-50, divergence=1e+100
    right preconditioning
    using UNPRECONDITIONED norm type for convergence test
  PC Object: 6 MPI processes
    type: bjacobi
      number of blocks = 6
      Local solver information for first block is in the following KSP and PC objects on rank 0:
      Use -ksp_view ::ascii_info_detail to display information for all blocks
      KSP Object: (sub_) 1 MPI process
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
        left preconditioning
        using NONE norm type for convergence test
      PC Object: (sub_) 1 MPI process
        type: ilu
          out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          matrix ordering: natural
          factor fill ratio given 1., needed 1.
            Factored matrix follows:
              Mat Object: (sub_) 1 MPI process
                type: seqaijhipsparse
                rows=711, cols=711
                package used to perform factorization: hipsparse
                total: nonzeros=13657, allocated nonzeros=13657
                  not using I-node routines
        linear system matrix = precond matrix:
        Mat Object: (sub_) 1 MPI process
          type: seqaijhipsparse
          rows=711, cols=711
          total: nonzeros=13657, allocated nonzeros=14630
          total number of mallocs used during MatSetValues calls=0
            not using I-node routines
    linear system matrix = precond matrix:
    Mat Object: () 6 MPI processes
      type: mpiaijhipsparse
      rows=3774, cols=3774
      total: nonzeros=78400, allocated nonzeros=81774
      total number of mallocs used during MatSetValues calls=0
        not using I-node (on process 0) routines
  Finished Solving                                                                       [ 19.67 s] [ 4036 MB]
 Solve Converged!
[e9a45d4c9858:04454] Read -1, expected 5328, errno = 14
[e9a45d4c9858:04454] Read -1, expected 4728, errno = 14
[e9a45d4c9858:04454] Read -1, expected 4824, errno = 14
[e9a45d4c9858:04454] Read -1, expected 4344, errno = 14
[e9a45d4c9858:04454] Read -1, expected 5280, errno = 14
Finished Executing                                                                       [ 19.73 s] [ 4036 MB]

[lindsayad]

excellent!

[ljeft]

Note that you cannot use these GPU mat types for the preconditioning matrix and use matrix-free finite differencing for the Jacobian, so you cannot use PJFNK, you must use NEWTON.

This is the reply you gave me in #21152 before. I want to know what are the limitations of MOOSE with HIP compared with MOOSE.

[lindsayad]

that is the biggest one I know of, but that doesn't mean there aren't others. We have not had a lot of people doing GPU solves from MOOSE. @NamjaeChoi is currently working on doing GPU assembly with MOOSE and I believe he is starting to interface that assembly work with the solve