Unified memory for HIP

[jeremylt]

I still need to test, but I think I got the important changes for the MI300A in here (note, made this branch on top of #1782)

This PR superceeds #1734

[jeremylt]

I'm working through the implications, but I think we still need to let someone set a host pointer as a host pointer, so that's the only important change in this branch

[jeremylt]

With my changes

$ make prove -j BACKENDS="/gpu/hip/ref /gpu/hip/shared /gpu/hip/gen"
...
t006-ceed ............... ok                                            
t007-ceed ............... ok                                            
t102-vector ............. ok                                            
t110-vector ............. ok                                            
t112-vector ............. ok                                            
t113-vector ............. ok                                            
t114-vector ............. ok                                            
t115-vector ............. ok                                            
t111-vector ............. ok                                            
t116-vector ............. ok                                            
t117-vector ............. ok                                            
t118-vector ............. ok                                            
t301-basis .............. ok                                            
t303-basis .............. ok                                            
t304-basis .............. ok                                            
t305-basis .............. ok                                            
t306-basis .............. ok                                            
t508-operator ........... ok                                            
t004-ceed ............... ok                                            
t211-elemrestriction .... ok                                            
t212-elemrestriction .... ok                                            
t001-ceed ............... ok                                            
t208-elemrestriction .... ok                                            
t008-ceed ............... ok                                            
t005-ceed ............... ok                                            
t407-qfunction .......... ok                                            
t413-qfunction .......... ok                                            
t213-elemrestriction .... ok                                            
t402-qfunction .......... ok                                            
t211-elemrestriction-f .. ok                                            
t002-ceed ............... ok                                            
t009-ceed ............... ok                                            
t202-elemrestriction .... ok                                            
t408-qfunction .......... ok                                            
t219-elemrestriction .... ok                                            
t202-elemrestriction-f .. ok                                            
t208-elemrestriction-f .. ok                                            
t000-ceed-f ............. ok                                            
t525-operator ........... ok                                            
t404-qfunction .......... ok                                            
t403-qfunction .......... ok                                            
t120-vector ............. ok                                            
t124-vector ............. ok                                            
t003-ceed ............... ok                                            
t212-elemrestriction-f .. ok                                            
t000-ceed ............... ok                                            
t203-elemrestriction .... ok                                            
t124-vector-f ........... ok                                            
t003-ceed-f ............. ok                                            
t004-ceed-f ............. ok                                            
t130-vector ............. ok                                            
t001-ceed-f ............. ok                                            
t215-elemrestriction .... ok                                            
t103-vector ............. ok                                            
t413-qfunction-f ........ ok                                            
t402-qfunction-f ........ ok                                            
t200-elemrestriction .... ok                                            
t233-elemrestriction .... ok                                            
t355-basis .............. ok                                            
t107-vector ............. ok                                            
t412-qfunction .......... ok                                            
t231-elemrestriction .... ok                                            
t362-basis .............. ok                                            
t411-qfunction-f ........ ok                                            
t310-basis .............. ok                                            
t324-basis .............. ok                                            
t210-elemrestriction .... ok                                            
t220-elemrestriction .... ok                                            
t109-vector ............. ok                                            
t320-basis .............. ok                                            
t342-basis .............. ok                                            
t410-qfunction .......... ok                                            
t104-vector ............. ok                                            
t216-elemrestriction .... ok                                            
t410-qfunction-f ........ ok                                            
t412-qfunction-f ........ ok                                            
t322-basis-f ............ ok                                            
t504-operator-f ......... ok                                            
t201-elemrestriction .... ok                                            
t320-basis-f ............ ok                                            
t306-basis-f ............ ok                                            
t107-vector-f ........... ok                                            
t101-vector-f ........... ok                                            
t100-vector-f ........... ok                                            
t323-basis-f ............ ok                                            
t108-vector ............. ok                                            
t510-operator-f ......... ok                                            
t502-operator-f ......... ok                                            
t503-operator-f ......... ok                                            
t109-vector-f ........... ok                                            
t401-qfunction-f ........ ok                                            
t106-vector ............. ok                                            
t123-vector ............. ok                                            
t409-qfunction .......... ok                                            
t102-vector-f ........... ok                                            
t204-elemrestriction .... ok                                            
t363-basis .............. ok                                            
t122-vector ............. ok                                            
t360-basis .............. ok                                            
t101-vector ............. ok                                            
t400-qfunction-f ........ ok                                            
t207-elemrestriction .... ok                                            
t523-operator-f ......... ok                                            
t128-vector ............. ok                                            
t406-qfunction .......... ok                                            
t332-basis .............. ok                                            
t125-vector ............. ok                                            
t100-vector ............. ok                                            
t300-basis .............. ok                                            
t350-basis .............. ok                                            
t411-qfunction .......... ok                                            
t209-elemrestriction-f .. ok                                            
t341-basis .............. ok                                            
t322-basis .............. ok                                            
t505-operator-f ......... ok                                            
t400-qfunction .......... ok                                            
t312-basis .............. ok                                            
t331-basis .............. ok                                            
t340-basis .............. ok                                            
t522-operator-f ......... ok                                            
t504-operator ........... ok                                            
t317-basis .............. ok                                            
t500-operator-f ......... ok                                            
t321-basis .............. ok                                            
t105-vector ............. ok                                            
t511-operator-f ......... ok                                            
t104-vector-f ........... ok                                            
t530-operator-f ......... ok                                            
t323-basis .............. ok                                            
t311-basis .............. ok                                            
t121-vector ............. ok                                            
t405-qfunction .......... ok                                            
t126-vector ............. ok                                            
t127-vector ............. ok                                            
t330-basis .............. ok                                            
t401-qfunction .......... ok                                            
t230-elemrestriction .... ok                                            
t232-elemrestriction .... ok                                            
t214-elemrestriction .... ok                                            
t325-basis .............. ok                                            
t209-elemrestriction .... ok                                            
t205-elemrestriction .... ok                                            
t217-elemrestriction .... ok                                            
t307-basis .............. ok                                            
t218-elemrestriction .... ok                                            
t521-operator-f ......... ok                                            
t234-elemrestriction .... ok                                            
t302-basis .............. ok                                            
t103-vector-f ........... ok                                            
t533-operator-f ......... ok                                            
t534-operator-f ......... ok                                            
t108-vector-f ........... ok                                            
t119-vector-f ........... ok                                            
t535-operator-f ......... ok                                            
t552-operator-f ......... ok                                            
t210-elemrestriction-f .. ok                                            
t106-vector-f ........... ok                                            
t314-basis-f ............ ok                                            
t201-elemrestriction-f .. ok                                            
t119-vector ............. ok                                            
t501-operator-f ......... ok                                            
t300-basis-f ............ ok                                            
t206-elemrestriction .... ok                                            
t536-operator-f ......... ok                                            
t540-operator-f ......... ok                                            
t553-operator-f ......... ok                                            
t105-vector-f ........... ok                                            
t314-basis .............. ok                                            
t550-operator-f ......... ok                                            
t302-basis-f ............ ok                                            
t200-elemrestriction-f .. ok                                            
t532-operator-f ......... ok                                            
t316-basis .............. ok                                            
t509-operator ........... ok                                            
t581-operator ........... ok                                            
t315-basis .............. ok                                            
t595-operator ........... ok                                            
t414-qfunction .......... ok                                            
t524-operator-f ......... ok                                            
t520-operator-f ......... ok                                            
t582-operator ........... ok                                            
t590-operator ........... ok                                            
t526-operator ........... ok                                            
t353-basis .............. ok                                            
t364-basis .............. ok                                            
t523-operator ........... ok                                            
t592-operator ........... ok                                            
t570-operator ........... ok                                            
t561-operator ........... ok                                            
t580-operator ........... ok                                            
t591-operator ........... ok                                            
t560-operator ........... ok                                            
t593-operator ........... ok                                            
t594-operator ........... ok                                            
t531-operator-f ......... ok                                            
t568-operator ........... ok                                            
t506-operator-f ......... ok                                            
t567-operator ........... ok                                            
t566-operator ........... ok                                            
t562-operator ........... ok                                            
t569-operator ........... ok                                            
t511-operator ........... ok                                            
t583-operator ........... ok                                            
t530-operator ........... ok                                            
t361-basis .............. ok                                            
t534-operator ........... ok                                            
t564-operator ........... ok                                            
t510-operator ........... ok                                            
t507-operator ........... ok                                            
t550-operator ........... ok                                            
t539-operator ........... ok                                            
t563-operator ........... ok                                            
t541-operator ........... ok                                            
t537-operator ........... ok                                            
t532-operator ........... ok                                            
t500-operator ........... ok                                            
t415-qfunction .......... ok                                            
t505-operator ........... ok                                            
t540-operator ........... ok                                            
t565-operator ........... ok                                            
t553-operator ........... ok                                            
t535-operator ........... ok                                            
t533-operator ........... ok                                            
t501-operator ........... ok                                            
t502-operator ........... ok                                            
t531-operator ........... ok                                            
t313-basis-f ............ ok                                            
t503-operator ........... ok                                            
t551-operator ........... ok                                            
t552-operator ........... ok                                            
t536-operator ........... ok                                            
t506-operator ........... ok                                            
ex3-volume .............. ok                                            
t521-operator ........... ok                                            
t538-operator ........... ok                                            
t524-operator ........... ok                                            
t520-operator ........... ok                                            
t522-operator ........... ok                                            
t356-basis .............. ok                                            
t365-basis .............. ok                                            
t357-basis .............. ok                                            
t351-basis .............. ok                                            
t313-basis .............. ok                                            
t318-basis .............. ok                                            
t352-basis .............. ok                                            
t354-basis .............. ok                                            
t554-operator ........... ok                                            
t319-basis .............. ok                                            
ex1-volume .............. ok                                            
ex2-surface ............. ok   
All tests successful.
Files=250, Tests=272, 127 wallclock secs ( 1.84 usr  0.63 sys + 9733.24 cusr 2277.66 csys = 12013.37 CPU)
Result: PASS

[zatkins-dev]

All tests look good for me on Tuolumne.

libCEED tests

Command:

make -j prove PROVE_OPTS="-j4" BACKENDS="/gpu/hip/ref /gpu/hip/shared /gpu/hip/gen" NPROC_POOL=4

Results:

Testing backends: /gpu/hip/ref /gpu/hip/shared /gpu/hip/gen
prove -j4 --exec '/opt/cray/pe/python/3.11.7/bin/python3 tests/junit.py --mode tap --ceed-backends /gpu/hip/ref /gpu/hip/shared /gpu/hip/gen --nproc 1 --pool-size 4' t000-ceed t001-ceed t002-ceed t003-ceed t004-ceed t005-ceed t006-ceed t007-ceed t008-ceed t009-ceed t010-config t100-vector t101-vector t102-vector t103-vector t104-vector t105-vector t106-vector t107-vector t108-vector t109-vector t110-vector t111-vector t112-vector t113-vector t114-vector t115-vector t116-vector t117-vector t118-vector t119-vector t120-vector t121-vector t122-vector t123-vector t124-vector t125-vector t126-vector t127-vector t128-vector t130-vector t200-elemrestriction t201-elemrestriction t202-elemrestriction t203-elemrestriction t204-elemrestriction t205-elemrestriction t206-elemrestriction t207-elemrestriction t208-elemrestriction t209-elemrestriction t210-elemrestriction t211-elemrestriction t212-elemrestriction t213-elemrestriction t214-elemrestriction t215-elemrestriction t216-elemrestriction t217-elemrestriction t218-elemrestriction t219-elemrestriction t220-elemrestriction t230-elemrestriction t231-elemrestriction t232-elemrestriction t233-elemrestriction t234-elemrestriction t300-basis t301-basis t302-basis t303-basis t304-basis t305-basis t306-basis t307-basis t310-basis t311-basis t312-basis t313-basis t314-basis t315-basis t316-basis t317-basis t318-basis t319-basis t320-basis t321-basis t322-basis t323-basis t324-basis t325-basis t330-basis t331-basis t332-basis t340-basis t341-basis t342-basis t350-basis t351-basis t352-basis t353-basis t354-basis t355-basis t356-basis t357-basis t360-basis t361-basis t362-basis t363-basis t364-basis t365-basis t400-qfunction t401-qfunction t402-qfunction t403-qfunction t404-qfunction t405-qfunction t406-qfunction t407-qfunction t408-qfunction t409-qfunction t410-qfunction t411-qfunction t412-qfunction t413-qfunction t414-qfunction t415-qfunction t500-operator t501-operator t502-operator t503-operator t504-operator t505-operator t506-operator t507-operator t508-operator t509-operator t510-operator t511-operator t520-operator t521-operator t522-operator t523-operator t524-operator t525-operator t526-operator t530-operator t531-operator t532-operator t533-operator t534-operator t535-operator t536-operator t537-operator t538-operator t539-operator t540-operator t541-operator t550-operator t551-operator t552-operator t553-operator t554-operator t560-operator t561-operator t562-operator t563-operator t564-operator t565-operator t566-operator t567-operator t568-operator t569-operator t570-operator t580-operator t581-operator t582-operator t583-operator t590-operator t591-operator t592-operator t593-operator t594-operator t595-operator ex1-volume ex2-surface ex3-volume
t001-ceed ............. ok                                              
t000-ceed ............. ok                                              
t002-ceed ............. ok                                              
t003-ceed ............. ok                                              
t007-ceed ............. ok                                              
t006-ceed ............. ok                                              
t004-ceed ............. ok                                              
t005-ceed ............. ok                                              
t008-ceed ............. ok                                              
t009-ceed ............. ok                                              
t102-vector ........... ok                                              
t010-config ........... ok                                              
t103-vector ........... ok                                              
t104-vector ........... ok                                              
t101-vector ........... ok                                              
t100-vector ........... ok                                              
t105-vector ........... ok                                              
t106-vector ........... ok                                              
t110-vector ........... ok                                              
t111-vector ........... ok                                              
t112-vector ........... ok                                              
t107-vector ........... ok                                              
t113-vector ........... ok                                              
t114-vector ........... ok                                              
t115-vector ........... ok                                              
t116-vector ........... ok                                              
t117-vector ........... ok                                              
t118-vector ........... ok                                              
t109-vector ........... ok                                              
t108-vector ........... ok                                              
t119-vector ........... ok                                              
t120-vector ........... ok                                              
t121-vector ........... ok                                              
t122-vector ........... ok                                              
t125-vector ........... ok                                              
t126-vector ........... ok                                              
t124-vector ........... ok                                              
t123-vector ........... ok                                              
t127-vector ........... ok                                              
t130-vector ........... ok                                              
t128-vector ........... ok                                              
t200-elemrestriction .. ok                                              
t202-elemrestriction .. ok                                              
t203-elemrestriction .. ok                                              
t201-elemrestriction .. ok                                              
t204-elemrestriction .. ok                                              
t205-elemrestriction .. ok                                              
t207-elemrestriction .. ok                                              
t206-elemrestriction .. ok                                              
t208-elemrestriction .. ok                                              
t211-elemrestriction .. ok                                              
t209-elemrestriction .. ok                                              
t210-elemrestriction .. ok                                              
t212-elemrestriction .. ok                                              
t213-elemrestriction .. ok                                              
t214-elemrestriction .. ok                                              
t215-elemrestriction .. ok                                              
t216-elemrestriction .. ok                                              
t217-elemrestriction .. ok                                              
t219-elemrestriction .. ok                                              
t218-elemrestriction .. ok                                              
t220-elemrestriction .. ok                                              
t230-elemrestriction .. ok                                              
t231-elemrestriction .. ok                                              
t232-elemrestriction .. ok                                              
t233-elemrestriction .. ok                                              
t301-basis ............ ok                                              
t303-basis ............ ok                                              
t304-basis ............ ok                                              
t305-basis ............ ok                                              
t306-basis ............ ok                                              
t234-elemrestriction .. ok                                              
t300-basis ............ ok                                              
t302-basis ............ ok                                              
t307-basis ............ ok                                              
t310-basis ............ ok                                              
t311-basis ............ ok                                              
t312-basis ............ ok                                              
t314-basis ............ ok                                              
t315-basis ............ ok                                              
t316-basis ............ ok                                              
t317-basis ............ ok                                              
t313-basis ............ ok                                              
t320-basis ............ ok                                              
t321-basis ............ ok                                              
t322-basis ............ ok                                              
t323-basis ............ ok                                              
t324-basis ............ ok                                              
t325-basis ............ ok                                              
t318-basis ............ ok                                              
t330-basis ............ ok                                              
t331-basis ............ ok                                              
t332-basis ............ ok                                              
t340-basis ............ ok                                              
t341-basis ............ ok                                              
t342-basis ............ ok                                              
t319-basis ............ ok                                              
t350-basis ............ ok                                              
t353-basis ............ ok                                              
t355-basis ............ ok                                              
t351-basis ............ ok                                              
t352-basis ............ ok                                              
t354-basis ............ ok                                              
t356-basis ............ ok                                              
t360-basis ............ ok                                              
t357-basis ............ ok                                              
t362-basis ............ ok                                              
t363-basis ............ ok                                              
t361-basis ............ ok                                              
t364-basis ............ ok                                              
t400-qfunction ........ ok                                              
t402-qfunction ........ ok                                              
t401-qfunction ........ ok                                              
t403-qfunction ........ ok                                              
t404-qfunction ........ ok                                              
t405-qfunction ........ ok                                              
t407-qfunction ........ ok                                              
t406-qfunction ........ ok                                              
t408-qfunction ........ ok                                              
t409-qfunction ........ ok                                              
t365-basis ............ ok                                              
t410-qfunction ........ ok                                              
t411-qfunction ........ ok                                              
t413-qfunction ........ ok                                              
t412-qfunction ........ ok                                              
t414-qfunction ........ ok                                              
t415-qfunction ........ ok                                              
t500-operator ......... ok                                              
t501-operator ......... ok                                              
t502-operator ......... ok                                              
t504-operator ......... ok                                              
t503-operator ......... ok                                              
t505-operator ......... ok                                              
t508-operator ......... ok                                              
t507-operator ......... ok                                              
t506-operator ......... ok                                              
t509-operator ......... ok                                              
t510-operator ......... ok                                              
t511-operator ......... ok                                              
t521-operator ......... ok                                              
t520-operator ......... ok                                              
t525-operator ......... ok                                              
t522-operator ......... ok                                              
t523-operator ......... ok                                              
t526-operator ......... ok                                              
t524-operator ......... ok                                              
t530-operator ......... ok                                              
t531-operator ......... ok                                              
t533-operator ......... ok                                              
t532-operator ......... ok                                              
t534-operator ......... ok                                              
t535-operator ......... ok                                              
t536-operator ......... ok                                              
t537-operator ......... ok                                              
t539-operator ......... ok                                              
t538-operator ......... ok                                              
t540-operator ......... ok                                              
t541-operator ......... ok                                              
t550-operator ......... ok                                              
t551-operator ......... ok                                              
t552-operator ......... ok                                              
t553-operator ......... ok                                              
t560-operator ......... ok                                              
t561-operator ......... ok                                              
t562-operator ......... ok                                              
t563-operator ......... ok                                              
t554-operator ......... ok                                              
t564-operator ......... ok                                              
t566-operator ......... ok                                              
t565-operator ......... ok                                              
t568-operator ......... ok                                              
t567-operator ......... ok                                              
t570-operator ......... ok                                              
t569-operator ......... ok                                              
t580-operator ......... ok                                              
t581-operator ......... ok                                              
t582-operator ......... ok                                              
t590-operator ......... ok                                              
t583-operator ......... ok                                              
t591-operator ......... ok                                              
t592-operator ......... ok                                              
t595-operator ......... ok                                              
t593-operator ......... ok                                              
t594-operator ......... ok                                              
ex3-volume ............ ok                                              
ex1-volume ............ ok                                              
ex2-surface ........... ok   
All tests successful.
Files=187, Tests=209, 169 wallclock secs ( 1.02 usr  0.39 sys + 1014.68 cusr 520.94 csys = 1537.03 CPU)
Result: PASS

Ratel tests

Command

make -j prove PROVE_OPTS="-j4" CEED_BACKENDS="/gpu/hip/ref /gpu/hip/shared /gpu/hip/gen" NPROC_POOL=4

Running unit tests
- Testing with libCEED backends: /gpu/hip/ref /gpu/hip/shared /gpu/hip/gen
- Testing on 1 processes
prove -j4 --exec '/opt/cray/pe/python/3.11.7/bin/python3 tests/junit.py --petsc-arch arch-tuolumne-kokkos-O-64 --ceed-backends /gpu/hip/ref /gpu/hip/shared /gpu/hip/gen  --mode tap --nproc 1 --pool-size 4' t000-init t001-view t002-view t003-ts-monitor t004-ts-checkpoint t010-eigensolver t050-mpm t100-static-elasticity t101-static-elasticity t102-static-elasticity t103-static-elasticity t110-static-elasticity t111-static-elasticity t120-static-elasticity t121-static-elasticity t122-static-elasticity t123-static-elasticity t130-static-elasticity-volume t203-quasistatic-elasticity-ts-monitor t211-quasistatic-elasticity t221-quasistatic-elasticity t222-quasistatic-elasticity ex01-static ex02-quasistatic ex03-dynamic
t000-init ............................... ok                            
t001-view ............................... ok                            
t010-eigensolver ........................ ok                            
t003-ts-monitor ......................... ok                            
t002-view ............................... ok                            
t101-static-elasticity .................. ok                            
t100-static-elasticity .................. ok                            
t050-mpm ................................ ok                            
t103-static-elasticity .................. ok                            
t102-static-elasticity .................. ok                            
t110-static-elasticity .................. ok                            
t111-static-elasticity .................. ok                            
t122-static-elasticity .................. ok                            
t004-ts-checkpoint ...................... ok                            
t120-static-elasticity .................. ok                            
t123-static-elasticity .................. ok                            
t130-static-elasticity-volume ........... ok                            
t121-static-elasticity .................. ok                            
t211-quasistatic-elasticity ............. ok                            
t222-quasistatic-elasticity ............. ok                            
t221-quasistatic-elasticity ............. ok                            
t203-quasistatic-elasticity-ts-monitor .. ok                            
ex03-dynamic ............................ ok                            
ex01-static ............................. ok                            
ex02-quasistatic ........................ ok     
All tests successful.
Files=25, Tests=248, 1411 wallclock secs ( 0.21 usr  0.05 sys + 8633.75 cusr 1058.31 csys = 9692.32 CPU)
Result: PASS