CarpetX: use pinned Particles in CarpetX_Interpolate

[lwJi]

No description provided.

[eschnett]

Why do you want to use pinned particles? What is the advantage?

[lwJi]

Why do you want to use pinned particles? What is the advantage?

I was using the Multipole which uses the interpolator when interpolating Psi4 on to a few spheres. I noticed that running it on GPU (nvidia) is much slower compare to run it on CPU. So I timer all the functions carefully and found out these segments of codes cause the slowdown. Then I post my problem on AMREX discussion and the AMReX people suggest to use pinned memory to solve the issue, which it did.
AMReX-Codes/amrex#4506

[eschnett]

Does this improve the speed? I do not understand how it works.

The code that uses pinned memory runs in the CPU, it's the code inserting particles into the AMReX data structures. In the end, these data structures are copied out of the pinned and into the regular memory again.

[eschnett]

Do you have a test case?

[rhaas80]

Does this improve the speed? I do not understand how it works.

The code that uses pinned memory runs in the CPU, it's the code inserting particles into the AMReX data structures. In the end, these data structures are copied out of the pinned and into the regular memory again.

It seems to mostly follow the code in AMReX-Codes/amrex#4506 (reply in thread) which was judged "basic approach looks right to me" by the AMReX crowd). But not tested by them.

[lwJi]

The fix to the old version worked and it was much faster.
I'm testing the current fix now and make sure it works and improve the speed as well

[lwJi]

Does this improve the speed? I do not understand how it works.

The code that uses pinned memory runs in the CPU, it's the code inserting particles into the AMReX data structures. In the end, these data structures are copied out of the pinned and into the regular memory again.

One of the comments from the AMReX discussion mentioned that push_back actually launches a GPU kernel

[eschnett]

#354