Interpolation Using Particles #141

tamaraevst · 2025-08-22T19:45:21Z

Hello,

This is my preliminary attempt to replace GRChombo's AMRInterpolator with amrex particles, subsequently called as ParticleInterpolators (although we can rename it to whatever you like later on). The relevant folder that will eventually replace the AMRInterpolator is located in Source/ParticleInterpolators. I borrowed InterpolationQuery.hpp from the original AMRInterpolator, although renamed it to InterpolationQueryParticle.hpp to not have conflicts, as the AMRInterpolator still remains in the repo.

The key features:

Lagrange interpolation is implemented as a separate class.
A test inTests/using a polynomial is implemented to test the Lagrange interpolation.
Particles are boundary aware. So if symmetric BCs are employed and the query point lies on the symmetric side and outside of the amrex grid, the particle is pushed back into the computational domain. The particle does store the value of the interpolated field with the required parity sign applied.
CustomExtraction is ported and for now supports interpolation of one field for a given query of points.
The BinaryBH example uses this CustomExtraction to interpolate $\chi$ along the line. Some arguments here are hard-coded to random values and are not user-specifiable; but this can be fixed very easily and is most likely the least of our problems.

TO-DO for me on a fresh head:

Modify check_domain() function that checks whether we are inside domain to be parity aware; otherwise it throws errors when we ask to interpolate inside the domain but on the side where symmetries are applied.
So far the code compiles on GPUs, but I need to run an example (I am having some technical difficulties). On CPUs everything is fine.

Please let me know your thoughts.

…de the GPU loops of the particle interpolators

github-actions · 2025-08-22T19:45:31Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/ParticleInterpolators/ParticleInterpolators.impl.hpp
Tests/LagrangeTest/PolynomialTest.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2025-08-22T19:47:24Z

Cpp-Linter Report ⚠️

Some files did not pass the configured checks!

clang-tidy (v19.1.1) reports: 38 concern(s)

Source/GRTeclynCore/AMReXParameters.hpp:229:57: warning: [readability-redundant-string-cstr]

redundant call to 'c_str'

  229 |             if (!((N_full > 0 || N > 0) && !pp.contains(name.c_str()) &&
      |                                                         ^~~~~~~~~~~~
      |                                                         name

Source/GRTeclynCore/AMReXParameters.hpp:230:32: warning: [readability-redundant-string-cstr]

redundant call to 'c_str'

  230 |                   !pp.contains(name_full.c_str())) &&
      |                                ^~~~~~~~~~~~~~~~~
      |                                name_full

Source/GRTeclynCore/AMReXParameters.hpp:231:58: warning: [readability-redundant-string-cstr]

redundant call to 'c_str'

  231 |                 !((N_full < 0 && N < 0) && !(pp.contains(name.c_str()) &&
      |                                                          ^~~~~~~~~~~~
      |                                                          name

Source/GRTeclynCore/AMReXParameters.hpp:232:58: warning: [readability-redundant-string-cstr]

redundant call to 'c_str'

  232 |                                              pp.contains(name_full.c_str()))))
      |                                                          ^~~~~~~~~~~~~~~~~
      |                                                          name_full

Source/GRTeclynCore/AMReXParameters.hpp:241:33: warning: [readability-redundant-string-cstr]

redundant call to 'c_str'

  241 |                 if (pp.contains(name_full.c_str()))
      |                                 ^~~~~~~~~~~~~~~~~
      |                                 name_full

Source/ParticleInterpolator/InterpolationQueryParticle.hpp:37:48: warning: [cppcoreguidelines-use-default-member-init]

use default member initializer for 'm_coords'

   37 |     std::array<const double *, AMREX_SPACEDIM> m_coords;
      |                                                ^       
      |                                                        {}
   38 |     comp_map_t m_comps;
   39 | 
   40 |   public:
   41 |     InterpolationQueryParticle(int num_points)
   42 |         : m_num_points(num_points), m_coords{}
      |                                     ~~~~~~~~~~

Source/ParticleInterpolator/InterpolationQueryParticle.hpp:47:5: warning: [modernize-use-nodiscard]

function 'coords' should be marked [[nodiscard]]
```
   47 |     inline const double *coords(int dim) const
      |     ^
      |     [[nodiscard]] 
```
Source/ParticleInterpolator/InterpolationQueryParticle.hpp:47:5: warning: [readability-redundant-inline-specifier]

function 'coords' has inline specifier but is implicitly inlined
```
   47 |     inline const double *coords(int dim) const
      |     ^~~~~~
```
Source/ParticleInterpolator/InterpolationQueryParticle.hpp:97:5: warning: [readability-redundant-inline-specifier]

function 'numComps' has inline specifier but is implicitly inlined
```
   97 |     inline int numComps()
      |     ^~~~~~
```
Source/ParticleInterpolator/InterpolationQueryParticle.hpp:109:19: warning: [readability-redundant-inline-specifier]

function 'numPoints' has inline specifier but is implicitly inlined
```
  109 |     [[nodiscard]] inline size_t numPoints() const { return m_num_points; }
      |                   ^~~~~~
```
Source/ParticleInterpolator/InterpolationQueryParticle.hpp:111:5: warning: [readability-redundant-inline-specifier]

function 'compsBegin' has inline specifier but is implicitly inlined
```
  111 |     inline iterator compsBegin() { return m_comps.begin(); }
      |     ^~~~~~
```
Source/ParticleInterpolator/InterpolationQueryParticle.hpp:113:5: warning: [readability-redundant-inline-specifier]

function 'compsEnd' has inline specifier but is implicitly inlined
```
  113 |     inline iterator compsEnd() { return m_comps.end(); }
      |     ^~~~~~
```
Source/ParticleInterpolator/Lagrange.hpp:27:9: warning: [cppcoreguidelines-avoid-c-arrays]

do not declare C-style arrays, use std::array<> instead
```
   27 |         int stencil[N];
      |         ^
```
Source/ParticleInterpolator/Lagrange.hpp:41:9: warning: [cppcoreguidelines-avoid-c-arrays]

do not declare C-style arrays, use std::array<> instead
```
   41 |         amrex::Real rel_pos[N];
      |         ^
```
Source/ParticleInterpolator/Lagrange.hpp:56:5: warning: [cppcoreguidelines-avoid-c-arrays]

do not declare C-style arrays, use std::array<> instead
```
   56 |     amrex::Real weights_x[N];
      |     ^
```
Source/ParticleInterpolator/Lagrange.hpp:57:5: warning: [cppcoreguidelines-avoid-c-arrays]

do not declare C-style arrays, use std::array<> instead
```
   57 |     amrex::Real weights_y[N];
      |     ^
```
Source/ParticleInterpolator/Lagrange.hpp:58:5: warning: [cppcoreguidelines-avoid-c-arrays]

do not declare C-style arrays, use std::array<> instead
```
   58 |     amrex::Real weights_z[N];
      |     ^
```

Source/ParticleInterpolator/Lagrange.hpp:60:22: warning: [cppcoreguidelines-pro-type-member-init]

constructor does not initialize these fields: i0, j0, k0, weights_x, weights_y, weights_z

   60 |     AMREX_GPU_DEVICE AMREX_FORCE_INLINE Lagrange() {};
      |                      ^
/home/runner/work/GRTeclyn/GRTeclyn/amrex/Src/Base/AMReX_Extension.H:110:28: note: expanded from macro 'AMREX_FORCE_INLINE'
  110 | #define AMREX_FORCE_INLINE inline __attribute__((always_inline))
      |                            ^

Source/ParticleInterpolator/Lagrange.hpp:60:41: warning: [modernize-use-equals-default]

use '= default' to define a trivial default constructor

   60 |     AMREX_GPU_DEVICE AMREX_FORCE_INLINE Lagrange() {};
      |                                         ^          ~~
      |                                                    = default

Source/ParticleInterpolator/Lagrange.hpp:64:30: warning: [readability-identifier-length]

parameter name 'p' is too short, expected at least 3 characters
```
   64 |     compute_weights(const P &p,
      |                              ^
```
Source/ParticleInterpolator/Lagrange.hpp:72:25: warning: [readability-identifier-length]

variable name 'lx' is too short, expected at least 3 characters
```
   72 |             amrex::Real lx =
      |                         ^
```
Source/ParticleInterpolator/Lagrange.hpp:75:27: warning: [readability-identifier-length]

variable name 'ly' is too short, expected at least 3 characters
```
   75 |             , amrex::Real ly =
      |                           ^
```
Source/ParticleInterpolator/Lagrange.hpp:78:27: warning: [readability-identifier-length]

variable name 'lz' is too short, expected at least 3 characters
```
   78 |             , amrex::Real lz =
      |                           ^
```
Source/ParticleInterpolator/ParticleInterpolator.hpp:18:7: warning: [cppcoreguidelines-pro-type-member-init]

constructor does not initialize these fields: m_query
```
   18 | class ParticleInterpolator
      |       ^
```
Source/ParticleInterpolator/ParticleInterpolator.hpp:55:45: warning: [readability-redundant-member-init]

initializer for member 'm_bc_params' is redundant
```
   55 |     BoundaryConditions::params_t m_bc_params{};
      |                                             ^~
```
Source/ParticleInterpolator/ParticleInterpolator.hpp:82:49: warning: [readability-identifier-length]

parameter name 'lo' is too short, expected at least 3 characters
```
   82 |     reflect_particle(amrex::Real x, amrex::Real lo, amrex::Real hi,
      |                                                 ^
```
Source/ParticleInterpolator/ParticleInterpolator.hpp:82:65: warning: [readability-identifier-length]

parameter name 'hi' is too short, expected at least 3 characters
```
   82 |     reflect_particle(amrex::Real x, amrex::Real lo, amrex::Real hi,
      |                                                                 ^
```
Source/ParticleInterpolator/ParticleInterpolator.hpp:90:52: warning: [readability-identifier-length]

parameter name 'p' is too short, expected at least 3 characters
```
   90 |     void set_derived_var_parity(int comp, BCParity p);
      |                                                    ^
```
Source/ParticleInterpolator/ParticleInterpolator.hpp:143:67: warning: [readability-identifier-length]

parameter name 'mf' is too short, expected at least 3 characters
```
  143 |     void interpolate_to_particle_helper(int lev, amrex::MultiFab &mf,
      |                                                                   ^
```
Tests/Common/SimulationParameters.hpp:17:5: warning: [cppcoreguidelines-pro-type-member-init]

constructor does not initialize these fields: num_points, verbosity
```
   17 |     SimulationParameters(GRParmParse &pp) : SimulationParametersBase(pp)
      |     ^
```
Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:32:10: warning: [cppcoreguidelines-explicit-virtual-functions]

annotate this function with 'override' or (rarely) 'final'
```
   32 |     void initData()
      |          ^         
      |                     override
```

Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:47:49: warning: [bugprone-easily-swappable-parameters]

2 adjacent parameters of 'operator()' of similar type ('int') are easily swapped by mistake

   47 |                            [=] AMREX_GPU_DEVICE(int box_no, int i, int j, int k)
      |                                                 ^~~~~~~~~~~~~~~~~
/home/runner/work/GRTeclyn/GRTeclyn/GRTeclyn/Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:47:53: note: the first parameter in the range is 'box_no'
   47 |                            [=] AMREX_GPU_DEVICE(int box_no, int i, int j, int k)
      |                                                     ^~~~~~
/home/runner/work/GRTeclyn/GRTeclyn/GRTeclyn/Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:47:65: note: the last parameter in the range is 'i'
   47 |                            [=] AMREX_GPU_DEVICE(int box_no, int i, int j, int k)
      |                                                                 ^

Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:65:10: warning: [cppcoreguidelines-explicit-virtual-functions]

annotate this function with 'override' or (rarely) 'final'

   65 |     void specificEvalRHS(amrex::MultiFab &a_soln, amrex::MultiFab &a_rhs,
      |          ^
   66 |                          const double a_time)
      |                                              
      |                                               override

Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:65:26: warning: [bugprone-easily-swappable-parameters]

2 adjacent parameters of 'specificEvalRHS' of similar type ('amrex::MultiFab &') are easily swapped by mistake

   65 |     void specificEvalRHS(amrex::MultiFab &a_soln, amrex::MultiFab &a_rhs,
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/runner/work/GRTeclyn/GRTeclyn/GRTeclyn/Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:65:43: note: the first parameter in the range is 'a_soln'
   65 |     void specificEvalRHS(amrex::MultiFab &a_soln, amrex::MultiFab &a_rhs,
      |                                           ^~~~~~
/home/runner/work/GRTeclyn/GRTeclyn/GRTeclyn/Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorLevel.hpp:65:68: note: the last parameter in the range is 'a_rhs'
   65 |     void specificEvalRHS(amrex::MultiFab &a_soln, amrex::MultiFab &a_rhs,
      |                                                                    ^~~~~

Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorUnitTest.cpp:142:9: warning: [cppcoreguidelines-avoid-c-arrays]

do not declare C-style arrays, use std::array<> instead
```
  142 |         ParticleInterpolator<1>::DerivedParity parities[] = {
      |         ^
```
Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorUnitTest.cpp:147:47: warning: [cppcoreguidelines-pro-bounds-array-to-pointer-decay]

do not implicitly decay an array into a pointer; consider using gsl::array_view or an explicit cast instead
```
  147 |                                    verbosity, parities);
      |                                               ^
```
Tests/ParticleInterpolatorUnitTest/PolynomialDerivedQuantity.hpp:27:70: warning: [readability-identifier-length]

parameter name 'c' is too short, expected at least 3 characters
```
   27 |     static void set_center(const std::array<double, AMREX_SPACEDIM> &c)
      |                                                                      ^
```
Tests/ParticleInterpolatorUnitTest/PolynomialDerivedQuantity.hpp:29:18: warning: [readability-identifier-length]

loop variable name 'd' is too short, expected at least 2 characters
```
   29 |         for (int d = 0; d < AMREX_SPACEDIM; ++d)
      |                  ^
```

Have any feedback or feature suggestions? Share it here.

…t mem where needed)

github-actions · 2025-08-25T00:12:49Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/ParticleInterpolators/ParticleInterpolators.impl.hpp
Tests/LagrangeTest/PolynomialTest.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

mirenradia · 2025-09-03T14:54:12Z

Thanks for opening this PR, @tamaraevst. It looks like a good start. 🙂

When I do a proper review, I will have more detailed comments on the code.

For now, I have some broader points to raise:

Persistent particles

Since we usually want to interpolate the same variables at the same points repeatedly during a simulation, we don't want to be adding and redistributing particles each time we want to interpolate. Can you add a method to add a "persistent" interpolation query which can be made once with the particles added just the first time and re-used subsequently? We will need to remember to "redistribute" after a regrid but don't need to if a regrid hasn't happened since the last interpolation call. I believe redistribution is expensive so we should try to minimise it when we can.

Maybe the ParticleInterpolators class can store a std::unordered_map<std::string, InterpolationQueryParticle> object. The first template argument can be a std::string "key" for the interpolation query which subsequent callers of the interpolator can use to re-interpolate the same variables at the same point? This is just a very quick thought so there may be a better way.

Derivatives

The AMRInterpolator in GRChombo supported interpolating spatial derivatives in each direction of the variables (and this is still present in InterpolationQueryParticle). We used this feature in GRChombo in the apparent horizon finder so it is probably something we will eventually want.

We might want to defer this to a future PR and just have an error for the moment if derivatives are requested.

Derived Quantities

A key feature we will want is being able to interpolate derived quantities e.g. $\Psi_4$. Unfortunately, unlike in GRChombo where we had a persistent GRLevelData (MultiFabs in AMReX) for each level for the "diagnostic" quantities, now in GRTeclyn MultiFabs for derived quantities are created on the fly as they are needed. We will probably need to add a function to GRAMR to allow it to create a vector of MultiFabs (1 per level) and also compute a derived quantity on each level's MultiFab. We can probably defer this to a separate PR.

For now, in this PR perhaps it's worth adding the ability to interpolate data from an arbitrary amrex::Vector or std::vector of MultiFabs assuming they have exactly the same geometry, box layout and distribution mapping as that of the state date?

Unit test

Can you make the unit test run on a full "mini" example (i.e. with GRAMR and multiple levels) as for the old GRChombo AMRInterpolator test?

Development practices

Would you be able to follow our conventions for commit messages?
Your commits have lots of different names and email addresses. Would you be able to make them all consistent? You can set this up on each system in your gitconfig.
Would you be able to set up pre-commit to ensure all of your commits are correctly formatted?

- Leave an error message for derivative queries - Track Redistribute() flag and allow the user to override it Other relevant modifications to the interpolation class: - Simplify copying from device to host memory in populate_from_query() - Make check_domain() parity compatible

github-actions · 2025-09-10T14:47:30Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/ParticleInterpolators/ParticleInterpolators.impl.hpp
Tests/LagrangeTest/PolynomialTest.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

Rewrite a unit test for interpolation to include use of GRAMR and some parameter inputs. More details: - Derived vars interpolation handled via Multifabs (for each level) - We do not have parity supported for derived vars (throw errors in these cases for now) - Interpolation test handles a polynomial as a derived variable and checks for errors using analytical expression

github-actions · 2025-09-12T16:03:40Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/ParticleInterpolators/ParticleInterpolators.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2025-09-12T18:56:49Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/DerivativeSetup.hpp
Source/ParticleInterpolators/ParticleInterpolators.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2025-09-13T15:37:06Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/DerivativeSetup.hpp
Source/ParticleInterpolators/ParticleInterpolators.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

…r GPU

github-actions · 2025-09-13T17:04:17Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/DerivativeSetup.hpp
Source/ParticleInterpolators/ParticleInterpolators.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2025-09-13T18:20:36Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/DerivativeSetup.hpp
Source/ParticleInterpolators/ParticleInterpolators.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2025-12-12T14:46:14Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/DerivativeSetup.hpp
Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2025-12-12T14:56:24Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/DerivativeSetup.hpp
Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

in Particle Interpolator test

github-actions · 2025-12-18T15:23:03Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/DerivativeSetup.hpp
Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

* Remove helper function for MultiFabs in GRAMR * Move Derivative* into ParticleInterpolator * Remove unnecessary param file

github-actions · 2025-12-18T16:47:42Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

compilation of an AMREX_GPU_DEVICE lambda is being weird :(

github-actions · 2025-12-18T16:56:34Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

tamaraevst · 2025-12-18T17:07:38Z

So I think I have now addressed all of the review comments to the best of my knowledge. I tried to go through the list of comments/new additions again throughly, so hopefully I have not missed anything.

I have now also implemented the ability for different ranks to have their own set of query points. This incurred a substantial rewrite of ParticleInterpolator functions and additions of new helper functions. I copied over lots of MPI* related classes from the old GRChombo code and added comments/explanations within those files (the way I understood them), as I was confused about the different structures used there and the logic. We may need to iterate over this part, as I do not rule out I missed other more optimal ways to implement this feature.

One remaining item is where to store the particle index (integer AoS or SoA), please see my comment here . I do not see an immediate advantage of using SoA, as we still need to copy AoS, but maybe I have missed something.

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

…tion in particle interpolators

github-actions · 2026-01-06T21:20:29Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2026-01-09T20:14:17Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

mirenradia

Sorry for taking a while to re-review it. Because it's such a large PR, I've found it quite mentally taxing.

Thanks for making the requested changes from my previous review and the few interim comments. I think we're getting closer! I think most of my comments are now related to naming of variables/functions.

In addition to the comments on files, I have some points which didn't fit elsewhere:

I find the term "owner" to refer the rank that queried the particle confusing. In my head the owner is the proc which currently has the particle. I think we should remove all uses of the word "owner" and instead just refer to "query" and "answer ranks/procs.
There are still files in this PR which only have whitespace changes. Even though they won't satisfy the pre-commit checks, please can you revert them to how they are on develop so they are not part of this PR?

Unfortunately, I haven't had any time to profile the particle interpolator yet. I will hopefully get to that in 2 weeks.

Source/BlackHoles/BHAMR.hpp

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

mirenradia · 2026-01-06T17:08:44Z

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

+template <int num_components>
+void ParticleInterpolator<num_components>::setup(
+    GRAMR *gr_amr_ptr, const BoundaryConditions::params_t &a_bc_params,
+    bool a_verbosity, const DerivedParity *parities)


Can you make the final argument a vector of DerivedParitys where the default argument is just a vector of length 0?

I will wait on your reply re how we want to store DerivedParity (see also comment in the ParticleInterpolator) test. Currently, it is {component_number, parity}. So it does not matter in which order the user provides parities, so long as the map comp -> parity is consistent. But maybe for simplicity we want this to be just a vector of parities and implicitly enforce that this is in the right order for each component. Let me know, I will make the default arg accordingly, if needed.

I'm now wondering if it makes sense to store the parity in InterpolationQueryParticle (specifically as a new member in the out_t struct) as then you would set it at the same time as you add the component to the query. What do you think?

That should work neatly. I will have a go at implementing this.

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorUnitTest.cpp

Tests/ParticleInterpolatorUnitTest/ParticleInterpolatorUnitTest.inputs

Source/ParticleInterpolator/MPIContext.impl.hpp

mirenradia

Sorry for taking a while to re-review it. Because it's such a large PR, I've found it quite mentally taxing.

Thanks for making the requested changes from my previous review and the few interim comments. I think we're getting closer! I think most of my comments are now related to naming of variables/functions.

In addition to the comments on files, I have some points which didn't fit elsewhere:

I find the term "owner" to refer the rank that queried the particle confusing. In my head the owner is the proc which currently has the particle. I think we should remove all uses of the word "owner" and instead just refer to "query" and "answer ranks/procs.
There are still files in this PR which only have whitespace changes. Even though they won't satisfy the pre-commit checks, please can you revert them to how they are on develop so they are not part of this PR?

Unfortunately, I haven't had any time to profile the particle interpolator yet. I will hopefully get to that in 2 weeks.

* Renaming of functions * Minor change in ParticleInterpolator test parameter inputs * Changing how some arrays are getting written into

github-actions · 2026-01-29T16:48:12Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

github-actions · 2026-01-30T15:28:07Z

This PR modifies the following files which are ignored by .lint-ignore:

Source/AMRInterpolator/Lagrange_old.hpp
Source/AMRInterpolator/Lagrange_old.impl.hpp
Source/ParticleInterpolator/MPIContext.impl.hpp
Source/ParticleInterpolator/MPILayout.impl.hpp
Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

Please consider removing the corresponding patterns from .lint-ignore so that these files can be linted.

mirenradia · 2026-02-02T14:09:29Z

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

+    InterpolationQueryParticle &query, VariableType variable_type,
+    const std::string &name_derived, double time_derived /*=0.0*/)


I've just realised that we're passing the VariableType in the query and also as another argument. Can we add a check/assertion in the InterpolationQueryParticle class that all of the VariableTypes in the query are the same and then use that value here rather than passing a separate VariableType argument?

As for a given InterpolationQueryParticle we will be using the same VariableType, maybe I move it out of out_t struct? I can just make variable type a private class member and check that it never changes whenever someone calls addComp() again?

And add a getter to get it outside of the class...

mirenradia · 2026-02-06T11:05:46Z

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp

-    amrex::Gpu::copyAsync(amrex::Gpu::hostToDevice, y, y + n, y_d.begin());
+    // coords on device
+    amrex::GpuArray<amrex::Gpu::DeviceVector<double>, AMREX_SPACEDIM> coords_d;
+    amrex::GpuArray<const double *, AMREX_SPACEDIM> coords_d_ptr = query_coords;


I'm a bit confused by this line since query_coords is an array of host pointers and you set the correct values in the loop immediately below.

Perhaps it would be best to just default initialize (which will set them to nullptr) this using {} to minimise confusion?

Suggested change

amrex::GpuArray<const double *, AMREX_SPACEDIM> coords_d_ptr = query_coords;

amrex::GpuArray<const double *, AMREX_SPACEDIM> coords_d_ptr{};

tamaraevst and others added 6 commits August 22, 2025 09:25

new branch with particle interpolators

81d603f

hopefully fix the gitlalab compilation issues with lambda errors insi…

b887b14

…de the GPU loops of the particle interpolators

the fight with GPU loops continues

77ad91a

remove print statements incompatible with GPUs

7ddf2d6

make compatibility for GPUs

a8b293a

remove file we do not need

a3e2ca1

tamaraevst requested review from KAClough and mirenradia August 22, 2025 19:45

tamaraevst added the enhancement New feature or request label Aug 22, 2025

tamaraevst added this to GRTeclyn Aug 22, 2025

tamaraevst linked an issue Aug 22, 2025 that may be closed by this pull request

Port AMRInterpolator #2

Open

make certain functions GPU compatible (cope things from device to hos…

2eaec25

…t mem where needed)

Move SimulationParameters for interpolation test into Tests/Common

bfee34d

Refactor test for interpolation for GPUs

3dd349f

ParticleInterpolators: add a guard for when the coords exist on CPU o…

ed3380a

…r GPU

Perform check for interpolator test in zeroth MPI rank

500beeb

tamaraevst added 2 commits September 13, 2025 15:42

Ensure particle redistribution at the start of interpolation

4d5e38c

Merge with particle_interp

c5ab803

Refactor receive and send buffer logic in ParticleInterpolator

6ec427e

Fix serial logic in particle interpolation

4d31d51

Add partitioning of query points across different ranks when MPI is used

7c27804

in Particle Interpolator test

tamaraevst added 3 commits December 18, 2025 11:34

Address remaing review comments

318aa31

* Remove helper function for MultiFabs in GRAMR * Move Derivative* into ParticleInterpolator * Remove unnecessary param file

Modify GNUmakefile for KG example

e7f0f5a

Revert some files to their initial state from develop branch

992f6f0

Adjust public functions in Particle Interpolator class as CUDA

7985527

compilation of an AMREX_GPU_DEVICE lambda is being weird :(

mirenradia reviewed Jan 5, 2026

View reviewed changes

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp Outdated Show resolved Hide resolved

mirenradia reviewed Jan 5, 2026

View reviewed changes

Source/ParticleInterpolator/ParticleInterpolator.impl.hpp Outdated Show resolved Hide resolved

mirenradia marked this pull request as ready for review January 6, 2026 15:58

mirenradia self-requested a review January 6, 2026 15:58

Remove num_ghosts argument from interpolation_to_particle_helper func…

3cf7591

…tion in particle interpolators

tamaraevst added 2 commits January 9, 2026 15:01

Remove AMREX_USE_MPI guards in ParticleInterpolator test

0b597c4

Remove AoS integer from ParticleContainer in interpolation

8f5a15b

mirenradia requested changes Jan 23, 2026

View reviewed changes

First chunk of comments addressed in ParticleInterpolator

594f9e1

* Renaming of functions * Minor change in ParticleInterpolator test parameter inputs * Changing how some arrays are getting written into

Add copy of coords on device

994dbfa

mirenradia reviewed Feb 2, 2026

View reviewed changes

mirenradia reviewed Feb 6, 2026

View reviewed changes

		InterpolationQueryParticle &query, VariableType variable_type,
		const std::string &name_derived, double time_derived /=0.0/)

	amrex::GpuArray<const double *, AMREX_SPACEDIM> coords_d_ptr = query_coords;
	amrex::GpuArray<const double *, AMREX_SPACEDIM> coords_d_ptr{};

Interpolation Using Particles #141

Are you sure you want to change the base?

Interpolation Using Particles #141

Uh oh!

Conversation

tamaraevst commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 22, 2025

Uh oh!

github-actions bot commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cpp-Linter Report ⚠️

Uh oh!

github-actions bot commented Aug 25, 2025

Uh oh!

mirenradia commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Persistent particles

Derivatives

Derived Quantities

Unit test

Development practices

Uh oh!

github-actions bot commented Sep 10, 2025

Uh oh!

github-actions bot commented Sep 12, 2025

Uh oh!

github-actions bot commented Sep 12, 2025

Uh oh!

github-actions bot commented Sep 13, 2025

Uh oh!

github-actions bot commented Sep 13, 2025

Uh oh!

github-actions bot commented Sep 13, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

tamaraevst commented Dec 18, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 6, 2026

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

mirenradia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mirenradia Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

tamaraevst Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

mirenradia Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

tamaraevst Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mirenradia left a comment

Choose a reason for hiding this comment

tamaraevst commented Aug 22, 2025 •

edited

Loading

github-actions bot commented Aug 22, 2025 •

edited

Loading

mirenradia commented Sep 3, 2025 •

edited

Loading

tamaraevst Feb 3, 2026 •

edited

Loading