[WIP] Add ReadDensityFromFile capabilities - Test 1 (3D only)

[sevicky]

Adding the capability to read a density from openpmd file to use in particle deposition. The current version has only 3D capabilities and is not quite correct with the indexing, which I am continuing to work on but am putting this version in the pull request. Edited files are InjectorDensity.H, InjectorDensity.cpp, and SpeciesUtils.cpp, with most of the changes in the former two. I have tested it using the uniform_plasma example script to give a rho (density) file which is then read in by another input file.

Close #618
Close #3948

Added by @RemiLehe: TODO:

• Add automated test: read an openPMD file with multiple MPI ranks ; check that the density of particles corresponds to the expected density.
• Fix error on GPU
• Avoid out-of-bounds when reading the array.
• Handle the case of the moving window, whereby we need to periodically read the file.

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[sevicky]

The current version in this pull request works for to read a 3D openpmd file and use the density for another simulation. There is no parallelization and further user input options to be implemented.

[sevicky]

Linked here are the compilation script, input files, and Jupyter notebook that was used in development.

Compilation script:
compile_script.txt

Files to create the data used in the tests:
plasma_test.txt
plasma_test_gradient.txt

input files run to test the parabolic and asymmetric cases respectively:
file_input.txt
file_asymm.txt

Jupyter Notebook: (actual version is in my Google Drive - 2024-Sarah_Vickers/misc.)
openpmd_read.pdf

[aodhanmci]

@ax3l @sevicky is it possible to input a density in a PICMI input file using a function? e.g. if I pass the function x, y, z, it will return rho(x, y, z) for the grid? this means I can add complicated density profiles directly in the input file e.g. micro/nanostructures that are limited to the functions which can be parsed by amrex

[ax3l]

Hi @aodhanmci,

Yes, this is already possible for the string based parser syntax and does not require the feature in this pull request.

We support loading density profiles with functions following this syntax:

• top link in https://warpx.readthedocs.io/en/latest/usage/parameters.html
• supports outside-of-function, pre-defined and user-added constants
• supports a variety of basic math and logic functions, e.g., exp, sqrt, pow, if, sin, cos, ...
• support temporary variables, e.g., "r=sqrt(x*x+y*y+z*z); if(r>0, ..., ...else...)"
• supports complex multi-line functions and piecewise functions

Here is a laser-ion acceleration example that uses this from PICMI, see density_expression_str: https://warpx.readthedocs.io/en/latest/usage/examples/laser_ion/README.html

You can go pretty complex in those expressions as multi-line strings, but you cannot pass Python functions (def ...(...): #3684). (Added in DM: support such as random number generators are not in the AMReX parser functions but needed for Aodhan's workflow.) I'll check with others if they have already done the latter, otherwise once the PR here is ready it could be a good basis to support that.

[RemiLehe]

It looks like the current code does work (including in cases with multiples boxes), if the openPMD data is C-ordered.

Here are the files that I used to create a C-ordered openPMD file, and then to run the
simulation.

file_asymm.txt
openpmd_create_3d.py.txt

While the simulation produces correct results, it segfaults later on. It does not segfault anymore.
TODO:

• Understand current SegFault
• Reuse function ReadExternalField to avoid code duplication

[RemiLehe]

We could make this a free function, and move it to the Utils folder.

[RemiLehe]

We need to make sure that we are not doing out-of-bound access here.

[aodhanmci]

im unable to compile these commits (error below). Wonder if anyone can help?

/global/homes/a/aodhan/src/warpx/Source/Initialization/PlasmaInjector.cpp(56): error: static assertion failed with "InjectorDensity must be trivially copyable"
static_assert(std::is_trivially_copyable::value,
^

1 error detected in the compilation of "/global/homes/a/aodhan/src/warpx/Source/Initialization/PlasmaInjector.cpp".
gmake[2]: *** [CMakeFiles/lib_2d.dir/build.make:1588: CMakeFiles/lib_2d.dir/Source/Initialization/PlasmaInjector.cpp.o] Error 2
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:1500: CMakeFiles/lib_rz.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:1388: CMakeFiles/lib_2d.dir/all] Error 2

following the 'ordinary' Perlmutter instructions, wonder if anyone can help?

[RemiLehe]

Thanks for reporting this @aodhanmci.
This is also one of the errors that is reported in the automated tests (specifically for compilation on GPU). I am not sure that there is a quick fix. We will need to take some dedicated time to finalize this PR.

The code might work on CPUs though.