When specifying domain sizes using variables, writing output files smashes the stack

[electroflow]

Hi all, thanks for patching the last issue so fast!

When I compile the latest version under Ubuntu 22.04.3 running in WSL i get "*** stack smashing detected ***: terminated" when I specify the domain in terms of a variable and write an output file. The problem does not occour when either not writing out files or specifying the domain size using constants. Example:

my_constants.rho = 1e20
my_constants.eV = 1.6e-19
my_constants.c = 299792458

my_constants.vm = 0.004

my_constants.energy = 10 * eV
my_constants.ve = (2 * energy / m_e) ** 0.5

my_constants.th_e = (2 * energy / m_e) ** 0.5 / c
my_constants.th_p = (2 * energy / m_p) ** 0.5 / c

my_constants.debye = (energy * epsilon0 / rho / q_e ** 2) ** 0.5

warpx.const_dt = 0.1 * debye / ve
warpx.do_electrostatic = labframe

amr.n_cell =  32 32 32

amr.blocking_factor = 32

geometry.dims = 3
geometry.prob_lo = 0.0 0.0 0.0  
geometry.prob_hi = debye debye debye

amr.max_level = 0

stop_time = 2e-10

boundary.field_lo = pec pec periodic 
boundary.field_hi = pec pec periodic
boundary.particle_lo = reflecting reflecting periodic
boundary.particle_hi = reflecting reflecting periodic

particles.species_names = e p

algo.particle_shape = 1

e.species_type = electron
e.injection_style = "NRandomPerCell"
e.num_particles_per_cell = 1
e.momentum_distribution_type = "gaussian"

e.uz_th = th_e
e.uy_th = th_e
e.ux_th = th_e

e.profile = constant
e.density = rho

p.species_type = proton
p.injection_style = "NRandomPerCell"
p.num_particles_per_cell = 1
p.momentum_distribution_type = "gaussian"

p.uz_th = th_p
p.uy_th = th_p
p.ux_th = th_p

p.profile = constant
p.density = rho

diagnostics.diags_names = particlesr

particlesr.intervals = 1000
particlesr.diag_type = Full
particlesr.format = openpmd
particlesr.write_species = 1
particlesr.species = e p

Gives the output

AMReX (23.10) initialized
WarpX (23.10)

    __        __             __  __
    \ \      / /_ _ _ __ _ __\ \/ /
     \ \ /\ / / _` | '__| '_ \\  /
      \ V  V / (_| | |  | |_) /  \
       \_/\_/ \__,_|_|  | .__/_/\_\
                        |_|

Level 0: dt = 1.253410912e-13 ; dx = 7.341317115e-08 ; dy = 7.341317115e-08 ; dz = 7.341317115e-08

Grids Summary:
  Level 0   1 grids  32768 cells  100 % of domain

-------------------------------------------------------------------------------
--------------------------- MAIN EM PIC PARAMETERS ----------------------------
-------------------------------------------------------------------------------
Precision:            | DOUBLE
Particle precision:   | DOUBLE
Geometry:             | 3D (XYZ)
Operation mode:       | Electrostatic
                      | - laboratory frame
                      | - vacuum
-------------------------------------------------------------------------------
Current Deposition:   | Esirkepov
Particle Pusher:      | Boris
Charge Deposition:    | standard
Field Gathering:      | energy-conserving
Particle Shape Factor:| 1
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
For full input parameters, see the file: warpx_used_inputs

MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 5
MLMG: Initial rhs               = 8.632922913e+11
MLMG: Initial residual (resid0) = 8.632922913e+11
MLMG: Iteration   1 Fine resid/bnorm = 0.002166337221
MLMG: Iteration   2 Fine resid/bnorm = 1.492317356e-05
MLMG: Iteration   3 Fine resid/bnorm = 1.629086106e-07
MLMG: Iteration   4 Fine resid/bnorm = 2.027774793e-09
MLMG: Iteration   5 Fine resid/bnorm = 2.784307573e-11
MLMG: Iteration   6 Fine resid/bnorm = 3.991394086e-13
MLMG: Final Iter. 6 resid, resid/bnorm = 0.3445739746, 3.991394086e-13
MLMG: Timers: Solve = 0.058513099 Iter = 0.056221699 Bottom = 0.0004454
*** stack smashing detected ***: terminated
SIGABRT
/usr/bin/addr2line: '/mnt/c/.../warpx.3d.NOMPI.NOACC.DP.PDP.OPMD': No such file
.....
/usr/bin/addr2line: '/mnt/c/.../warpx.3d.NOMPI.NOACC.DP.PDP.OPMD': No such file
See Backtrace.0 file for details

Backtrace.0 contains

Host Name: Host
=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x528aa0) [0x562060f4eaa0]

 1: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x52a96a) [0x562060f5096a]

 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8309f1c520]

 3: /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x7f8309f70a7c]

 4: /lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x7f8309f1c476]

 5: /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7f8309f027f3]

 6: /lib/x86_64-linux-gnu/libc.so.6(+0x896f6) [0x7f8309f636f6]

 7: /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x2a) [0x7f830a01074a]

 8: /lib/x86_64-linux-gnu/libc.so.6(+0x136716) [0x7f830a010716]

 9: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x2c0795) [0x562060ce6795]

10: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x2aaeaf) [0x562060cd0eaf]

11: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x2b1b9e) [0x562060cd7b9e]

12: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x1c1b61) [0x562060be7b61]

13: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x4203cf) [0x562060e463cf]

14: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x4119f3) [0x562060e379f3]

15: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x411d68) [0x562060e37d68]

16: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x35ae39) [0x562060d80e39]

17: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x19a2f0) [0x562060bc02f0]

18: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x83943) [0x562060aa9943]

19: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f8309f03d90]

20: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f8309f03e40]

21: warpx.3d.NOMPI.NOACC.DP.PDP.OPMD(+0x925f5) [0x562060ab85f5]


===== TinyProfilers ======
main()
WarpX::InitData()
Diagnostics::FilterComputePackFlush()
WarpXParticleContainer::DepositCurrent::CurrentDeposition

[ax3l]

Uiuiui, you might have found a bug in the parser.
Let me verify on Linux.

This is how we debug.

I can reproduce the bug and will backtrace it now with:

cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j 12

cd build/bin
gdb -ex r -ex bt --args ./warpx.3d inputs amrex.throw_exception = 1 amrex.signal_handling = 0

seems to point to Source/Particles/WarpXParticleContainer.cpp:531 which is in doEsirkepovDepositionShapeN aka current deposition (as already seen in your Backtrace above).

Issues here usually mean that particles are out-of-bounds / traveling too far, so this could be an earlier issue with fields or initialization that "pushes" the particles far or into NaN territory.

@WeiqunZhang since this issue only occurs with the parser, it could also be a memory violation from this. I would run compiler sanitizers in the next days to see if we get more info from it.

[ax3l]

@electroflow looking at what you do, you might enjoy avoiding the parser and using our Python (PICMI) interface instead. For all kinds of constants and auxiliary calculations, you can perform them in Python there.

[WeiqunZhang]

We could print out my_constants.debye with 17 digits of precision and then set that in inputs as prob_hi to see if there are issues.

[WeiqunZhang]

or specifying the domain size using constants

@electroflow what constants are you using?

[WeiqunZhang]

I think this is too big, warpx.const_dt = 0.1 * debye / ve. At least it's too big for current deposition in particle diagnostics.

[electroflow]

@WeiqunZhang Thank you for the notice! I thought it would be a problem with variables because I did not put in exactly the same values as the ones that were computed in the equations. Thats my bad. My original code was geometry.prob_hi = "4 * debye" "4 * debye" "4 * debye" and it turns out that "4 * debye" is just at the boundary where this problem is occouring. When I replaced it with an estimated number I was just over the boundary.

I can confirm that using a larger domain like "10 * debye" is sufficent to resolve the issue.

@ax3l Thank you for the suggestion of using the PICMI interface, I will give it a try.

[WeiqunZhang]

The underlying issue is still there. If you keep all parameters the same except for using more cells, the issue will likely come back because dx has become smaller while dt is still the same. @RemiLehe

[ax3l]

@electroflow Not that I see it: Yes, as Weiqun suggested the issue could be that your warpx.const_dt does not fulfill the CFL criteria of the default field solvers you are using. Instead of warpx.const_dt, we usually use the warpx.cfl option to simplify fulfilling the dt to dx/dy/dz relation for em solvers - that way you just need to calculate the spatial resolution (amr.n_cell) and the temporal one follows from it.

[WeiqunZhang]

This uses the electrostatic solver. In the code, there is an assertion requiring const_dt.