try initial suggestions for btd output for each level

[prkkumar]

I modified the things we discussed.

I also created a small 2D input script which I can run locally to see if we get output fields on each level.

input_2d.txt

With these changes, it segfaults at amrex::get_slice_data in

Source/Diagnostics/ComputeDiagFunctors/BackTransformFunctor.cpp:63

[prkkumar]

Current status:

• I get Ex, Ey, Ez, Bx, By, Bz, jx, jy, jz, and rho separately on each level using the attached input script which runs in 2D locally.

input_2d.txt

Here is the Ez field on each level using the attached script:

[prkkumar]

Running the original 3D problem (input.txt) resulted in Segfault. Here is the relevant part of Backtrace

4: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x106327c4]
    (anonymous namespace)::allocateSlice(int, amrex::MultiFab const&, int, amrex::Geometry const&, double, amrex::Vector<int, std::allocator<int> >&) at ??:?

 5: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x106366a8]
    amrex::get_slice_data(int, double, amrex::MultiFab const&, amrex::Geometry const&, int, int, bool) at ??:?

 6: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x1038b9d8]
    BackTransformFunctor::operator()(amrex::MultiFab&, int, int) const at ??:?

 7: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x10414278]
    Diagnostics::ComputeAndPack() at ??:?

 8: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x10414758]
    Diagnostics::FilterComputePackFlush(int, bool) at ??:?

 9: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x103588bc]
    MultiDiagnostics::FilterComputePackFlush(int, bool, bool) at ??:?

10: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x100e93e0]
    WarpX::Evolve(int) at ??:?

11: ./warpx.3d.MPI.CUDA.DP.PDP.OPMD.PSATD.QED() [0x100389a4]
    main at ??:?

12: /lib64/power9/libc.so.6(+0x24078) [0x200011ba4078]

13: /lib64/power9/libc.so.6(__libc_start_main+0xb4) [0x200011ba4264]


===== TinyProfilers ======
main()
REG::WarpX::Evolve()
WarpX::Evolve()
WarpX::Evolve::step
Diagnostics::FilterComputePackFlush()
amrex::get_slice_data

[RevathiJambunathan]

@prkkumar your 3D input file has

warpx.fine_tag_lo = -5.e-6  -5.e-6  -100.e-6
warpx.fine_tag_hi =  5.e-6   5.e-6  -60.e-6

which is different from the 2D input file that worked above

warpx.fine_tag_lo = -5.e-6  -110.e-6
warpx.fine_tag_hi =  5.e-6   50.e-6

Also, the very first 2D input that you have had the same coordinates as your 3D input in the latest comment, and that crashed too with the segfault at amrex::get_slice_data

This is because,the z-slice location in boosted frame needed for BTD is inside the domain, but not inside the patch.
That is why the 2D input with the wider patch in z worked without a segfault.
I tried to coarsen the 3D simulation and run it with the same patch coordinates as your working 2D input, and that ran without a segfault.

[RevathiJambunathan]

Closing this PR, and re-opening from my fork/branch in #4526