Index out of bounds error for eb_mf_lincc_interp #4693
Description
I am getting a rather odd index out of bounds error when using eb_mf_lincc_interp interpolator with FillPatchTwoLevels when filling 2 ghost cells. The error only occurs when running with more than 7 MPI processes. Initially, I suspected it might be related to amr.max_grid_size, but after testing various grid sizes on a single MPI process, the error did not appear.
Here is the code where I use the interpolation function
amrex::InterpBase* mapper = epsilon_interp; // This is &amrex::eb_mf_lincc_interp
const auto& fgeom = m_sim->Geom(alev);
const auto& cgeom = m_sim->Geom(alev-1);
MultiFab& epsilon = m_sim->get_scalar(alev, "epsilon");
fillEpsilonInflowAndWall(alev, epsilon);
amrex::Vector<amrex::BCRec> crse_bcrec(ncomp);
for (int n = 0; n < ncomp; ++n) {
for (OrientationIter oit; oit; ++oit) {
crse_bcrec[n].set(oit(), amrex::BCType::foextrap);
}
}
PhysBCFunct fbndry_fn(fgeom, {epsilon_bcrec}, GpuBndryFuncFab{FabFillNoOp{}});
PhysBCFunct cbndry_fn(cgeom, crse_bcrec, GpuBndryFuncFab{FabFillNoOp{}});
amrex::FillPatchTwoLevels(
epsilon, IntVect(nghost), time,
{&(m_sim->get_scalar(alev-1, "epsilon"))}, {time},
{&(m_sim->get_scalar(alev, "epsilon"))}, {time},
0, 0, ncomp, cgeom, fgeom, cbndry_fn, 0, fbndry_fn, 0,
m_sim->refRatio(alev-1), mapper, {epsilon_bcrec}, 0
);
The Backtrace
Host Name: ankith-desktop
=== If no file names and line numbers are shown below, one can run
addr2line -Cpfie my_exefile my_line_address
to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
Or one can use amrex/Tools/Backtrace/parse_bt.py.
=== Please note that the line number reported by addr2line may not be accurate.
One can use
readelf -wl my_exefile | grep my_line_address'
to find out the offset for that line.
0: cartMG.ex(+0x531696) [0x64f7a9b4e696]
amrex::BLBackTrace::print_backtrace_info(_IO_FILE*) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_BLBackTrace.cpp:205
1: cartMG.ex(+0x530fb2) [0x64f7a9b4dfb2]
amrex::BLBackTrace::handler(int) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_BLBackTrace.cpp:103
2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x73c0ef242520]
3: /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x73c0ef2969fc]
4: /lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x73c0ef242476]
5: /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x73c0ef2287f3]
6: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa2b9e) [0x73c0ef6a2b9e]
amrex::FabArrayBase::localindex(int) const at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_FabArrayBase.H:121
7: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae20c) [0x73c0ef6ae20c]
amrex::FabArray<amrex::IArrayBox>::clear() at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_FabArray.H:1849
8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xad1e9) [0x73c0ef6ad1e9]
amrex::Array4<int>::index_assert(int, int, int, int) const at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_Array4.H:283 (discriminator 19)
9: /lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x99) [0x73c0ef6ad959]
?? ??:0
10: /lib/x86_64-linux-gnu/libgcc_s.so.1(+0x16884) [0x73c0ef884884]
?? ??:0
11: /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x12d) [0x73c0ef8852dd]
?? ??:0
12: cartMG.ex(+0x6b5d4) [0x64f7a96885d4]
amrex::Array4<double const>::index_assert(int, int, int, int) const at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_Array4.H:294
13: cartMG.ex(+0x663559) [0x64f7a9c80559]
double const& amrex::Array4<double const>::operator()<double const, 0>(int, int, int, int) const at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_Array4.H:144
(inlined by) amrex::mf_cell_cons_lin_interp(int, int, int, int, amrex::Array4<double> const&, int, amrex::Array4<double const> const&, amrex::Array4<double const> const&, int, int, amrex::IntVectND<2> const&) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/AmrCore/AMReX_MFInterp_2D_C.H:201
(inlined by) operator() at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/AmrCore/AMReX_MFInterpolater.cpp:362
14: cartMG.ex(+0x673993) [0x64f7a9c90993]
void amrex::LoopConcurrentOnCpu<amrex::MFCellConsLinInterp::interp(amrex::MultiFab const&, int, amrex::MultiFab&, int, int, amrex::IntVectND<2> const&, amrex::Geometry const&, amrex::Geometry const&, amrex::BoxND<2> const&, amrex::IntVectND<2> const&, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int)::{lambda(int, int, int, int)#9}, 2>(amrex::BoxND<2> const&, int, amrex::MFCellConsLinInterp::interp(amrex::MultiFab const&, int, amrex::MultiFab&, int, int, amrex::IntVectND<2> const&, amrex::Geometry const&, amrex::Geometry const&, amrex::BoxND<2> const&, amrex::IntVectND<2> const&, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int)::{lambda(int, int, int, int)#9} const&) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_Loop.H:83
(inlined by) call_f_intvect_ncomp_cpu<amrex::MFCellConsLinInterp::interp(const amrex::MultiFab&, int, amrex::MultiFab&, int, int, const IntVect&, const amrex::Geometry&, const amrex::Geometry&, const Box&, const IntVect&, const amrex::Vector<amrex::BCRec>&, int)::<lambda(int, int, int, int)>, 2> at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_Loop.H:118
(inlined by) LoopConcurrentOnCpu_impND<2, amrex::MFCellConsLinInterp::interp(const amrex::MultiFab&, int, amrex::MultiFab&, int, int, const IntVect&, const amrex::Geometry&, const amrex::Geometry&, const Box&, const IntVect&, const amrex::Vector<amrex::BCRec>&, int)::<lambda(int, int, int, int)>, 2> at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_Loop.H:541
(inlined by) LoopConcurrentOnCpu<amrex::MFCellConsLinInterp::interp(const amrex::MultiFab&, int, amrex::MultiFab&, int, int, const IntVect&, const amrex::Geometry&, const amrex::Geometry&, const Box&, const IntVect&, const amrex::Vector<amrex::BCRec>&, int)::<lambda(int, int, int, int)>, 2> at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/Base/AMReX_Loop.H:567
15: cartMG.ex(+0x664795) [0x64f7a9c81795]
amrex::MFCellConsLinInterp::interp(amrex::MultiFab const&, int, amrex::MultiFab&, int, int, amrex::IntVectND<2> const&, amrex::Geometry const&, amrex::Geometry const&, amrex::BoxND<2> const&, amrex::IntVectND<2> const&, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/AmrCore/AMReX_MFInterpolater.cpp:247 (discriminator 1)
16: cartMG.ex(+0x73c335) [0x64f7a9d59335]
amrex::EBMFCellConsLinInterp::interp(amrex::MultiFab const&, int, amrex::MultiFab&, int, int, amrex::IntVectND<2> const&, amrex::Geometry const&, amrex::Geometry const&, amrex::BoxND<2> const&, amrex::IntVectND<2> const&, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:16
17: cartMG.ex(+0x6455df) [0x64f7a9c625df]
amrex::FillPatchInterp(amrex::MultiFab&, int, amrex::MultiFab const&, int, int, amrex::IntVectND<2> const&, amrex::Geometry const&, amrex::Geometry const&, amrex::BoxND<2> const&, amrex::IntVectND<2> const&, amrex::MFInterpolater*, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/AmrCore/AMReX_FillPatchUtil.cpp:148
18: cartMG.ex(+0x1c1cf0) [0x64f7a97decf0]
std::enable_if<amrex::IsFabArray<amrex::MultiFab, void>::value, void>::type amrex::FillPatchInterp<amrex::MultiFab>(amrex::MultiFab&, int, amrex::MultiFab const&, int, int, amrex::IntVectND<2> const&, amrex::Geometry const&, amrex::Geometry const&, amrex::BoxND<2> const&, amrex::IntVectND<2> const&, amrex::InterpBase*, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/AmrCore/AMReX_FillPatchUtil_I.H:239
19: cartMG.ex(+0x1bfa8d) [0x64f7a97dca8d]
std::enable_if<amrex::IsFabArray<amrex::MultiFab, void>::value, int>::type amrex::detail::FillPatchTwoLevels_doit<amrex::MultiFab, amrex::PhysBCFunct<amrex::GpuBndryFuncFab<amrex::FabFillNoOp> >, amrex::InterpBase, amrex::NullInterpHook<amrex::FArrayBox>, amrex::NullInterpHook<amrex::FArrayBox> >(amrex::MultiFab&, amrex::IntVectND<2> const&, double, amrex::Vector<amrex::MultiFab*, std::allocator<amrex::MultiFab*> > const&, amrex::Vector<double, std::allocator<double> > const&, amrex::Vector<amrex::MultiFab*, std::allocator<amrex::MultiFab*> > const&, amrex::Vector<double, std::allocator<double> > const&, int, int, int, amrex::Geometry const&, amrex::Geometry const&, amrex::PhysBCFunct<amrex::GpuBndryFuncFab<amrex::FabFillNoOp> >&, int, amrex::PhysBCFunct<amrex::GpuBndryFuncFab<amrex::FabFillNoOp> >&, int, amrex::IntVectND<2> const&, amrex::InterpBase*, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int, amrex::NullInterpHook<amrex::FArrayBox> const&, amrex::NullInterpHook<amrex::FArrayBox> const&, amrex::EB2::IndexSpace const*, bool) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/AmrCore/AMReX_FillPatchUtil_I.H:603
20: cartMG.ex(+0x1be2c0) [0x64f7a97db2c0]
std::enable_if<amrex::IsFabArray<amrex::MultiFab, void>::value, void>::type amrex::FillPatchTwoLevels<amrex::MultiFab, amrex::PhysBCFunct<amrex::GpuBndryFuncFab<amrex::FabFillNoOp> >, amrex::InterpBase, amrex::NullInterpHook<amrex::FArrayBox>, amrex::NullInterpHook<amrex::FArrayBox> >(amrex::MultiFab&, amrex::IntVectND<2> const&, double, amrex::Vector<amrex::MultiFab*, std::allocator<amrex::MultiFab*> > const&, amrex::Vector<double, std::allocator<double> > const&, amrex::Vector<amrex::MultiFab*, std::allocator<amrex::MultiFab*> > const&, amrex::Vector<double, std::allocator<double> > const&, int, int, int, amrex::Geometry const&, amrex::Geometry const&, amrex::PhysBCFunct<amrex::GpuBndryFuncFab<amrex::FabFillNoOp> >&, int, amrex::PhysBCFunct<amrex::GpuBndryFuncFab<amrex::FabFillNoOp> >&, int, amrex::IntVectND<2> const&, amrex::InterpBase*, amrex::Vector<amrex::BCRec, std::allocator<amrex::BCRec> > const&, int, amrex::NullInterpHook<amrex::FArrayBox> const&, amrex::NullInterpHook<amrex::FArrayBox> const&) at /home/ankith/Documents/Projects/cartMG/Submodules/amrex/Src/AmrCore/AMReX_FillPatchUtil_I.H:814
21: cartMG.ex(+0x1fa6c9) [0x64f7a98176c9]
cartMG::KEpsilon::fillPatchEpsilon(int) at /home/ankith/Documents/Projects/cartMG/Src/TurbulenceModel/KEpsilon/KEpsilon_BC.cpp:316 (discriminator 8)
22: cartMG.ex(+0x1fd06a) [0x64f7a981a06a]
cartMG::KEpsilon::interpolate_fields(int, int) at /home/ankith/Documents/Projects/cartMG/Src/TurbulenceModel/KEpsilon/KEpsilon_BC.cpp:472 (discriminator 3)
23: cartMG.ex(+0x3afb18) [0x64f7a99ccb18]
cartMG::CFDSim::MGFcycleCoarseInitialize() at /home/ankith/Documents/Projects/cartMG/Src/cartMG_Multigrid.cpp:488 (discriminator 2)
24: cartMG.ex(+0x2adba4) [0x64f7a98caba4]
cartMG::CFDSim::SolveMultigrid() at /home/ankith/Documents/Projects/cartMG/Src/cartMG.cpp:585
25: cartMG.ex(+0x2ad752) [0x64f7a98ca752]
cartMG::CFDSim::Solve() at /home/ankith/Documents/Projects/cartMG/Src/cartMG.cpp:550
26: cartMG.ex(+0x42f4ad) [0x64f7a9a4c4ad]
main at /home/ankith/Documents/Projects/cartMG/Src/main.cpp:27
27: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x73c0ef229d90]
28: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x73c0ef229e40]
29: cartMG.ex(+0x3f7e5) [0x64f7a965c7e5]
?? ??:0
Changing the number of ghost cells filled by FillPatchTwoLevels to 3 seems to fix this issue. I can share my code privately to help in reproducing and diagnosing the error.