Domain decomposition artifact shows up with `EffectivePotential` ES solver

[roelof-groenewald]

Description

We're seeing artifacts of the domain decomposition when using the semi-implicit Poisson solver.
See

• https://github.com/BLAST-WarpX/warpx/blob/development/Source/FieldSolver/ElectrostaticSolvers/EffectivePotentialES.cpp
• https://github.com/BLAST-WarpX/warpx/blob/development/Source/ablastr/fields/EffectivePotentialPoissonSolver.H

The image below shows charge density for two simulations where the domain consisted of just one tile versus an identical simulation where the domain was decomposed into 4 tiles:

The same issue does not come occur with the explicit ES solver.

The above case uses an embedded boundary (in a 3d simulation), so both the explicit solver and the semi-implicit solver uses amrex::MLEBNodeFDLaplacian. But only the semi-implicit case uses a spatially varying sigma (see

warpx/Source/FieldSolver/ElectrostaticSolvers/EffectivePotentialES.cpp

Line 93 in 480c53d

void EffectivePotentialES::ComputeSigma (MultiFab& sigma) const

).

It would appear that somewhere we are missing a call to fill the guard cells. @WeiqunZhang do you have any idea where this problem might be coming from?

[WeiqunZhang]

If you provide me a test with c++ inputs file, I can take a look.