Curl2 matrix BC masks

[JustinRayAngus]

Add masks to implement the boundary conditions in the curl curl matrix used for the PETSc preconditioner.

• Add comments explaining the masking logic
• Add CI test:
  • https://github.com/BLAST-WarpX/warpx/blob/development/.azure-pipelines.yml add https://packages.ubuntu.com/search?keywords=libpetsc&searchon=names&suite=questing&section=all & flags

[JustinRayAngus]

@EZoni I've added a new 2D CI tests that uses PETSc, but it doesn't work because the PETSc flag is not defined when building for the CI tests. What should we do about this?

[JustinRayAngus]

[ ] https://github.com/BLAST-WarpX/warpx/blob/development/.azure-pipelines.yml add https://packages.ubuntu.com/search?keywords=libpetsc&searchon=names&suite=questing&section=all & flags

@ax3l It's not clear to me what I'm supposed to do here. Can you elaborate?

Here is how I would normally build with CMake when using PETSc:
cmake -S . -B build_2D_petsc -DWarpX_DIMS=2 -DAMReX_PETSC=YES -DPETSC_DIR=${PETSC_DIR} -DPETSC_ARCH=${PETSC_ARCH}

[EZoni]

@JustinRayAngus

Some info on the CI configuration topic.

This is the step where we configure and compile WarpX before running the CI tests in the Azure pipelines:

warpx/.azure-pipelines.yml

Lines 142 to 158 in 83cfdd0

	- bash: |
	# set options
	set -o nounset errexit pipefail
	# display disk space usage
	df -h
	# configure
	export AMReX_CMAKE_FLAGS="-DAMReX_ASSERTIONS=ON -DAMReX_TESTING=ON"
	export WARPX_TEST_FLAGS="-DWarpX_TEST_CLEANUP=ON -DWarpX_TEST_FPETRAP=ON -DWarpX_BACKTRACE_INFO=ON"
	cmake -S . -B build \
	${AMReX_CMAKE_FLAGS} \
	${WARPX_CMAKE_FLAGS} \
	${WARPX_TEST_FLAGS}
	# build
	cmake --build build -j 2
	# display disk space usage
	df -h
	displayName: 'Build'

You see that the configuration relies on the environment variables AMReX_CMAKE_FLAGS, WARPX_CMAKE_FLAGS, and WARPX_TEST_FLAGS.

AMReX_CMAKE_FLAGS and WARPX_TEST_FLAGS are set right above the cmake command, while WARPX_CMAKE_FLAGS is set here, depending on which pipeline we are running:

warpx/.azure-pipelines.yml

Lines 39 to 53 in 83cfdd0

	strategy:
	matrix:
	# 1D Cartesian, Cylindrical, Spherical
	1d:
	WARPX_CMAKE_FLAGS: -DWarpX_DIMS='1;RCYLINDER;RSPHERE' -DWarpX_FFT=ON -DWarpX_PYTHON=ON
	# Cartesian 2D
	cartesian_2d:
	WARPX_CMAKE_FLAGS: -DWarpX_DIMS=2 -DWarpX_FFT=ON -DWarpX_PYTHON=ON
	# Cartesian 3D
	cartesian_3d:
	WARPX_CMAKE_FLAGS: -DWarpX_DIMS=3 -DWarpX_FFT=ON -DWarpX_PYTHON=ON
	# Cylindrical RZ
	cylindrical_rz:
	WARPX_CMAKE_FLAGS: -DWarpX_DIMS=RZ -DWarpX_FFT=ON -DWarpX_PYTHON=ON
	WARPX_RZ_FFT: 'TRUE'

In your case it looks like the extra build options are

• -DAMReX_PETSC=YES
• -DPETSC_DIR=${PETSC_DIR}
• -DPETSC_ARCH=${PETSC_ARCH}

I would maybe add them to AMReX_CMAKE_FLAGS, since at least the first one is indeed an AMReX flag, so here:

warpx/.azure-pipelines.yml

Lines 147 to 149 in 83cfdd0

	# configure
	export AMReX_CMAKE_FLAGS="-DAMReX_ASSERTIONS=ON -DAMReX_TESTING=ON"
	export WARPX_TEST_FLAGS="-DWarpX_TEST_CLEANUP=ON -DWarpX_TEST_FPETRAP=ON -DWarpX_BACKTRACE_INFO=ON"

The outstanding question for me is on the environment variables PETSC_DIR and PETSC_ARCH (so the right hand side of the build options -DPETSC_DIR and -DPETSC_ARCH). I'm not sure how those are supposed to be set. How do you set them locally?

@ax3l Please feel free to chime in and comment/correct my suggestion.

[JustinRayAngus]

@EZoni Thanks for the explanation. I agree that the AMReX_PETSC flag should go in the AMReX_CMAKE_FLAGS. The other flags would be set depending on where PETSc is configured and compiled. It is still not clear to me how to accomplish that. Perhaps @ax3l @WeiqunZhang @debog can answer that.

[WeiqunZhang]

@JustinRayAngus Could you add this to your PR? It's unrelated to this PR. But it's just a one-line change that is needed for PETSc in debug mode.

--- a/Source/NonlinearSolvers/WarpX_PETSc.cpp
+++ b/Source/NonlinearSolvers/WarpX_PETSc.cpp
@@ -341,6 +341,7 @@ void PETScSolver_impl::applyPC( VecType& a_F,
 //! Assemble preconditioner matrix
 int PETScSolver_impl::assemblePCMatrix(LinOpType* const a_linop)
 {
+    PetscFunctionBeginUser;
     BL_PROFILE("PETScSolver_impl::assemblePCMatrix()");
 
     AMREX_ALWAYS_ASSERT(m_is_defined);

[JustinRayAngus]

PetscFunctionBeginUser;

Sure. I believe there is also a PetscFunctionEndUser function. Should I add that too?

[WeiqunZhang]

PetscFunctionBeginUser;

Sure. I believe there is also a PetscFunctionEndUser function. Should I add that too?

No. The issue is there is a PetscFunctionEndUser already without Begin.