Cryptic error message when value_function parameter in Histogram2D not set

[n01r]

I recently encountered a cryptic error message when working with the ParticleHistogram2D reduced diagnostics.
I forgot to set the parameter value_function(t,x,y,z,ux,uy,uz,w). It would be good if we had something that told the user more clearly what to do.

How to reproduce:
Just run the example at Examples/Physical_applications/laser_ion but comment out PhaseSpaceElectrons.value_function(t,x,y,z,ux,uy,uz,w) = "w" from inputs_2d before running it.

Tested on one node on Crusher (OLCF)

Error output

Memory access fault by GPU node-9 (Agent handle: 0x2ac0aa0) on address (nil). Reason: Unknown.
SIGABRT
Memory access fault by GPU node-6 (Agent handle: 0x2ac0aa0) on address (nil). Reason: Unknown.
SIGABRT
Memory access fault by GPU node-7 (Agent handle: 0x2ac0aa0) on address (nil). Reason: Unknown.
SIGABRT
Memory access fault by GPU node-8 (Agent handle: 0x2ac0aa0) on address (nil). Reason: Unknown.
SIGABRT
See Backtrace.0 file for details
See Backtrace.2 file for details
See Backtrace.1 file for details
See Backtrace.3 file for details
MPICH ERROR [Rank 3] [job id 424102.0] [Mon Dec 18 21:15:34 2023] [crusher020] - Abort(6) (rank 3 in comm 496): application called MPI_Abort(comm=0x84000001, 6) - process 3

Segfault
MPICH ERROR [Rank 0] [job id 424102.0] [Mon Dec 18 21:15:35 2023] [crusher020] - Abort(6) (rank 0 in comm 496): application called MPI_Abort(comm=0x84000002, 6) - process 0

Segfault
MPICH ERROR [Rank 2] [job id 424102.0] [Mon Dec 18 21:15:35 2023] [crusher020] - Abort(6) (rank 2 in comm 496): application called MPI_Abort(comm=0x84000001, 6) - process 2

Segfault
MPICH ERROR [Rank 1] [job id 424102.0] [Mon Dec 18 21:15:35 2023] [crusher020] - Abort(6) (rank 1 in comm 496): application called MPI_Abort(comm=0x84000001, 6) - process 1

Segfault
See Backtrace.3 file for details
See Backtrace.0 file for details
See Backtrace.2 file for details
See Backtrace.1 file for details
MPICH ERROR [Rank 3] [job id 424102.0] [Mon Dec 18 21:15:38 2023] [crusher020] - Abort(11) (rank 3 in comm 496): application called MPI_Abort(comm=0x84000001, 11) - process 3

MPICH ERROR [Rank 0] [job id 424102.0] [Mon Dec 18 21:15:38 2023] [crusher020] - Abort(11) (rank 0 in comm 496): application called MPI_Abort(comm=0x84000002, 11) - process 0

MPICH ERROR [Rank 2] [job id 424102.0] [Mon Dec 18 21:15:38 2023] [crusher020] - Abort(11) (rank 2 in comm 496): application called MPI_Abort(comm=0x84000001, 11) - process 2

MPICH ERROR [Rank 1] [job id 424102.0] [Mon Dec 18 21:15:38 2023] [crusher020] - Abort(11) (rank 1 in comm 496): application called MPI_Abort(comm=0x84000001, 11) - process 1

srun: error: crusher020: task 0: Segmentation fault
srun: Terminating StepId=424102.0
slurmstepd: error: *** STEP 424102.0 ON crusher020 CANCELLED AT 2023-12-18T21:15:38 ***
srun: error: crusher020: tasks 1-2: Segmentation fault
srun: error: crusher020: tasks 4-7: Terminated
srun: error: crusher020: task 3: Segmentation fault (core dumped)
srun: Force Terminated StepId=424102.0

[ax3l]

Hi @n01r, what was printed on stdout?

[n01r]

Hi @ax3l, this is how far the stdout got:

Initializing AMReX (23.12)...
MPI initialized with 8 MPI processes
MPI initialized with thread support level 3
Initializing HIP...
HIP initialized with 8 devices.
AMReX (23.12) initialized
PICSAR (23.09)
WarpX (23.11-63-gb9b0748f1d00)

    __        __             __  __
    \ \      / /_ _ _ __ _ __\ \/ /
     \ \ /\ / / _` | '__| '_ \\  /
      \ V  V / (_| | |  | |_) /  \
       \_/\_/ \__,_|_|  | .__/_/\_\
                        |_|

Level 0: dt = 1.530214125e-17 ; dx = 5.580357143e-09 ; dz = 8.081896552e-09

Grids Summary:
  Level 0   8 grids  9977856 cells  100 % of domain
            smallest grid: 1344 x 928  biggest grid: 1344 x 928

-------------------------------------------------------------------------------
--------------------------- MAIN EM PIC PARAMETERS ----------------------------
-------------------------------------------------------------------------------
Precision:            | DOUBLE
Particle precision:   | DOUBLE
Geometry:             | 2D (XZ)
Operation mode:       | Electromagnetic
                      | - vacuum
-------------------------------------------------------------------------------
Current Deposition:   | Esirkepov
Particle Pusher:      | Boris
Charge Deposition:    | standard
Field Gathering:      | energy-conserving
Particle Shape Factor:| 3
-------------------------------------------------------------------------------
Maxwell Solver:       | Yee
                      | - staggered grid
Guard cells           | - ng_alloc_EB = (4,4)
 (allocated for E/B)  |
-------------------------------------------------------------------------------
For full input parameters, see the file: warpx_used_inputs

--- INFO    : Writing plotfile diags/diag1000000
--- INFO    : Writing openPMD file diags/openPMDfw000000
--- INFO    : Writing openPMD file diags/openPMDbw000000

And here the submit script:

crusher_2D3V.sbatch

#!/usr/bin/env bash

#SBATCH -A aph114
#    note: WarpX ECP members use aph114
#SBATCH -J WarpX
#SBATCH -o %x-%j.out
#SBATCH -t 01:00:00
#SBATCH -p batch
#SBATCH --ntasks-per-node=8
# Since 2022-12-29 Crusher is using a low-noise mode layout,
# making only 7 instead of 8 cores available per process
# https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html#id6
#SBATCH --cpus-per-task=7
#SBATCH --gpus-per-task=1
#SBATCH --gpu-bind=closest
#SBATCH -N 1

# From the documentation:
# Each Crusher compute node consists of [1x] 64-core AMD EPYC 7A53
# "Optimized 3rd Gen EPYC" CPU (with 2 hardware threads per physical core) with
# access to 512 GB of DDR4 memory.
# Each node also contains [4x] AMD MI250X, each with 2 Graphics Compute Dies
# (GCDs) for a total of 8 GCDs per node. The programmer can think of the 8 GCDs
# as 8 separate GPUs, each having 64 GB of high-bandwidth memory (HBM2E).

# note (5-16-22, OLCFHELP-6888)
# this environment setting is currently needed on Crusher to work-around a
# known issue with Libfabric
#export FI_MR_CACHE_MAX_COUNT=0  # libfabric disable caching
# or, less invasive:
export FI_MR_CACHE_MONITOR=memhooks  # alternative cache monitor

# Seen since August 2023 on Frontier, adapting the same for Crusher
# OLCFDEV-1597: OFI Poll Failed UNDELIVERABLE Errors
# https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#olcfdev-1597-ofi-poll-failed-undeliverable-errors
export MPICH_SMP_SINGLE_COPY_MODE=NONE
export FI_CXI_RX_MATCH_MODE=software

# note (9-2-22, OLCFDEV-1079)
# this environment setting is needed to avoid that rocFFT writes a cache in
# the home directory, which does not scale.
export ROCFFT_RTC_CACHE_PATH=/dev/null

export OMP_NUM_THREADS=1
export WARPX_NMPI_PER_NODE=8
export TOTAL_NMPI=$(( ${SLURM_JOB_NUM_NODES} * ${WARPX_NMPI_PER_NODE} ))
srun -N${SLURM_JOB_NUM_NODES} -n${TOTAL_NMPI} --ntasks-per-node=${WARPX_NMPI_PER_NODE} \
    ./warpx.2d inputs_2d > output_${SLURM_JOBID}.txt

[pordyna]

@ax3l I just run today into the exact same problem. It was somewhat hard to debug, would you consider making this a required option until this has been solved? That way, other users won't run into this error again.

Btw. this is the stderr that I get on perlmutter:

amrex::Abort::6::CUDA error 700 in file /global/homes/p/pordyna/src/warpx/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 614: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::1::CUDA error 700 in file /global/homes/p/pordyna/src/warpx/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 614: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::3::CUDA error 700 in file /global/homes/p/pordyna/src/warpx/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 614: an illegal memory access was encountered !!!
SIGABRT