Skip to content

Commit 75e2fd9

Browse files
lucafedeli88RemiLehe
lucafedeli88
and
RemiLehe
authored
Add thread sanitizer test in CI (#4799)
* add thread sanitizer * move sanitizers in new yml file * fix bug * update script to try to use thread sanitizers in CI * update script to run race condition sanitizer * fix libarcher option * use docker image for newer clang in thread sanitizer test * fix bug * add new dependencies file in github workflows * add sudo installation * add sudo installation * add wget to the dependencies * add xz utils to dependencies * downgrade ubuntu version to 23.12 * correct mistake 23.12 -> 23.10 * remove wget and xz from installation list and add curl * add back wget to installation list * add back xz-utils to installation list * add git to installation list * use clang-17 * change numbers of retries * revert number of retries * use clang 17 * properly set clang 17 as the compiler to be used * add ccache to installation list * add pkg-config to install list * allow running mpi with root privilegies * fix race condition * fix bug * disable serialization of initial conditions * fix race condition * fix race conditions * fixed bug * Update .github/workflows/clang_sanitizers.yml * update AMReX * implement WeiqunZhang's suggestion to fix race conditions * increase required precision for self fields in clang sanitizers CI test * add comment line * update comment * further increase tolerance * further increase tolerance * Update .github/workflows/clang_sanitizers.yml * Apply suggestions from code review * revert change of warpx.self_fields_required_precision * add more sanitizer checks to UB sanitizer test * disable leak check * set current_correction to true when it is first declared * use double precision for thread sanitizer test * add instruction to debug CI failure * remove comment * try to debug fail in sanitizer test * fix bug in yml * add instructions to download output files in order to debug a segfault * debug issues with thread sanitizer * remove code added for debugging, compile separately for Embedded Boundary --------- Co-authored-by: Remi Lehe <[email protected]>
1 parent f044929 commit 75e2fd9

File tree

4 files changed

+251
-58
lines changed

4 files changed

+251
-58
lines changed

‎.github/workflows/clang_sanitizers.yml‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
name: 🧴 clang sanitizers
2+
3+
on: [push, pull_request]
4+
5+
concurrency:
6+
group: ${{ github.ref }}-${{ github.head_ref }}-clangsanitizers
7+
cancel-in-progress: true
8+
9+
jobs:
10+
build_UB_sanitizer:
11+
name: Clang UB sanitizer
12+
runs-on: ubuntu-22.04
13+
if: github.event.pull_request.draft == false
14+
env:
15+
CC: clang
16+
CXX: clang++
17+
# On CI for this test, Ninja is slower than the default:
18+
#CMAKE_GENERATOR: Ninja
19+
steps:
20+
- uses: actions/checkout@v4
21+
- name: install dependencies
22+
run: |
23+
.github/workflows/dependencies/clang15.sh
24+
- name: CCache Cache
25+
uses: actions/cache@v4
26+
with:
27+
path: ~/.cache/ccache
28+
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
29+
restore-keys: |
30+
ccache-${{ github.workflow }}-${{ github.job }}-git-
31+
- name: build WarpX
32+
run: |
33+
export CCACHE_COMPRESS=1
34+
export CCACHE_COMPRESSLEVEL=10
35+
export CCACHE_MAXSIZE=100M
36+
ccache -z
37+
38+
export CXX=$(which clang++-15)
39+
export CC=$(which clang-15)
40+
export CXXFLAGS="-fsanitize=undefined,address,pointer-compare -fno-sanitize-recover=all"
41+
42+
cmake -S . -B build \
43+
-GNinja \
44+
-DCMAKE_VERBOSE_MAKEFILE=ON \
45+
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
46+
-DWarpX_DIMS="RZ;1;2;3" \
47+
-DWarpX_FFT=ON \
48+
-DWarpX_QED=ON \
49+
-DWarpX_QED_TABLE_GEN=ON \
50+
-DWarpX_OPENPMD=ON \
51+
-DWarpX_PRECISION=SINGLE \
52+
-DWarpX_PARTICLE_PRECISION=SINGLE
53+
cmake --build build -j 4
54+
55+
ccache -s
56+
du -hs ~/.cache/ccache
57+
58+
- name: run with UB sanitizer
59+
run: |
60+
61+
export OMP_NUM_THREADS=2
62+
63+
#MPI implementations often leak memory
64+
export "ASAN_OPTIONS=detect_leaks=0"
65+
66+
mpirun -n 2 ./build/bin/warpx.rz Examples/Physics_applications/laser_acceleration/inputs_rz
67+
mpirun -n 2 ./build/bin/warpx.1d Examples/Physics_applications/laser_acceleration/inputs_1d
68+
mpirun -n 2 ./build/bin/warpx.2d Examples/Physics_applications/laser_acceleration/inputs_2d
69+
mpirun -n 2 ./build/bin/warpx.3d Examples/Physics_applications/laser_acceleration/inputs_3d
70+
71+
build_thread_sanitizer:
72+
name: Clang thread sanitizer
73+
runs-on: ubuntu-22.04
74+
container: ubuntu:23.10
75+
if: github.event.pull_request.draft == false
76+
env:
77+
CC: clang
78+
CXX: clang++
79+
# On CI for this test, Ninja is slower than the default:
80+
#CMAKE_GENERATOR: Ninja
81+
steps:
82+
- uses: actions/checkout@v4
83+
- name: install dependencies
84+
run: |
85+
.github/workflows/dependencies/clang17.sh
86+
- name: CCache Cache
87+
uses: actions/cache@v4
88+
with:
89+
path: ~/.cache/ccache
90+
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
91+
restore-keys: |
92+
ccache-${{ github.workflow }}-${{ github.job }}-git-
93+
- name: build WarpX
94+
run: |
95+
export CCACHE_COMPRESS=1
96+
export CCACHE_COMPRESSLEVEL=10
97+
export CCACHE_MAXSIZE=100M
98+
ccache -z
99+
100+
export CXX=$(which clang++-17)
101+
export CC=$(which clang-17)
102+
export CXXFLAGS="-fsanitize=thread"
103+
104+
cmake -S . -B build \
105+
-GNinja \
106+
-DCMAKE_VERBOSE_MAKEFILE=ON \
107+
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
108+
-DWarpX_DIMS="RZ;1;2;3" \
109+
-DWarpX_FFT=ON \
110+
-DWarpX_QED=ON \
111+
-DWarpX_QED_TABLE_GEN=ON \
112+
-DWarpX_OPENPMD=ON \
113+
-DWarpX_EB=OFF \
114+
-DWarpX_PRECISION=DOUBLE \
115+
-DWarpX_PARTICLE_PRECISION=DOUBLE
116+
cmake --build build -j 4
117+
118+
cmake -S . -B build_EB \
119+
-GNinja \
120+
-DCMAKE_VERBOSE_MAKEFILE=ON \
121+
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
122+
-DWarpX_DIMS="2" \
123+
-DWarpX_FFT=ON \
124+
-DWarpX_QED=ON \
125+
-DWarpX_QED_TABLE_GEN=ON \
126+
-DWarpX_OPENPMD=ON \
127+
-DWarpX_EB=ON \
128+
-DWarpX_PRECISION=DOUBLE \
129+
-DWarpX_PARTICLE_PRECISION=DOUBLE
130+
cmake --build build_EB -j 4
131+
132+
ccache -s
133+
du -hs ~/.cache/ccache
134+
135+
- name: run with thread sanitizer
136+
run: |
137+
export PMIX_MCA_gds=hash
138+
export TSAN_OPTIONS='ignore_noninstrumented_modules=1'
139+
export ARCHER_OPTIONS="verbose=1"
140+
141+
# We need these two lines because these tests run inside a docker container
142+
export OMPI_ALLOW_RUN_AS_ROOT=1
143+
export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
144+
145+
export OMP_NUM_THREADS=2
146+
147+
mpirun -n 2 ./build/bin/warpx.rz Examples/Physics_applications/laser_acceleration/inputs_rz warpx.serialize_initial_conditions = 0
148+
mpirun -n 2 ./build/bin/warpx.1d Examples/Physics_applications/laser_acceleration/inputs_1d warpx.serialize_initial_conditions = 0
149+
mpirun -n 2 ./build/bin/warpx.2d Examples/Physics_applications/laser_acceleration/inputs_2d warpx.serialize_initial_conditions = 0
150+
mpirun -n 2 ./build/bin/warpx.3d Examples/Physics_applications/laser_acceleration/inputs_3d warpx.serialize_initial_conditions = 0
151+
152+
git clone https://github.com/ECP-WarpX/warpx-data ../warpx-data
153+
cd Examples/Tests/embedded_circle
154+
155+
ulimit -c unlimited
156+
157+
mpirun -n 2 ../../../build_EB/bin/warpx.2d inputs_2d warpx.serialize_initial_conditions = 0
158+
159+
save_pr_number:
160+
if: github.event_name == 'pull_request'
161+
runs-on: ubuntu-latest
162+
steps:
163+
- name: Save PR number
164+
env:
165+
PR_NUMBER: ${{ github.event.number }}
166+
run: |
167+
echo $PR_NUMBER > pr_number.txt
168+
- uses: actions/upload-artifact@v4
169+
with:
170+
name: pr_number
171+
path: pr_number.txt
172+
retention-days: 1

‎.github/workflows/dependencies/clang17.sh‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Copyright 2024 The WarpX Community
4+
#
5+
# License: BSD-3-Clause-LBNL
6+
# Authors: Luca Fedeli
7+
8+
set -eu -o pipefail
9+
10+
# This dependency file is currently used within a docker container,
11+
# which does not come with sudo.
12+
apt-get -qqq update
13+
apt-get -y install sudo
14+
15+
# `man apt.conf`:
16+
# Number of retries to perform. If this is non-zero APT will retry
17+
# failed files the given number of times.
18+
echo 'Acquire::Retries "3";' | sudo tee /etc/apt/apt.conf.d/80-retries
19+
20+
# This dependency file is currently used within a docker container,
21+
# which does not come (among others) with wget, xz-utils, curl, git,
22+
# ccache, and pkg-config pre-installed.
23+
sudo apt-get -qqq update
24+
sudo apt-get install -y \
25+
cmake \
26+
clang-17 \
27+
clang-tidy-17 \
28+
libblas-dev \
29+
libc++-17-dev \
30+
libboost-math-dev \
31+
libfftw3-dev \
32+
libfftw3-mpi-dev \
33+
libhdf5-openmpi-dev \
34+
liblapack-dev \
35+
libopenmpi-dev \
36+
libomp-17-dev \
37+
ninja-build \
38+
wget \
39+
xz-utils \
40+
curl \
41+
git \
42+
ccache \
43+
pkg-config
44+
45+
# Use clang 17
46+
export CXX=$(which clang++-17)
47+
export CC=$(which clang-17)
48+
49+
# cmake-easyinstall
50+
#
51+
sudo curl -L -o /usr/local/bin/cmake-easyinstall https://raw.githubusercontent.com/ax3l/cmake-easyinstall/main/cmake-easyinstall
52+
sudo chmod a+x /usr/local/bin/cmake-easyinstall
53+
export CEI_SUDO="sudo"
54+
export CEI_TMP="/tmp/cei"
55+
56+
# BLAS++ & LAPACK++
57+
cmake-easyinstall \
58+
--prefix=/usr/local \
59+
git+https://github.com/icl-utk-edu/blaspp.git \
60+
-Duse_openmp=OFF \
61+
-Dbuild_tests=OFF \
62+
-DCMAKE_CXX_COMPILER_LAUNCHER=$(which ccache) \
63+
-DCMAKE_VERBOSE_MAKEFILE=ON
64+
65+
cmake-easyinstall \
66+
--prefix=/usr/local \
67+
git+https://github.com/icl-utk-edu/lapackpp.git \
68+
-Duse_cmake_find_lapack=ON \
69+
-Dbuild_tests=OFF \
70+
-DCMAKE_CXX_COMPILER_LAUNCHER=$(which ccache) \
71+
-DCMAKE_VERBOSE_MAKEFILE=ON

‎.github/workflows/ubuntu.yml‎

Lines changed: 0 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -224,61 +224,6 @@ jobs:
224224
export OMP_NUM_THREADS=1
225225
mpirun -n 2 Examples/Physics_applications/laser_acceleration/PICMI_inputs_3d.py
226226
227-
build_UB_sanitizer:
228-
name: Clang UB sanitizer
229-
runs-on: ubuntu-22.04
230-
if: github.event.pull_request.draft == false
231-
env:
232-
CC: clang
233-
CXX: clang++
234-
# On CI for this test, Ninja is slower than the default:
235-
#CMAKE_GENERATOR: Ninja
236-
steps:
237-
- uses: actions/checkout@v4
238-
- name: install dependencies
239-
run: |
240-
.github/workflows/dependencies/clang15.sh
241-
- name: CCache Cache
242-
uses: actions/cache@v4
243-
with:
244-
path: ~/.cache/ccache
245-
key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }}
246-
restore-keys: |
247-
ccache-${{ github.workflow }}-${{ github.job }}-git-
248-
- name: build WarpX
249-
run: |
250-
export CCACHE_COMPRESS=1
251-
export CCACHE_COMPRESSLEVEL=10
252-
export CCACHE_MAXSIZE=100M
253-
ccache -z
254-
255-
export CXX=$(which clang++-15)
256-
export CC=$(which clang-15)
257-
export CXXFLAGS="-fsanitize=undefined -fno-sanitize-recover=all"
258-
259-
cmake -S . -B build \
260-
-GNinja \
261-
-DCMAKE_VERBOSE_MAKEFILE=ON \
262-
-DWarpX_DIMS="RZ;1;2;3" \
263-
-DWarpX_FFT=ON \
264-
-DWarpX_QED=ON \
265-
-DWarpX_QED_TABLE_GEN=ON \
266-
-DWarpX_OPENPMD=ON \
267-
-DWarpX_PRECISION=SINGLE \
268-
-DWarpX_PARTICLE_PRECISION=SINGLE
269-
cmake --build build -j 4
270-
271-
ccache -s
272-
du -hs ~/.cache/ccache
273-
274-
- name: run with UB sanitizer
275-
run: |
276-
export OMP_NUM_THREADS=2
277-
mpirun -n 2 ./build/bin/warpx.rz Examples/Physics_applications/laser_acceleration/inputs_rz
278-
mpirun -n 2 ./build/bin/warpx.1d Examples/Physics_applications/laser_acceleration/inputs_1d
279-
mpirun -n 2 ./build/bin/warpx.2d Examples/Physics_applications/laser_acceleration/inputs_2d
280-
mpirun -n 2 ./build/bin/warpx.3d Examples/Physics_applications/laser_acceleration/inputs_3d
281-
282227
save_pr_number:
283228
if: github.event_name == 'pull_request'
284229
runs-on: ubuntu-latest

‎Source/Particles/ParticleBoundaryBuffer.cpp‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -401,17 +401,22 @@ void ParticleBoundaryBuffer::gatherParticlesFromDomainBoundaries (MultiParticleC
401401

402402
for (int lev = 0; lev < pc.numLevels(); ++lev)
403403
{
404+
for (PIter pti(pc, lev); pti.isValid(); ++pti) {
405+
species_buffer.DefineAndReturnParticleTile(
406+
lev, pti.index(), pti.LocalTileIndex());
407+
}
408+
404409
const auto& plevel = pc.GetParticles(lev);
405410
#ifdef AMREX_USE_OMP
406411
#pragma omp parallel if (amrex::Gpu::notInLaunchRegion())
407412
#endif
408413
for(PIter pti(pc, lev); pti.isValid(); ++pti)
409414
{
410415
auto index = std::make_pair(pti.index(), pti.LocalTileIndex());
411-
if(plevel.find(index) == plevel.end()) { continue; }
412416

413-
auto& ptile_buffer = species_buffer.DefineAndReturnParticleTile(
414-
lev, pti.index(), pti.LocalTileIndex());
417+
auto& ptile_buffer =
418+
species_buffer.ParticlesAt(lev, pti.index(), pti.LocalTileIndex());
419+
415420
const auto& ptile = plevel.at(index);
416421
auto np = ptile.numParticles();
417422
if (np == 0) { continue; }

0 commit comments

Comments
 (0)