Skip to content

Commit 17053af

Browse files
ambradambrad
authored
Merge pull request #50 from E3SM-Project/develop
Develop
2 parents 0deb504 + c008c29 commit 17053af

File tree

5 files changed

+159
-18
lines changed

5 files changed

+159
-18
lines changed

‎methods/islet/Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ pum_sweep: libislet pum_sweep.o
2828
cslunstab: cslunstab.o
2929
$(CXX) cslunstab.o $(LDFLAGS) $(LINK_LAPACK_BLAS) -fopenmp -o cslunstab
3030

31+
pum_perturb_plot: libislet pum_perturb_plot.o
32+
$(CXX) pum_perturb_plot.o $(LDFLAGS) $(LINK_LAPACK_BLAS) -L. -lislet -fopenmp -o pum_perturb_plot
33+
3134
clean:
3235
rm -f *.o *.so search np4 pum_sweep pum_perturb_plot run_meam1_sweep
3336

‎methods/islet/figures/figs.tex‎

Lines changed: 80 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -129,24 +129,18 @@
129129

130130
% hy figs-methods.hy write-slmmir-script
131131
% bash run-slmmir-on-basis-lines.sh > slmmir-on-basis-lines-2.txt
132-
% hy figs-methods.hy plot-slmmir-vs-heuristic # uses slmmir-on-basis-lines-2.txt
133-
\begin{figure}[tbh]
132+
% hy figs-methods.hy plot-slmmir-vs-heuristic-ab # uses slmmir-on-basis-lines-2.txt
133+
\begin{figure}[tb]
134134
\centering
135-
\includegraphics[width=0.48\linewidth]{slmmir-vs-heuristic-gau-nopp-l2}
136-
\caption{$l_2$ norm on the nondivergent flow problem
135+
\includegraphics[width=1\linewidth]{slmmir-vs-heuristic-ab}
136+
\caption{$l_2$ errors for the nondivergent flow problem
137137
using basis $\basisns_{\np}$ vs.~$a_2(\basisns_{\np})$,
138138
for a large number of \abtps~bases and $\np=6$ to $10$.
139-
The legend lists the marker type for each $\np$.
140-
Large red circles outline the bases in Table \ref{tbl:gll}.
141-
The configuration uses the Gaussian hills IC and no property preservation.}
142-
\label{fig:slmmir-vs-heuristic-a}
143-
\end{figure}
144-
\begin{figure}[tbh]
145-
\centering
146-
\includegraphics[width=0.48\linewidth]{slmmir-vs-heuristic-cos-pp-l2}
147-
\caption{Same as Fig.~\ref{fig:slmmir-vs-heuristic-a} except that the configuration
148-
uses the cosine bells IC with property preservation.}
149-
\label{fig:slmmir-vs-heuristic-b}
139+
The legends list the marker type for each $\np$.
140+
Large red circles outline the points corresponding to the bases in Table \ref{tbl:gll}.
141+
(a) With the Gaussian hills IC and no property preservation.
142+
(b) With the cosine bells IC and property preservation.}
143+
\label{fig:slmmir-vs-heuristic}
150144
\end{figure}
151145

152146
% hy figs-adv-diag.hy figs-acc acc-0.txt
@@ -374,3 +368,74 @@
374368
}
375369
\label{fig:summit-perf}
376370
\end{figure}
371+
372+
% hy figs-methods.hy cubed-sphere-subelem-grid-schematic
373+
\begin{figure}[tb]
374+
\centering
375+
\includegraphics[width=0.5\linewidth]{cubed-sphere-subelem-grid-schematic}
376+
\caption{
377+
Cubed-sphere grid (black lines in foreground, gray lines in background)
378+
with $\neface\!\times\!\neface$ spectral elements per cube face;
379+
$\neface=2$ in this example.
380+
The green dashed line outlines the sphere's projection onto the two-dimensional plane of the figure.
381+
The upper-right element of the front cubed-sphere face shows the subelement tensor-product Gauss--Lobatto--Legendre (GLL) grid points.
382+
In this example, the dynamics solver's subelement grid uses $\npv=4$ (large blue circles) GLL points per dimension,
383+
and the transport solver's subelement grid uses $\npt=6$ (small red circles).
384+
}
385+
\label{fig:cubed-sphere-subelem-grid-schematic}
386+
\end{figure}
387+
388+
% hy figs-methods.hy isl-1d-schematic
389+
\begin{figure}[tb]
390+
\centering
391+
\includegraphics[width=1\linewidth]{isl-1d-schematic}
392+
\caption{
393+
Illustration of the classical and element-based interpolation semi-Lagrangian methods.
394+
See the discussion in Sect.~\ref{sec:setting:sl}.
395+
}
396+
\label{fig:isl-1d-schematic}
397+
\end{figure}
398+
399+
% hy figs-methods.hy matrix-schematic
400+
\begin{figure}[tb]
401+
\centering
402+
\includegraphics[width=0.5\linewidth]{matrix-schematic}
403+
\caption{
404+
Correspondence between the ISL space--time operator $\mat{A}$ (top)
405+
and the target and source one-dimensional grids (bottom)
406+
for one time step of the test problem.
407+
In the matrix $\mat{A}$, of which the upper-left corner is pictured,
408+
numbered columns correspond to source-grid degrees of freedom (DOF),
409+
and numbered rows correspond to target-grid DOF.
410+
Nonzeros occur in the red rectangular blocks $\mat{B} \equiv (\mat{\bar{B}} \ \vec{b})$.
411+
In this example, the target grid (green dashed line) is advected backward in time
412+
so that one subelement grid point moves one element to the left;
413+
in the resulting matrix $\mat{A}$, the blocks are shifted one row down.
414+
}
415+
\label{fig:matrix-schematic}
416+
\end{figure}
417+
418+
% bash run-instab-imgs.sh
419+
% hy figs-adv-diag.hy fig-instab
420+
\begin{figure}[tb]
421+
\centering
422+
\includegraphics[width=0.75\linewidth]{img-instab}
423+
\caption{
424+
Images for unstable (top) and stable (bottom) ISL transport.
425+
The problem is nondivergent flow with the slotted cylinders IC,
426+
with $\neface = 20$, $\npv = \npt = 6$, and the large time step.
427+
The snapshot is at the end of day 11 of the first cycle;
428+
the images are zoomed to just the region containing the slotted cylinders.
429+
The color range is [-0.05, 1.15],
430+
which clips the top-left image's range of [-30.4, 32.7].
431+
The bases are GLL (top) and Islet (bottom),
432+
without (left) and with (right) property preservation.
433+
The GLL basis yields an unstable ISL method.
434+
Although a nonlinear property preservation step makes the method stable,
435+
the linear advection operator's instability still manifests as spurious oscillations.
436+
The Islet basis yields a stabilized ISL method;
437+
the nonlinear property preservation step now just controls mass conservation and extrema,
438+
as intended.
439+
}
440+
\label{fig:islet-vs-gll-img}
441+
\end{figure}

‎methods/islet/figures/run-instab-imgs.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
exe=../../slmm/slmmir
2+
3+
ne=20
4+
np=6
5+
nstep=$(($ne * 6))
6+
we=110
7+
8+
glbcdrs=(none caas-node none caas-node)
9+
bases=(Gll Gll GllNodal GllNodal)
10+
ncycles=(1 1 1 1)
11+
12+
for i in $(seq 0 3); do
13+
ncycle=${ncycles[$i]}
14+
glbcdr=${glbcdrs[$i]}
15+
basis=${bases[$i]}
16+
echo $i $ncycle $glbcdr $basis
17+
name=instab-nondiv-ne${ne}np${np}-${glbcdr}-cyc${ncycle}-$basis
18+
time=$(($ncycle * 12))
19+
cmd="OMP_PROC_BIND=false OMP_NUM_THREADS=8 $exe -method pcsl -ode nondivergent -ic gaussianhills -ic cosinebells -ic slottedcylinders -T $time -nsteps $nstep -timeint exact -ne $ne -np $np -dmc eh -d2c -mono $glbcdr -lim caas -we $we -io-type internal -io-start-cycle $ncycle -res 256 -o $name -rit -prefine 0 -basis $basis -rhot0"
20+
echo "cmd> $cmd"
21+
eval "$cmd"
22+
done

‎methods/islet/pum_perturb_plot.cpp‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#include "islet_isl.hpp"
2+
#include "islet_pum.hpp"
3+
4+
#include <cstdio>
5+
6+
static void write_array (const std::string& name, const std::vector<Real>& a) {
7+
printf("%s [", name.c_str());
8+
for (const auto& e : a) printf(" %1.5e", e);
9+
printf("]\n");
10+
}
11+
12+
static void gen_plot_data (const islet::Operator::ConstPtr& op, const Int np,
13+
const bool print_perturb = true) {
14+
static const Int nperturb = 48;
15+
std::vector<Real> perturb(nperturb), meam1(nperturb);
16+
for (Int i = 0; i < nperturb; ++i) perturb[i] = 0.1/std::pow(1.18, i);
17+
if (print_perturb) write_array(">> perturb", perturb);
18+
const auto oim = std::make_shared<islet::OperatorInterpMethod>(np, op);
19+
const auto im = std::make_shared<InterpMethod>(oim);
20+
# pragma omp parallel for
21+
for (Int i = 0; i < nperturb; ++i) {
22+
meam1[i] = 0;
23+
pum::Options o;
24+
o.threaded = false;
25+
o.ntrial = 33;
26+
o.perturb = perturb[i];
27+
for (const Int ne : {4, 7, 15})
28+
for (const Int mec_ne : {33}) {
29+
o.ne = ne;
30+
o.mec_ne = mec_ne;
31+
pum::PerturbedUniformMeshMetric pum(im, o);
32+
meam1[i] = std::max(meam1[i], pum.run());
33+
}
34+
}
35+
write_array(">> meam1", meam1);
36+
}
37+
38+
int main (int argc, char** argv) {
39+
const auto gll_best = islet::Operator::create(islet::Operator::gll_best);
40+
const auto uofs = islet::Operator::create(islet::Operator::uniform_offset_nodal_subset);
41+
bool first = true;
42+
for (int np = 2; np <= islet::np_max; ++np) {
43+
if (np >= 4) {
44+
printf(">>> gll_best %d\n", np);
45+
gen_plot_data(gll_best, np, false);
46+
}
47+
printf(">>> uniform_offset_nodal_subset %d\n", np);
48+
gen_plot_data(uofs, np, first);
49+
first = false;
50+
}
51+
}

‎methods/islet/readme.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,15 +53,15 @@ installation, then
5353
ln -s make.inc.gnu make.inc
5454
make -j16
5555
Optionally run regression tests:
56-
OMP_NUM_THREADS=16 KMP_AFFINITY=balanced python2 slmm_runtests.py
56+
OMP_NUM_THREADS=16 KMP_AFFINITY=balanced python slmm_runtests.py
5757
Bash scripts in the methods/islet/figures directory call the slmmir program.
5858

5959
We use the language hy to create the figures. hy is a Lisp that compiles to
6060
Python AST. We used hy 0.18.0 ('pip install hy' for the latest version) with
6161
CPython 3.7.6 provided by Anaconda 3.
6262

63-
The code used to obtain performance data on Summit will be part of main E3SM
64-
soon. The exact version used to generate the data is archived here:
63+
The code used to obtain performance data on Summit is part of the main E3SM
64+
repo. The exact version used to generate the data is archived here:
6565
https://github.com/ambrad/E3SM/releases/tag/islet-2d-paper-summit-sl-gpu-timings
6666
The data are here:
6767
https://github.com/E3SM-Project/perf-data/tree/main/nhxx-sl-summit-mar2021

0 commit comments

Comments
 (0)