Merge pull request #295 from mwarusz/omega/kokkos-wrappers

Add tests and developer docs for Kokkos wrappers

Author: sbrus89

components/omega/doc/devGuide/ParallelLoops.md

@@ -0,0 +1,76 @@
+(omega-dev-parallel-loops)=
+
+# Parallel loops
+
+Omega adopts the Kokkos programming model to express on-node parallelism. To provide
+simplified syntax for the most frequently used computational patterns, Omega provides
+wrappers funtions that internally handle creating and setting-up Kokkos policies.
+
+## Flat multi-dimensional parallelism
+
+### parallelFor
+
+To perform parallel iteration over a multi-dimensional index range Omega provides the
+`parallelFor` wrapper. For example, the following code shows how to set every element of
+a 3D array in parallel.
+```c++
+   Array3DReal A("A", N1, N2, N3);
+   parallelFor(
+       {N1, N2, N3},
+       KOKKOS_LAMBDA(int J1, int J2, int J3) {
+          A(J1, J2, J3) = J1 + J2 + J3;
+       });
+```
+Ranges with up to five dimensions are supported.
+Optionally, a label can be provided as the first argument of `parallelFor`.
+```c++
+   parallelFor("Set A",
+       {N1, N2, N3},
+       KOKKOS_LAMBDA(int J1, int J2, int J3) {
+          A(J1, J2, J3) = J1 + J2 + J3;
+       });
+```
+Adding labels can result in more informative messages when
+Kokkos debug variables are defined.
+
+### parallelReduce
+
+To perform parallel reductions over a multi-dimensional index range the
+`parallelReduce` wrapper is available. The following code sums
+every element of `A`.
+```c++
+   Real SumA;
+   parallelReduce(
+       {N1, N2, N3},
+       KOKKOS_LAMBDA(int J1, int J2, int J3, Real &Accum) {
+          Accum += A(J1, J2, J3);
+       },
+       SumA);
+```
+Note the presence of an accumulator variable `Accum` in the `KOKKOS_LAMBDA` arguments.
+You can use `parallelReduce` to perform other types of reductions.
+As an example, the following snippet finds the maximum of `A`.
+```c++
+   Real MaxA;
+   parallelReduce(
+       {N1, N2, N3},
+       KOKKOS_LAMBDA(int J1, int J2, int J3, Real &Accum) {
+          Accum = Kokkos::max(Accum, A(J1, J2, J3));
+       },
+       Kokkos::Max<Real>(MaxA));
+```
+To perform reductions that are not sums, in addition to modifying the lambda body,
+the final reduction variable needs to be cast to the appropriate type. In the above example,
+`MaxA` is cast to `Kokkos::Max<Real>` to perform a max reduction.
+The `parallelReduce` wrapper supports performing multiple reduction at the same time.
+You can compute `SumA` and `MaxA` in one pass over the data:
+```c++
+   parallelReduce(
+       {N1, N2, N3},
+       KOKKOS_LAMBDA(int J1, int J2, int J3, Real &AccumSum, Real &AccumMax) {
+          AccumSum += A(J1, J2, J3);
+          AccumMax = Kokkos::max(AccumMax, A(J1, J2, J3));
+       },
+       SumA, Kokkos::Max<Real>(MaxA));
+```
+Similarly to `parallelFor`, `parallelReduce` supports labels and up to five dimensions.

components/omega/doc/index.md

@@ -62,6 +62,7 @@ devGuide/Linting
 devGuide/Docs
 devGuide/BuildDocs
 devGuide/DataTypes
+devGuide/ParallelLoops
 devGuide/MachEnv
 devGuide/Config
 devGuide/Driver