Merge pull request #32479 from NamjaeChoi/kokkos_thread

Kokkos add surface normal, code cleanup using variadic template

Author: NamjaeChoi

framework/doc/content/source/kokkos/bcs/KokkosDirectionalNeumannBC.md

@@ -0,0 +1,20 @@
+# KokkosDirectionalNeumannBC
+
+!if! function=hasCapability('kokkos')
+
+This is the Kokkos version of [DirectionalNeumannBC](DirectionalNeumannBC.md). See the original document for details.
+
+## Example Input Syntax
+
+!listing test/tests/kokkos/bcs/directional_neumann/kokkos_2d_directional_neumann_bc_test.i start=[top] end=[] include-end=true
+
+!syntax parameters /BCs/KokkosDirectionalNeumannBC
+
+!syntax inputs /BCs/KokkosDirectionalNeumannBC
+
+!syntax children /BCs/KokkosDirectionalNeumannBC
+
+!if-end!
+
+!else
+!include kokkos/kokkos_warning.md

framework/doc/content/syntax/Kokkos/index.md

@@ -20,22 +20,22 @@ Here we provide instructions on some programming practices to write an efficient
 
 ### Separate Memory Space
 
-Except for a very few rare cases that provide physically unified memory space for CPU and GPU (such as AMD MI300A), most GPUs have separate memory spaces from their conuterpart CPUs.
-Therefore, the you need to take a special care to properly identify which data are accessible and not accessible on either CPU or GPU and whether the data on CPU and GPU are properly synchronized.
+Except for a very few rare cases that provide physically unified memory space for CPU and GPU (such as AMD MI300A), most GPUs have separate memory spaces from their counterpart CPUs.
+Therefore, you need to take a special care to properly identify which data are accessible and not accessible on either CPU or GPU and whether the data on CPU and GPU are properly synchronized.
 Standard containers such as `std::vector`, `std::set`, `std::map` and others are not usable on GPU, and managed pointers are also inaccessible on GPU.
 Basically, it is safe to assume that any dynamically-allocated data on CPU cannot be accessed on GPU.
 Therefore, we provide alternative data containers to be used on GPU: `Moose::Kokkos::Array`, `Moose::Kokkos::JaggedArray`, and `Moose::Kokkos::Map`.
 
 `Moose::Kokkos::Array` is a template class designed to hold arbitrary type of data.
 It receives up to four template arguments: data type, dimension, index type, and layout type.
-It supports multi-dimensional indexing, and up to five-dimensional arrays are supported.
+It supports multi-dimensional indexing.
 The dimension can either be specified through the second template argument with the default being one-dimension or using type aliases: for instance, a three-dimensional array of type `double` can be declared either by `Array<double, 3>` or `Array3D<double>`.
 The entries of an array can be accessed with either `operator()` with multi-dimensional indices or `operator[]` with a flattened, dimensionless index, where the flattening follows a layout in which the innermost dimension varies the fastest.
+They automatically return either CPU or GPU data depending on where they are being accessed.
 The index type template argument is set to 8-byte integer by default to accomodate large arrays.
 However, 8-byte integer computation is significantly more expensive than 4-byte integer computation.
 If your array size is small enough, consider using 4-byte indices to optimize index calculations.
 If having the outermost dimension run the fastest is desired for multi-dimensional arrays, the fourth layout template argument can be optionally set to `Moose::Kokkos::LayoutType::RIGHT` (default is `LEFT`).
-They automatically return either CPU or GPU data depending on where they are being accessed.
 Arrays can be allocated through the following APIs: `create()`, `createHost()`, and `createDevice()`.
 `create()` allocates memories on both CPU and GPU, while `createHost()` or `createDevice()` only allocates memory on either CPU or GPU.
 It is important to note that if the creation APIs are called for an initialized array, the original array will be destroyed and a new array will be created.
@@ -61,14 +61,15 @@ vector.aliasHost(petsc_ptr);
 vector.copyToDevice();
 ```
 
-If the data type is not default-constructable, `create()` will only allocate a raw block of uninitialized memory using `malloc()`.
-It is your responsibility to loop over the array and perform placement new to properly construct each entry.
+If the data type is not default-constructible or if you do not want to initialize array using the default constructor, you can set an optional template argument to `false` when calling `create()` or `createHost()`.
+It will only allocate a raw chunk of uninitialized memory using `malloc()` instead of `new`.
+Then, it becomes your responsibility to loop over the array and properly construct each entry using placement new.
 For example:
 
 ```cpp
 Array<NotDefaultConstructable> data;
 
-data.create(n);
+data.create<false>(n);
 
 for (auto & datum : data)
   new (&datum) NotDefaultConstructable(...);
@@ -139,7 +140,6 @@ It is divided into inner and outer arrays.
 The outer array is the regular part of a jagged array.
 Each entry of the outer array is the inner array, whose size can vary with each other.
 As a result, it is defined with up to five template arguments: the data type, inner array dimension size, outer array dimension size, outer array index type (defaults to 8-byte integer; inner arrays always use 4-byte integer), and inner array layout type (defaults to `Moose::Kokkos::LayoutType::LEFT`).
-Both inner and outer arrays can be up to three-dimensional.
 However, it is not possible to have inner arrays with different dimensions in a single jagged array.
 
 The accessors of a jagged array, `operator()` (dimensional) or `operator[]` (dimensionless), receive the indices for the outer array.

framework/doc/content/syntax/KokkosMaterials/index.md

@@ -40,7 +40,7 @@ Same applies to a class with in-class member initialization, which is equivalent
 Using a class with dynamic allocations will [incur a significant performance hit](syntax/Kokkos/index.md#kokkos_dynamic_allocation) and will break when it is used for stateful material properties.
 
 Instead, the material properties in Kokkos-MOOSE can be multi-dimensional to partially support the needs for dynamically-sized material properties.
-The dimension is provided as the second template argument `dimension`, which has the default value of 0 (scalar) and can be up to 4.
+The dimension is provided as the second template argument `dimension`, which has the default value of 0 (scalar).
 The size of each dimension is provied as a vector as the function argument `dims`.
 When a material property is declared by multiple materials, it should have the same dimension over the entire domain, while the size of each dimension can be different between non-overlapping subdomains.
 However, a material property declared by boundary-restricted materials should have identical dimension sizes over the entire domain, even though the materials do not have overlapping boundaries.

framework/include/interfaces/BlockRestrictable.h

@@ -277,7 +277,7 @@ class BlockRestrictable
    * @param tid The thread ID
    * @returns The contiguous element ID
    */
-  KOKKOS_FUNCTION ContiguousElementID kokkosBlockElementID(ThreadID tid) const
+  KOKKOS_FUNCTION ContiguousElementID kokkosBlockElementID(Moose::Kokkos::ThreadID tid) const
   {
     return _kokkos_element_ids[tid];
   }
@@ -286,7 +286,7 @@ class BlockRestrictable
    * @param tid The thread ID
    * @returns the contiguous node ID
    */
-  KOKKOS_FUNCTION ContiguousElementID kokkosBlockNodeID(ThreadID tid) const
+  KOKKOS_FUNCTION ContiguousElementID kokkosBlockNodeID(Moose::Kokkos::ThreadID tid) const
   {
     return _kokkos_node_ids[tid];
   }
@@ -295,7 +295,7 @@ class BlockRestrictable
    * @param tid The thread ID
    * @returns The contiguous element ID - side index pair
    */
-  KOKKOS_FUNCTION auto kokkosBlockElementSideID(ThreadID tid) const
+  KOKKOS_FUNCTION auto kokkosBlockElementSideID(Moose::Kokkos::ThreadID tid) const
   {
     return _kokkos_element_side_ids[tid];
   }

framework/include/interfaces/BoundaryRestrictable.h

@@ -261,7 +261,7 @@ class BoundaryRestrictable
    * @param tid The thread ID
    * @returns The contiguous node ID
    */
-  KOKKOS_FUNCTION ContiguousNodeID kokkosBoundaryNodeID(ThreadID tid) const
+  KOKKOS_FUNCTION ContiguousNodeID kokkosBoundaryNodeID(Moose::Kokkos::ThreadID tid) const
   {
     return _kokkos_node_ids[tid];
   }
@@ -270,7 +270,7 @@ class BoundaryRestrictable
    * @param tid The thread ID
    * @returns The contiguous element ID - side index pair
    */
-  KOKKOS_FUNCTION auto kokkosBoundaryElementSideID(ThreadID tid) const
+  KOKKOS_FUNCTION auto kokkosBoundaryElementSideID(Moose::Kokkos::ThreadID tid) const
   {
     return _kokkos_element_side_ids[tid];
   }