Enable MultiApp transfers between MFEM and libMesh problems on CPU

framework/doc/content/source/mfem/transfers/MFEMNodalProjector.md

@@ -0,0 +1,15 @@
+# MFEMNodalProjector
+
+!if! function=hasCapability('mfem')
+
+Auxiliary class for extracting nodal positions/interpolation points from MFEM FESpaces, and for setting DoFs on MFEM GridFunctions
+via projection given a vector of values evaluated at these nodes.
+
+End users should not need to interact with `MFEMNodalProjector` directly. `MFEMNodalProjector` is 
+used by MFEM transfers that set values of an MFEMGridFunction, such as 
+[MultiAppMFEMShapeEvaluationTransfer](MultiAppMFEMShapeEvaluationTransfer.md).
+
+!if-end!
+
+!else
+!include mfem/mfem_warning.md

framework/doc/content/source/mfem/transfers/MultiAppMFEMShapeEvaluationTransfer.md

@@ -0,0 +1,47 @@
+# MultiAppMFEMShapeEvaluationTransfer
+
+!if! function=hasCapability('mfem')
+
+Allows transfers of [MFEM variables](MFEMVariable.md) between MFEM-based applications via local
+evaluation of shape functions at target nodal projection points. The variables may be defined on
+different meshes and FESpaces of different orders, but must be real, share the same dimensionality,
+and both belong to an `MFEMProblem` in their respective applications.
+
+## General Description
+
+`MultiAppMFEMShapeEvaluationTransfer` executes transfers in three steps:
+
+1. Extraction of a vector of node positions in the `ParFiniteElementSpace` of the destination
+   `mfem::ParGridFunction`, using [`MFEMNodalProjector`](MFEMNodalProjector.md).
+2. Interpolation of the source `mfem::ParGridFunction` at this set of node locations, using
+   `mfem::FindPointsGSLIB` to perform the required gather/scatter operations to obtain a set of
+   points on the local mesh partition, and perform shape function evaluations at these points.
+3. Projection of the evaluated source variable values onto the destination variable nodes, to set
+   the destination variable degrees of freedom, using [`MFEMNodalProjector`](MFEMNodalProjector.md).
+
+For transfers between identical variables defined on the same mesh, users are recommended to use
+[`MultiAppMFEMCopyTransfer`](MultiAppMFEMCopyTransfer.md) for performance.
+
+## Features Supported
+
+All MFEM to MFEM transfers executed from this class should be able to support:
+
+- transfers of scalar and vector variables
+- transfers between variables belonging to different finite element spaces and orders
+- transfers between complex variables
+- transfers between variables defined on submeshes and parent meshes
+- arbitrary number of parallel processes for both the source and target application
+- transfers between parent and child applications
+- transfers between sibling applications (child to child)
+- transfers from multiple variables to multiple variables
+
+!syntax parameters /Transfers/MultiAppMFEMShapeEvaluationTransfer
+
+!syntax inputs /Transfers/MultiAppMFEMShapeEvaluationTransfer
+
+!syntax children /Transfers/MultiAppMFEMShapeEvaluationTransfer
+
+!if-end!
+
+!else
+!include mfem/mfem_warning.md

framework/doc/content/source/mfem/transfers/MultiAppMFEMTolibMeshShapeEvaluationTransfer.md

@@ -0,0 +1,45 @@
+# MultiAppMFEMTolibMeshShapeEvaluationTransfer
+
+!if! function=hasCapability('mfem')
+
+Allows transfers of [MFEM variables](MFEMVariable.md) from MFEM-based applications to libMesh-based [MooseVariables](MooseVariable.md) via local
+evaluation of shape functions at target nodal projection points. This class supports transfers between different meshes, from scalar MFEM GridFunctions to first and second order LAGRANGE and constant MONOMIAL libMesh variable types.
+
+For transfers in the opposite direction, from libMesh-based applications to MFEM-based applications, please see [MultiApplibMeshToMFEMShapeEvaluationTransfer](MultiApplibMeshToMFEMShapeEvaluationTransfer.md).
+
+## General Description
+
+`MultiAppMFEMTolibMeshShapeEvaluationTransfer` executes transfers in three steps:
+
+1. Extraction of a vector of node positions or element centroids of the destination libMesh solution
+   variable (similar to the method used in
+   [`MultiAppGeneralFieldTransfer`](MultiAppGeneralFieldTransfer.md)).
+2. Interpolation of the source `mfem::ParGridFunction` at this set of node locations, using
+   `mfem::FindPointsGSLIB` to perform the required gather/scatter operations to obtain a set of
+   points on the local mesh partition, and perform shape function evaluations at these points.
+3. Projection of the evaluated source variable values onto the destination libMesh variable nodes,
+   to set the destination variable degrees of freedom.
+
+## Features Supported
+
+All MFEM to libMesh transfers executed from this class should be able to support:
+
+- transfers to lowest order scalar LAGRANGE and MONOMIAL variables
+- transfers to second order scalar LAGRANGE variables 
+- transfers between variables belonging to dissimilar meshes
+- arbitrary number of parallel processes for both the source and target application
+- transfers between parent and child applications
+- transfers between sibling applications (child to child)
+- transfers from multiple variables to multiple variables
+- transfers between displaced meshes
+
+!syntax parameters /Transfers/MultiAppMFEMTolibMeshShapeEvaluationTransfer
+
+!syntax inputs /Transfers/MultiAppMFEMTolibMeshShapeEvaluationTransfer
+
+!syntax children /Transfers/MultiAppMFEMTolibMeshShapeEvaluationTransfer
+
+!if-end!
+
+!else
+!include mfem/mfem_warning.md

framework/doc/content/source/mfem/transfers/MultiApplibMeshToMFEMShapeEvaluationTransfer.md

@@ -0,0 +1,48 @@
+# MultiApplibMeshToMFEMShapeEvaluationTransfer
+
+!if! function=hasCapability('mfem')
+
+Allows transfers of libMesh-based [MooseVariables](MooseVariable.md) to 
+[MFEM variables](MFEMVariable.md),
+via local evaluation of shape functions at target nodal projection points. This class supports
+transfers between different meshes, from first and second order LAGRANGE and constant MONOMIAL libMesh
+variable types to scalar MFEM GridFunctions defined on H1 and L2 conforming finite element spaces.
+
+For transfers in the opposite direction, from MFEM-based applications to libMesh-based applications, please see [MultiAppMFEMTolibMeshShapeEvaluationTransfer](MultiAppMFEMTolibMeshShapeEvaluationTransfer.md).
+
+## General Description
+
+`MultiApplibMeshToMFEMShapeEvaluationTransfer` executes transfers in three steps:
+
+1. Extraction of a vector of node positions in the `ParFiniteElementSpace` of the destination
+   `mfem::ParGridFunction`, using [`MFEMNodalProjector`](MFEMNodalProjector.md).
+2. Interpolation of the source libMesh-based `MooseVariable` at this set of node locations. Required
+   gather/scatter operations are performed similarly to those in
+   [`MultiAppGeneralFieldTransfer`](MultiAppGeneralFieldTransfer.md), with interpolation analagous
+   to [`MultiAppGeneralFieldShapeEvaluationTransfer`](MultiAppGeneralFieldShapeEvaluationTransfer.md).
+3. Projection of the evaluated source variable values onto the destination variable nodes, to set
+   the destination variable degrees of freedom, using [`MFEMNodalProjector`](MFEMNodalProjector.md).
+
+## Features Supported
+
+All libMesh to MFEM transfers executed from this class should be able to support:
+
+- transfers from lowest order scalar LAGRANGE and MONOMIAL variables
+- transfers from second order scalar LAGRANGE variables 
+- transfers between variables belonging to dissimilar meshes
+- arbitrary number of parallel processes for both the source and target application
+- transfers between parent and child applications
+- transfers between sibling applications (child to child)
+- transfers from multiple variables to multiple variables
+- transfers between displaced meshes
+
+!syntax parameters /Transfers/MultiApplibMeshToMFEMShapeEvaluationTransfer
+
+!syntax inputs /Transfers/MultiApplibMeshToMFEMShapeEvaluationTransfer
+
+!syntax children /Transfers/MultiApplibMeshToMFEMShapeEvaluationTransfer
+
+!if-end!
+
+!else
+!include mfem/mfem_warning.md

framework/include/mfem/functormaterials/MFEMFunctorMaterial.h

@@ -22,7 +22,7 @@ class MFEMFunctorMaterial : public MFEMObject,
 {
 public:
   static InputParameters validParams();
-  static libMesh::Point pointFromMFEMVector(const mfem::Vector & vec);
+  static libMesh::Point libMeshPointFromMFEMVector(const mfem::Vector & vec);
 
   MFEMFunctorMaterial(const InputParameters & parameters);
   virtual ~MFEMFunctorMaterial();

framework/include/mfem/outputs/MFEMDataCollection.h

@@ -25,9 +25,12 @@ class MFEMDataCollection : public FileOutput
   virtual mfem::DataCollection & getDataCollection() = 0;
 
 protected:
+  /// Register fields (GridFunctions) to be saved in the DataCollection
   void registerFields();
+  /// Write out data
   void output() override;
-
+  /// Update the DataCollection path when the internal file base path is set
+  void setFileBaseInternal(const std::string & file_base) override;
   /// Reference to the MFEMProblemData struct storing the output variables.
   MFEMProblemData & _problem_data;
   /// Mesh set of output variables are defined on. May differ from main problem mesh if SubMesh

framework/include/mfem/transfers/MFEMMultiAppTransfer.h

@@ -0,0 +1,156 @@
+//* This file is part of the MOOSE framework
+//* https://mooseframework.inl.gov
+//*
+//* All rights reserved, see COPYRIGHT for full restrictions
+//* https://github.com/idaholab/moose/blob/master/COPYRIGHT
+//*
+//* Licensed under LGPL 2.1, please see LICENSE for details
+//* https://www.gnu.org/licenses/lgpl-2.1.html
+
+#ifdef MOOSE_MFEM_ENABLED
+
+#pragma once
+
+#include "MultiAppTransfer.h"
+
+/**
+ * Virtual base class for MultiApp transfers to and/or from MFEMProblems.
+ */
+class MFEMMultiAppTransfer : public MultiAppTransfer
+{
+public:
+  static InputParameters validParams();
+  MFEMMultiAppTransfer(InputParameters const & params);
+  /// Set active source and destination problems, and execute variable transfer
+  void execute() override;
+  /// Allow sibling transfers
+  void checkSiblingsTransferSupported() const override {}
+
+  /// Getter for source variable name
+  const VariableName & getFromVarName(int i) const { return _from_var_names.at(i); }
+  /// Getter for destination variable name
+  const VariableName & getToVarName(int i) const { return _to_var_names.at(i); }
+  /// Return the number of source variables
+  unsigned int numFromVar() const { return _from_var_names.size(); }
+  /// Return for the number of destination variables
+  unsigned int numToVar() const { return _to_var_names.size(); }
+
+protected:
+  /// Transfer all variables from active source problem to active destination problem.
+  virtual void transferVariables(bool is_target_local) = 0;
+
+  /// Set current problem to fetch source variables from
+  void setActiveFromProblem(FEProblemBase & from_problem, const unsigned int global_app_index)
+  {
+    _active_from_problem = &from_problem;
+    _active_from_global_app_index = global_app_index;
+  }
+  /// Set current problem to fetch destination variables from
+  void setActiveToProblem(FEProblemBase & to_problem, const unsigned int global_app_index)
+  {
+    _active_to_problem = &to_problem;
+    _active_to_global_app_index = global_app_index;
+  }
+  /// Getter for current problem containing source variables
+  virtual FEProblemBase & getActiveFromProblem() { return *_active_from_problem; }
+  /// Getter for current problem containing destination variables
+  virtual FEProblemBase & getActiveToProblem() { return *_active_to_problem; }
+  /// Getter for current destination problem global app index.
+  virtual unsigned int & getActiveToProblemGlobalAppIndex() { return _active_to_global_app_index; }
+  /// Getter for current destination problem global app index.
+  virtual unsigned int & getActiveFromProblemGlobalAppIndex()
+  {
+    return _active_from_global_app_index;
+  }
+
+  /// Getter for the active source transform
+  const MultiAppCoordTransform & getActiveFromTransform() const
+  {
+    return *_from_transforms[_active_from_global_app_index];
+  }
+  /// Getter for the active destination transform
+  const MultiAppCoordTransform & getActiveToTransform() const
+  {
+    return *_to_transforms[_active_to_global_app_index];
+  }
+  /// Map a point in the active destination app frame to the active source app frame
+  libMesh::Point mapPointToActiveSourceFrame(const Point & point_in_target_frame) const
+  {
+    return getActiveFromTransform().mapBack(getActiveToTransform()(point_in_target_frame));
+  }
+  /// Get libMesh EquationSystem, which may or may not be displaced
+  libMesh::EquationSystems & getlibMeshEquationSystem(FEProblemBase & problem,
+                                                      bool use_displaced) const;
+
+  /// Set default value for transfers evaluated at points outside source mesh
+  void setMFEMOutOfMeshValue(mfem::real_t mfem_out_of_mesh_value)
+  {
+    _mfem_out_of_mesh_value = mfem_out_of_mesh_value;
+  }
+  /// Getter for default value for transfers evaluated at points outside source mesh
+  mfem::real_t getMFEMOutOfMeshValue() const { return _mfem_out_of_mesh_value; }
+
+  /// Templated method to check source and destination problems are of the expected types
+  template <typename TO_PROBLEM, typename FROM_PROBLEM>
+  void checkValidTransferProblemTypes();
+
+private:
+  /// Vector of source variable names to be transferred
+  const std::vector<VariableName> & _from_var_names;
+  /// Vector of destination variable names to transfer to
+  const std::vector<VariableName> & _to_var_names;
+  /// Pointer to active destination problem variable is being transferred to
+  FEProblemBase * _active_to_problem{nullptr};
+  /// Pointer to active source problem variable is being transferred from
+  FEProblemBase * _active_from_problem{nullptr};
+  /// Global app index for the active destination problem
+  unsigned int _active_to_global_app_index{0};
+  /// Global app index for the active source problem
+  unsigned int _active_from_global_app_index{0};
+  /// Default value to return for transfers from points outside the source mesh
+  mfem::real_t _mfem_out_of_mesh_value{std::numeric_limits<mfem::real_t>::infinity()};
+};
+
+template <typename TO_PROBLEM, typename FROM_PROBLEM>
+inline void
+MFEMMultiAppTransfer::checkValidTransferProblemTypes()
+{
+  auto bad_problem = [this]()
+  {
+    mooseError(
+        type(),
+        " is not compatible with the provided source and/or destination Problem types of the "
+        "provided variables.");
+  };
+
+  if (hasFromMultiApp())
+  {
+    // Check if source sub-apps exist, and if so, if they are of the expected type
+    if (getFromMultiApp()->numGlobalApps())
+    {
+      for (const auto i : make_range(getFromMultiApp()->numGlobalApps()))
+        if (getFromMultiApp()->hasLocalApp(i) &&
+            !dynamic_cast<FROM_PROBLEM *>(&getFromMultiApp()->appProblemBase(i)))
+          bad_problem();
+    }
+    else // check source sub-app is of the expected type if it is the transfer source
+      if (!dynamic_cast<FROM_PROBLEM *>(&getFromMultiApp()->problemBase()))
+        bad_problem();
+  }
+  if (hasToMultiApp())
+  {
+    // Check if destination sub-apps exist, and if so, if they are of the expected type
+    if (getToMultiApp()->numGlobalApps())
+    {
+      for (const auto i : make_range(getToMultiApp()->numGlobalApps()))
+        if (getToMultiApp()->hasLocalApp(i) &&
+            !dynamic_cast<TO_PROBLEM *>(&getToMultiApp()->appProblemBase(i)))
+          bad_problem();
+    }
+    else // check destination sub-app is of the expected type if it is the transfer destination
+      if (!dynamic_cast<TO_PROBLEM *>(&getToMultiApp()->problemBase()))
+        bad_problem();
+  }
+}
+
+#endif

framework/include/mfem/transfers/MultiAppMFEMCopyTransfer.h

@@ -11,39 +11,30 @@
 
 #pragma once
 
-#include <vector>
-
-#include "MultiAppTransfer.h"
-#include "MultiApp.h"
-#include "MooseAppCoordTransform.h"
+#include "MFEMMultiAppTransfer.h"
 #include "MFEMProblem.h"
 
-class MooseMesh;
-
-//*
-// Copy MFEMVariables between multiapps
-// The variables must be of the same type and dimension
-// and the MFEMMesh must be identical in both multiapps
-// */
-
-class MultiAppMFEMCopyTransfer : public MultiAppTransfer
+/**
+ * Transfer to copy MFEMVariables between multiapps.
+ * The variables must be of the same type and dimension
+ * and the MFEMMesh must be identical in both multiapps
+ */
+class MultiAppMFEMCopyTransfer : public MFEMMultiAppTransfer
 {
 public:
   static InputParameters validParams();
   MultiAppMFEMCopyTransfer(InputParameters const & params);
-  void execute() override;
-  auto const & getFromVarName(int i) { return _from_var_names.at(i); };
-  auto const & getToVarName(int i) { return _to_var_names.at(i); };
-  auto numFromVar() { return _from_var_names.size(); }
-  auto numToVar() { return _to_var_names.size(); }
 
 protected:
-  std::vector<VariableName> _from_var_names;
-  std::vector<AuxVariableName> _to_var_names;
-
-  void transfer(MFEMProblem & to_problem, MFEMProblem & from_problem);
-
+  /// Transfer all variables from active source problem to active destination problem.
+  virtual void transferVariables(bool is_target_local) override;
+  /// Check number of source and target child apps match for sibling transfer
   void checkSiblingsTransferSupported() const override;
+
+  /// Set current MFEM problem to fetch source variables from
+  virtual MFEMProblem & getActiveFromProblem() override;
+  /// Set current MFEM problem to fetch destination variables from
+  virtual MFEMProblem & getActiveToProblem() override;
 };
 
 #endif