Merge pull request #767 from baagaard-usgs/fix-petsc-defaults-collective-hdf5-output

baagaard-usgs · web-flow · commit 318e93e0c6f3 · 2024-07-19T13:10:56.000-06:00
Turn on collective PETSc HDF5 output by default (resolves HDF5 1.14 errors with NFS)
diff --git a/CHANGES.md b/CHANGES.md
@@ -11,8 +11,12 @@ The version numbers are in the form `MAJOR.MINOR.PATCH`, where major releases in
   * Expanded the section of the manual discussing mesh generation with Cubit and Gmsh. Added a list of useful functions in the Cubit and Gmsh Python interfaces.
   * Added a test to verify Gmsh files with entities that are not embedded generate errors when read via `MeshIOPetsc`.
   * Expanded section of manual discussing installation on Windows Subsystem for Linux, including components needed to run GUI applications (Gmsh and `pylith_viz`).
+  * Add Gmsh Python script and corresponding mesh in `examples/magma-2d`.
 * **Fixed**
   * Added output of fault traction change when computing static Green's functions.
+  * Turn on PETSc HDF5 collective input and output by default. This resolves errors when using HDF5 1.14 on NFS filesystems.
+  * Include all necessary files from `examples/subduction-3d` in source and binary packages.
+  * Fix typos in documentation and update Components section of User Guide (sync with code).
 
 ## Version 4.1.2 (2024/06/12)
 
diff --git a/docs/user/components/bc/AuxSubfieldsTimeDependent.md b/docs/user/components/bc/AuxSubfieldsTimeDependent.md
@@ -25,7 +25,7 @@ The association of these functions with the auxiliary subfields is:
 * `initial_amplitude`: Initial amplitude, f_0(x), subfield.
   - **current value**: 'subfield', from {default}
   - **configurable as**: subfield, initial_amplitude
-* `rate_amplitude`: Rate amplitude, \dot{f}_1(x), subfield.
+* `rate_amplitude`: Rate amplitude, dot{f}_1(x), subfield.
   - **current value**: 'subfield', from {default}
   - **configurable as**: subfield, rate_amplitude
 * `rate_start_time`: Rate starting time, t_1(x), subfield.
diff --git a/docs/user/components/materials/AuxSubfieldsIsotropicLinearPoroelasticity.md b/docs/user/components/materials/AuxSubfieldsIsotropicLinearPoroelasticity.md
@@ -20,6 +20,12 @@ Auxiliary subfields associated with the isotropic linear poroelastic bulk rheolo
 * `isotropic_permeability`: Isotropic permeability subfield.
   - **current value**: 'subfield', from {default}
   - **configurable as**: subfield, isotropic_permeability
+* `reference_strain`: Reference strain subfield.
+  - **current value**: 'subfield', from {default}
+  - **configurable as**: subfield, reference_strain
+* `reference_stress`: Reference stress subfield.
+  - **current value**: 'subfield', from {default}
+  - **configurable as**: subfield, reference_stress
 * `shear_modulus`: Shear modulus subfield.
   - **current value**: 'subfield', from {default}
   - **configurable as**: subfield, shear_modulus
diff --git a/docs/user/components/materials/DerivedSubfieldsPoroelasticity.md b/docs/user/components/materials/DerivedSubfieldsPoroelasticity.md
@@ -0,0 +1,38 @@
+# DerivedSubfieldsPoroelasticity
+
+% WARNING: Do not edit; this is a generated file!
+:Full name: `pylith.materials.DerivedSubfieldsPoroelasticity`
+:Journal name: `derivedsubfieldsporoelasticity`
+
+Derived subfields associated with the poroelasticity equation.
+
+For poroelastic materials these derived subfields are available for output in addition to the solution and auxiliary subfields.
+
+## Pyre Facilities
+
+* `bulk_density`: bulk density subfield.
+  - **current value**: 'subfield', from {default}
+  - **configurable as**: subfield, bulk_density
+* `cauchy_strain`: Cauchy strain subfield.
+  - **current value**: 'subfield', from {default}
+  - **configurable as**: subfield, cauchy_strain
+* `cauchy_stress`: Cauchy stress subfield.
+  - **current value**: 'subfield', from {default}
+  - **configurable as**: subfield, cauchy_stress
+* `water_content`: water content subfield
+  - **current value**: 'subfield', from {default}
+  - **configurable as**: subfield, water_content
+
+## Example
+
+Example of setting `DerivedSubfieldsPoroelasticity` Pyre properties and facilities in a parameter file.
+
+:::{code-block} cfg
+# The basis order for stress and strain should be at least 1 less than the basis order for displacement.
+[pylithapp.problem.materials.mat_elastic.derived_subfields]
+cauchy_stress.basis_order = 0
+cauchy_strain.basis_order = 0
+bulk_density.basis_order = 1
+water_content.basis_order = 1
+:::
+
diff --git a/docs/user/components/materials/index.md b/docs/user/components/materials/index.md
@@ -14,6 +14,7 @@ AuxSubfieldsIsotropicLinearPoroelasticity.md
 AuxSubfieldsIsotropicPowerLaw.md
 AuxSubfieldsPoroelasticity.md
 DerivedSubfieldsElasticity.md
+DerivedSubfieldsPoroelasticity.md
 Elasticity.md
 Homogeneous.md
 IncompressibleElasticity.md
diff --git a/docs/user/components/problems/InitialConditionPatch.md b/docs/user/components/problems/InitialConditionPatch.md
@@ -6,6 +6,11 @@
 
 Initial conditions over a portion of the domain (patch).
 
+:::{warning}
+Initial conditions over a patch currently only work for Gmsh input files.
+Exodus II files from Cubit do not contain the information needed by the current PyLith implementation.
+:::
+
 Implements `InitialCondition`.
 
 ## Pyre Facilities
diff --git a/docs/user/components/problems/SolnDispLagrange.md b/docs/user/components/problems/SolnDispLagrange.md
@@ -11,9 +11,9 @@ Container for solution subfields with displacement and fault Lagrange multiplier
 * `displacement`: Displacement subfield.
   - **current value**: 'subfielddisplacement', from {default}
   - **configurable as**: subfielddisplacement, displacement
-* `lagrange_fault`: Fault Lagrange multiplier subfield.
+* `lagrange_multiplier_fault`: Fault Lagrange multiplier subfield.
   - **current value**: 'subfieldlagrangefault', from {default}
-  - **configurable as**: subfieldlagrangefault, lagrange_fault
+  - **configurable as**: subfieldlagrangefault, lagrange_multiplier_fault
 
 ## Example
 
diff --git a/docs/user/components/problems/SolnDispPresLagrange.md b/docs/user/components/problems/SolnDispPresLagrange.md
@@ -11,9 +11,9 @@ Container for solution subfields with displacement, pressure, and fault Lagrange
 * `displacement`: Displacement subfield.
   - **current value**: 'subfielddisplacement', from {default}
   - **configurable as**: subfielddisplacement, displacement
-* `lagrange_fault`: Fault Lagrange multiplier subfield.
+* `lagrange_multiplier_fault`: Fault Lagrange multiplier subfield.
   - **current value**: 'subfieldlagrangefault', from {default}
-  - **configurable as**: subfieldlagrangefault, lagrange_fault
+  - **configurable as**: subfieldlagrangefault, lagrange_multiplier_fault
 * `pressure`: Pressure subfield.
   - **current value**: 'subfieldpressure', from {default}
   - **configurable as**: subfieldpressure, pressure
diff --git a/docs/user/components/problems/SolnDispPresTracStrainLagrange.md b/docs/user/components/problems/SolnDispPresTracStrainLagrange.md
@@ -0,0 +1,32 @@
+# SolnDispPresTracStrainLagrange
+
+% WARNING: Do not edit; this is a generated file!
+:Full name: `pylith.problems.SolnDispPresTracStrainLagrange`
+:Journal name: `SolnDispPresTracStrainLagrange`
+
+Container for solution subfields with displacement, pore pressure, trace strain subfields, and fault Lagrange multiplier subfields.
+
+## Pyre Facilities
+
+* `displacement`: Displacement subfield.
+  - **current value**: 'subfielddisplacement', from {default}
+  - **configurable as**: subfielddisplacement, displacement
+* `lagrange_multiplier_fault`: Fault Lagrange multiplier subfield.
+  - **current value**: 'subfieldlagrangefault', from {default}
+  - **configurable as**: subfieldlagrangefault, lagrange_multiplier_fault
+* `pressure`: Pressure subfield.
+  - **current value**: 'subfieldpressure', from {default}
+  - **configurable as**: subfieldpressure, pressure
+* `trace_strain`: Trace strain subfield.
+  - **current value**: 'subfieldtracestrain', from {default}
+  - **configurable as**: subfieldtracestrain, trace_strain
+
+## Example
+
+Example of setting `SolnDispPresTracStrainLagrange` Pyre properties and facilities in a parameter file.
+
+:::{code-block} cfg
+[pylithapp.problem]
+solution = pylith.problems.SolnDispPresTracStrainLagrange
+:::
+
diff --git a/docs/user/components/problems/SolnDispPresTracStrainVelPdotTdotLagrange.md b/docs/user/components/problems/SolnDispPresTracStrainVelPdotTdotLagrange.md
@@ -0,0 +1,41 @@
+# SolnDispPresTracStrainVelPdotTdotLagrange
+
+% WARNING: Do not edit; this is a generated file!
+:Full name: `pylith.problems.SolnDispPresTracStrainVelPdotTdotLagrange`
+:Journal name: `solndispprestracstrainvelpdottdotlagrange`
+
+Python solution field with displacement, pressure, and trace strain subfields, along with their time derivatives, and a lagrange fault.
+
+## Pyre Facilities
+
+* `displacement`: Displacement subfield.
+  - **current value**: 'subfielddisplacement', from {default}
+  - **configurable as**: subfielddisplacement, displacement
+* `lagrange_multiplier_fault`: Fault Lagrange multiplier subfield.
+  - **current value**: 'subfieldlagrangefault', from {default}
+  - **configurable as**: subfieldlagrangefault, lagrange_multiplier_fault
+* `pressure`: Pressure subfield.
+  - **current value**: 'subfieldpressure', from {default}
+  - **configurable as**: subfieldpressure, pressure
+* `pressure_t`: PressureT subfield.
+  - **current value**: 'subfieldpressure_t', from {default}
+  - **configurable as**: subfieldpressure_t, pressure_t
+* `trace_strain`: Trace strain subfield.
+  - **current value**: 'subfieldtracestrain', from {default}
+  - **configurable as**: subfieldtracestrain, trace_strain
+* `trace_strain_t`: TraceStrainT subfield.
+  - **current value**: 'subfieldtracestrain_t', from {default}
+  - **configurable as**: subfieldtracestrain_t, trace_strain_t
+* `velocity`: Velocity subfield.
+  - **current value**: 'subfieldvelocity', from {default}
+  - **configurable as**: subfieldvelocity, velocity
+
+## Example
+
+Example of setting `SolnDispPresTracStrainVelPdotTdotLagrange` Pyre properties and facilities in a parameter file.
+
+:::{code-block} cfg
+[pylithapp.problem]
+solution = pylith.problems.SolnDispPresTracStrainVelPdotTdotLagrange
+:::
+
diff --git a/docs/user/components/problems/SolnDispVelLagrange.md b/docs/user/components/problems/SolnDispVelLagrange.md
@@ -11,9 +11,9 @@ Container for solution subfields with displacement, velocity, and fault Lagrange
 * `displacement`: Displacement subfield.
   - **current value**: 'subfielddisplacement', from {default}
   - **configurable as**: subfielddisplacement, displacement
-* `lagrange_fault`: Fault Lagrange multiplier subfield.
+* `lagrange_multiplier_fault`: Fault Lagrange multiplier subfield.
   - **current value**: 'subfieldlagrangefault', from {default}
-  - **configurable as**: subfieldlagrangefault, lagrange_fault
+  - **configurable as**: subfieldlagrangefault, lagrange_multiplier_fault
 * `velocity`: Velocity subfield.
   - **current value**: 'subfieldvelocity', from {default}
   - **configurable as**: subfieldvelocity, velocity
diff --git a/docs/user/components/problems/SubfieldLagrangeFault.md b/docs/user/components/problems/SubfieldLagrangeFault.md
@@ -40,7 +40,7 @@ Implements `SolutionSubfield`.
 Example of setting `SubfieldLagrangeFault` Pyre properties and facilities in a parameter file.
 
 :::{code-block} cfg
-[pylithapp.problems.solution.subfields.lagrange_fault]
+[pylithapp.problems.solution.subfields.lagrange_multiplier_fault]
 alias = lagrange_multiplier_fault
 basis_order = 1
 :::
diff --git a/docs/user/components/problems/index.md b/docs/user/components/problems/index.md
@@ -22,7 +22,9 @@ SolnDispLagrange.md
 SolnDispPres.md
 SolnDispPresLagrange.md
 SolnDispPresTracStrain.md
+SolnDispPresTracStrainLagrange.md
 SolnDispPresTracStrainVelPdotTdot.md
+SolnDispPresTracStrainVelPdotTdotLagrange.md
 SolnDispPresVel.md
 SolnDispVel.md
 SolnDispVelLagrange.md
diff --git a/docs/user/components/topology/Distributor.md b/docs/user/components/topology/Distributor.md
@@ -9,15 +9,18 @@ Distributor of the the mesh among processes.
 ## Pyre Facilities
 
 * `data_writer`: Data writer for partition information.
-  - **current value**: 'datawritervtk', from {default}
-  - **configurable as**: datawritervtk, data_writer
+  - **current value**: 'datawriterhdf5', from {default}
+  - **configurable as**: datawriterhdf5, data_writer
 
 ## Pyre Properties
 
-* `partitioner`=\<str\>: Name of mesh partitioner.
-  - **default value**: 'chaco'
-  - **current value**: 'chaco', from {default}
-  - **validator**: (in ['chaco', 'metis', 'parmetis', 'simple'])
+* `partitioner`=\<str\>: Name of mesh partitioner (PETSc must be built with partitioner).
+  - **default value**: 'parmetis'
+  - **current value**: 'parmetis', from {default}
+  - **validator**: (in ['parmetis', 'chaco', 'simple'])
+* `use_edge_weighting`=\<bool\>: Use edge weighting (parmetis only).
+  - **default value**: True
+  - **current value**: True, from {default}
 * `write_partition`=\<bool\>: Write partition information to file.
   - **default value**: False
   - **current value**: False, from {default}
diff --git a/docs/user/components/utils/PetscDefaults.md b/docs/user/components/utils/PetscDefaults.md
@@ -9,6 +9,12 @@ No user-specified settings will be overwritten.
 
 ## Pyre Properties
 
+* `collective_io`=\<bool\>: Use default PETSc collective I/O options.
+  - **default value**: True
+  - **current value**: True, from {default}
+* `initial_guess`=\<bool\>: Use initial guess options.
+  - **default value**: True
+  - **current value**: True, from {default}
 * `monitors`=\<bool\>: Use default solver monitors.
   - **default value**: True
   - **current value**: True, from {default}
@@ -18,7 +24,7 @@ No user-specified settings will be overwritten.
 * `solver`=\<bool\>: Use default solver settings based on governing equations.
   - **default value**: True
   - **current value**: True, from {default}
-* `testing`=\<bool\>: Use default PETSc testingging options.
+* `testing`=\<bool\>: Use default PETSc testing options.
   - **default value**: False
   - **current value**: False, from {default}
 
@@ -31,6 +37,7 @@ Example of setting `PetscDefaults` Pyre properties and facilities in a parameter
 solver = True
 monitors = True
 parallel = False
+initial_guess = True
 testing = False
 :::
 
diff --git a/docs/user/run-pylith/petsc-options.md b/docs/user/run-pylith/petsc-options.md
@@ -22,7 +22,8 @@ We separate the defaults into a few categories to make it easy to select desired
 
 :solver: Options for the preconditioner and solver;
 :parallel: Options used when running in parallel (can be used in serial as well);
-:monitors: Options for basic monitoring of the solver; and
+:monitors: Options for basic monitoring of the solver;
+:collective_io: Options for collective input/output; and
 :testing: Options used in testing.
 
 :::{tip}
@@ -262,6 +263,24 @@ ts_monitor = tru
 ts_error_if_step_fails = true
 ```
 
+### Collective I/O
+
+The collective input and output options are enabled by default and turn on HDF5 collective output.
+We use parallel HDF5 implementation, which in turn relies on MPI IO.
+Many MPI IO implementations require collective input and output to be enabled for parallel HDF5 output even if only one process is being used.
+
+```{code-block} cfg
+---
+caption: Default PETSc options for collective input and output.
+---
+[pylithapp.problem.petsc_defaults]
+collective_io = True
+
+# Corresponding PETSc options
+[pylithapp.petsc]
+viewer_hdf5_collective = true
+```
+
 ### Testing
 
 The options in the testing category are intended for use in internal testing.
diff --git a/libsrc/pylith/utils/PetscOptions.cc b/libsrc/pylith/utils/PetscOptions.cc
@@ -63,6 +63,13 @@ namespace pylith {
             static
             void addMonitoring(PetscOptions* options);
 
+            /** Add collective I/O options.
+             *
+             * @param[in] options PETSc options.
+             */
+            static
+            void addCollectiveIO(PetscOptions* options);
+
             /** Add default solver tolerances to options.
              *
              * @param[in] options PETSc options.
@@ -88,6 +95,7 @@ const int pylith::utils::PetscDefaults::SOLVER = 0x2;
 const int pylith::utils::PetscDefaults::PARALLEL = 0x4;
 const int pylith::utils::PetscDefaults::INITIAL_GUESS = 0x8;
 const int pylith::utils::PetscDefaults::TESTING = 0x10;
+const int pylith::utils::PetscDefaults::COLLECTIVE_IO = 0x20;
 
 // ------------------------------------------------------------------------------------------------
 // Set default PETSc solver options based on solution field and material.
@@ -123,6 +131,9 @@ pylith::utils::PetscDefaults::set(const pylith::topology::Field& solution,
     if (flags & MONITORS) {
         _PetscOptions::addMonitoring(options);
     } // if
+    if (flags & COLLECTIVE_IO) {
+        _PetscOptions::addCollectiveIO(options);
+    } // if
 
     options->set();
     delete options;options = NULL;
@@ -305,6 +316,17 @@ pylith::utils::_PetscOptions::addMonitoring(PetscOptions* options) {
 } // addMonitoring
 
 
+// ------------------------------------------------------------------------------------------------
+// Add collective I/O options.
+void
+pylith::utils::_PetscOptions::addCollectiveIO(PetscOptions* options) {
+    assert(options);
+
+    options->add("-viewer_hdf5_collective");
+
+} // addMonitoring
+
+
 // ------------------------------------------------------------------------------------------------
 // Add default solver tolerances to options.
 void
diff --git a/libsrc/pylith/utils/PetscOptions.hh b/libsrc/pylith/utils/PetscOptions.hh
@@ -31,6 +31,7 @@ public:
     static const int PARALLEL;
     static const int INITIAL_GUESS;
     static const int TESTING;
+    static const int COLLECTIVE_IO;
 
     // PUBLIC METHODS /////////////////////////////////////////////////////////////////////////////
 public:
diff --git a/modulesrc/utils/PetscOptions.i b/modulesrc/utils/PetscOptions.i
@@ -26,6 +26,7 @@ public:
             static const int PARALLEL;
             static const int INITIAL_GUESS;
             static const int TESTING;
+            static const int COLLECTIVE_IO;
 
             // NOT IMPLEMENTED ////////////////////////////////////////////////////////////////////
 private:
diff --git a/pylith/utils/PetscDefaults.py b/pylith/utils/PetscDefaults.py
@@ -41,6 +41,9 @@ class PetscDefaults(Component):
     initialGuess = pythia.pyre.inventory.bool("initial_guess", default=True)
     initialGuess.meta["tip"] = "Use initial guess options."
 
+    collectiveIO = pythia.pyre.inventory.bool("collective_io", default=True)
+    collectiveIO.meta["tip"] = "Use default PETSc collective I/O options."
+
     testing = pythia.pyre.inventory.bool("testing", default=False)
     testing.meta["tip"] = "Use default PETSc testing options."
 
@@ -60,6 +63,8 @@ def flags(self):
             value |= ModuleDefaults.MONITORS
         if self.initialGuess:
             value |= ModuleDefaults.INITIAL_GUESS
+        if self.collectiveIO:
+            value |= ModuleDefaults.COLLECTIVE_IO
         if self.testing:
             value |= ModuleDefaults.TESTING
         return value
