Parallel MODFLOW 6 Capabilities

[jdhughes-dev]

Parallel capabilities are available in the current version of MODFLOW (6.4.2). Parallel functionality is part of the main code base and will be a core capability moving forward.

Although parallel functionality is part of the main code base it is considered alpha->beta functionality at this point. In the current develop branch on GitHub, GWF, GWT, and coupled GWF-GWT models can be run in parallel. As of 11/13/2023, parallelization of the MVR package is in progress.

Currently, the parallel version of MODFLOW is not included in the release distribution because we have not worked out the best way to distribute the parallel libraries (MPI and PETSc).

Instructions for compiling the parallel version of MODFLOW on MacOS and Linux is available at

https://github.com/MODFLOW-USGS/modflow6/blob/develop/PARALLEL.md

On Windows, the best solution we have so far is to use WSL. Instructions for compiling parallel MODFLOW on the Windows Subsystem for Linux (WSL) is available at

https://github.com/jdhughes-usgs/parallel-modflow6-class/blob/main/wsl_setup.md

Parallel capabilities are still new so not all options are fully supported. Some external users have been using the parallel version of MODFLOW, but we are only providing limited support currently because of the alpha->beta status of the functionality. However, we are interested in hearing about problems users run into with their models but don’t guarantee that we can fix an issue in the time frame users may need for actual production modeling.

We have also added model splitting functionality in FloPy to generate multiple models from a single model. The multi-model functionality in MODFLOW is how we are currently supporting parallel capabilities. An example of using the FloPy model splitter can be found at

https://flopy.readthedocs.io/en/latest/Notebooks/mf6_parallel_model_splitting_example.html

[wpbonelli]

Quick note: there is now a parallel nightly build for Windows. With flopy installed you can get it with get-modflow : --repo modflow6-nightly-build --ostag win64par. Non-system dlls are bundled, no dependency installations needed.

While we have tested the distribution in a few different Windows environments, there may still be rough edges. If you hit any troubles, please file an issue and tag it with the "parallel" and "nightly build" labels.

[a0wunsch]

Is there a publication or case study of the scalability of MF6 parallelization?

[jdhughes-dev]

Not yet but we have done a fair amount of internal testing. Was are in the process of preparing a publication.

[jensdebruijn]

Do the released versions for linux and mac currently support parallel operation, or should it be compiled?

[wpbonelli]

At this time it must be compiled on linux and mac

[mgozdor2]

A couple questions:

1. Has anyone tried using MODPATH on a model run that has been parallelized? If so, was it successful?

2. I have many HFB boundary conditions in my model domain. It appears that you can't split the model through an HFB and it also seems that using the "mfsplit.optimize_splitting_mask(nparts=8)" or any other nparts ends in error in my case. Any other workarounds other than manually searching for breaks that I can use?

[wpbonelli]

Re 1), for particle tracking on the full grid, you'd need to merge the flow model's results into new head and budget files aligned to the combined grid. Flopy's model splitter can reconstruct heads, budget would need to be done manually, for the moment. If you only wanted tracking on the subdomains independently I don't see why that shouldn't work though I don't know of anyone trying it yet.

[JordanPhethean]

Hello, is anyone able to provide an update on the release of pre-compiled parallel MODFLOW for linux and mac?

I'm running MODFLOW 6.6.2 on Ubuntu and see the option for parallel mode in the startup message, although when I run sim.run_simulation(processors=2) I get the error 1. Can not run parallel mode with this executable: no MPI. I do have OpenMPI installed and can run the simulation with mpirun -np 2 mf6, but based on computation times I think this is running the whole simulation twice rather than the respective parts of the split model.

Any advice on how to install parallel MODFLOW and link with MPI on Linux would be much appreciated.

[mjr-deltares]

Hi @JordanPhethean , we do not precompile MODFLOW in parallel for linux and mac so I am thinking you are using the serial build and the message is correct. Running the serial mf6 with mpirun is probably running the same process twice, like you say.

Would it be possible for you to follow the installation instruction in the following guide:

https://github.com/MODFLOW-ORG/modflow6/wiki/Extended-MODFLOW-User-Guide

Let us know if this works for you.

[JordanPhethean]

Hi @mjr-deltares,

Thanks, exactly what I was looking for!

In case of help for others, I've created a Google Colab notebook that installs Condacolab, flopy, and Parallel MODFLOW6 in the Colab environment. Users can therefore test and use the Parallel MODFLOW6 installation in a virtual cloud environment.

And here is a copy of the required Conda environment file.

Hope of use, any comments/suggestions please let me know.

[mjr-deltares]

That's great, @JordanPhethean. And thank you for sharing your MODFLOW related work with the community!