MPI fails with DESC

[nadav-nt-tao]

When running a DESC task with more than 2 processes, I get the following error (MPI works well):

Traceback (most recent call last):
  File "/home/ori/struphy/inps/params_C2A.py", line 67, in <module>
    main.run(model,
  File "/home/ori/struphy/src/struphy/main.py", line 210, in run
    model.allocate_feec(grid, derham_opts)
  File "/home/ori/struphy/src/struphy/models/base.py", line 180, in allocate_feec
    self._derham = setup_derham(
                   ^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/io/setup.py", line 138, in setup_derham
    derham = Derham(
             ^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 150, in __init__
    _derham = self.init_derham(
              ^^^^^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 835, in init_derham
    _derham = self._discretize_derham(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 989, in _discretize_derham
    self._discretize_space(
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 1077, in _discretize_space
Traceback (most recent call last):
  File "/home/ori/struphy/inps/params_C2A.py", line 67, in <module>
    main.run(model,
  File "/home/ori/struphy/src/struphy/main.py", line 210, in run
    model.allocate_feec(grid, derham_opts)
  File "/home/ori/struphy/src/struphy/models/base.py", line 180, in allocate_feec
    self._derham = setup_derham(
                   ^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/io/setup.py", line 138, in setup_derham
    derham = Derham(
             ^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 150, in __init__
    _derham = self.init_derham(
              ^^^^^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 835, in init_derham
    _derham = self._discretize_derham(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 989, in _discretize_derham
    self._discretize_space(
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 1077, in _discretize_space
Traceback (most recent call last):
  File "/home/ori/struphy/inps/params_C2A.py", line 67, in <module>
    main.run(model,
  File "/home/ori/struphy/src/struphy/main.py", line 210, in run
    model.allocate_feec(grid, derham_opts)
  File "/home/ori/struphy/src/struphy/models/base.py", line 180, in allocate_feec
    self._derham = setup_derham(
                   ^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/io/setup.py", line 138, in setup_derham
    derham = Derham(
             ^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 150, in __init__
    _derham = self.init_derham(
              ^^^^^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 835, in init_derham
    _derham = self._discretize_derham(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 989, in _discretize_derham
    self._discretize_space(
  File "/home/ori/struphy/src/struphy/feec/psydac_derham.py", line 1077, in _discretize_space
    carts = create_cart([ddm], [spaces_1d])
            ^^^^^^    carts = create_cart([ddm], [spaces_1d])
            ^^^^^^^^    carts = create_cart([ddm], [spaces_1d])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^        ^^^

  File "/home/ori/struphy/stenv/lib/python3.12/site-packages/psydac/fem/partitioning.py", line 133, in create_cart
    ^  File "/home/ori/struphy/stenv/lib/python3.12/site-packages/psydac/fem/partitioning.py", line 133, in create_cart
^^^^^^^^^^^^^^^^^    global_starts, global_ends = partition_coefficients(domain_decomposition, spaces)
^^^    global_starts, global_ends = partition_coefficients(domain_decomposition, spaces)
         ^^^^^^^^^^
  File "/home/ori/struphy/stenv/lib/python3.12/site-packages/psydac/fem/partitioning.py", line 133, in create_cart
                                     ^          global_starts, global_ends = partition_coefficients(domain_decomposition, spaces)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^        ^^^^^^^      ^^^^^^^^^^^^^^^^^^^^^^^^^^^          ^^^^^^^^         ^^^^^^^^^^^^^^^^^^
^^^^^^^  File "/home/ori/struphy/stenv/lib/python3.12/site-packages/psydac/fem/partitioning.py", line 68, in partition_coefficients
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^  File "/home/ori/struphy/stenv/lib/python3.12/site-packages/psydac/fem/partitioning.py", line 68, in partition_coefficients
    assert all(e-s+1 >= V.degree)
    ^^^^^^^^   ^^^^^^^    ^^^    assert all(e-s+1 >= V.degree)
^^^^^^^^^^^^         ^^^^^^^^^^^^^^^
  ^^^^^^^^
^^^^^^^^  File "/home/ori/struphy/stenv/lib/python3.12/site-packages/psydac/fem/partitioning.py", line 68, in partition_coefficients
AssertionError
^^^^^^^^    assert all(e-s+1 >= V.degree)
^^^^
AssertionError
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[9444,1],0]
  Exit code:    1

The .py file is:

from struphy.io.options import EnvironmentOptions, BaseUnits, Time
from struphy.geometry import domains
from struphy.fields_background import equils
from struphy.topology import grids
from struphy.io.options import DerhamOptions
from struphy.io.options import FieldsBackground
from struphy.initial import perturbations
from struphy.kinetic_background import maxwellians
from struphy.pic.utilities import (LoadingParameters,
                                   WeightsParameters,
                                   BoundaryParameters,
                                   BinningPlot,
                                   KernelDensityPlot,
                                   )
from struphy import main

# import model, set verbosity
from struphy.models.fluid import LinearMHD

# environment options
env = EnvironmentOptions()

# units
base_units = BaseUnits()

# time stepping
time_opts = Time(dt=0.003, Tend=0.03, split_algo="LieTrotter")

# geometry
domain = domains.DESCunit()

# fluid equilibrium (can be used as part of initial conditions)
equil = equils.DESCequilibrium(eq_name="c2a_2_beta_0.1.h5", rel_path=True, use_nfp=False, rmin=0.05,
                               Nel=(15, 17, 19), p=(3, 3, 3), T_kelvin=116060)

# grid
grid = grids.TensorProductGrid(Nel=(4, 4, 4))

# derham options
derham_opts = DerhamOptions(p=(2, 3, 2), spl_kind=(False, True, True), nq_pr=(2, 2, 1), polar_ck=-1)

# light-weight model instance
model = LinearMHD()

# species parameters
model.mhd.set_phys_params()

# propagator options
model.propagators.shear_alf.options = model.propagators.shear_alf.Options(solver="pcg", precond="MassMatrixPreconditioner")
model.propagators.mag_sonic.options = model.propagators.mag_sonic.Options(b_field=model.em_fields.b_field, solver="pbicgstab", precond="MassMatrixPreconditioner")

# background, perturbations and initial conditions
model.mhd.velocity.add_perturbation(perturbations.TorusModesCos(ms=(1, 1, 1), ns=(1, 1, 1), amps=(0.1, 0, 0), pfuns=("exp", "exp", "exp"), pfun_params=((0.5, 0.03), (0.5, 0.03), (0.5, 0.03)),
                                                                comp=2, given_in_basis="2"))

# optional: exclude variables from saving
# model.mhd.pressure.save_data = False

if __name__ == "__main__":
    # start run
    verbose = True

    main.run(model,
             params_path=__file__,
             env=env,
             base_units=base_units,
             time_opts=time_opts,
             domain=domain,
             equil=equil,
             grid=grid,
             derham_opts=derham_opts,
             verbose=verbose,
             )

This happens only for DESC. Works for LinearMHD example and Maxwell example.

[spossann]

This happens because you do not have enough elements after domain decomposition. I have added a better warning message to psydac here: struphy-hub/psydac-for-struphy#58

AssertionError: Local number of elements (after domain decomposition) is to small for spline degree p=2: [2 1 1] is not >= 2 everywhere.
             You can:
                 1. increase Nel
                 2. lower p
                 3. decrease the MPI size.

The easiest way to fix this is to increase Nel. For instance, your example will work for Nel=(4, 4, 6) on 3 processes. You could also lower p.

[spossann]

@nadav-nt-tao can this be closed?

[nadav-nt-tao]

Yes, it worked.