Variable layers extrusion: non-collective behaviour in parallel

[ReubenHill]

Creating this mesh with variable layers

m = ExtrudedMesh(UnitIntervalMesh(3), np.array([[0, 3], [0, 3], [0, 2]]), np.array([3, 3, 2]))

causes a hang if you run with 2 MPI processes.

On one of the ranks it raises a value error here, on the other it does not.

On the rank where we get the value error we have

layers.shape = (3, 2)

and

(mesh.cell_set.total_size, 2) = (2, 2)

hence the error being triggered. On the other rank they are both (3, 2) so no error is triggered and you get a hang.
The ExtrudedMesh documentation does not indicate that this could happen;

In the variable layer height tests an indicator function is used to create the layers (see https://github.com/firedrakeproject/firedrake/blob/master/tests/extrusion/test_variable_layers_poisson.py) but this ought to be documented as necessary, fail if not obeyed, and certainly not cause hanging. Elsewhere when we supply mesh characteristics to a mesh constructor we do so globally and expect things to either 'just work' in parallel or give us an error.

[ksagiyam]

See also #2169

[wence-]

TBF, I did document the restriction on distribution of the layers in the manual https://firedrakeproject.org/extruded-meshes.html#variable-numbers-of-mesh-cell-layers

[ReubenHill]

TBF, I did document the restriction on distribution of the layers in the manual https://firedrakeproject.org/extruded-meshes.html#variable-numbers-of-mesh-cell-layers

Ah, fair enough. I guess it just needs to be added to the API documentation too then