pip install firedrake Q&A

[connorjward]

This discussion is a place for Firedrake users and downstream package maintainers to have their questions answered about the impact of moving from firedrake-install to pip install firedrake (#4011).

[jrmaddison]

Is the recommendation to switch CI to the 'pip' docker tag now? Will this tag continue to exist, or will I need to update CI again in the future?

[connorjward]

For testing purposes I have been giving all of the new containers the pip tag but that will change to latest when we merge. If you want to get your CI passing I recommend using firedrake-vanilla-default:pip (or similar) to test things out and then change to firedrake-vanilla-default:latest once #4011 is merged.

[golowimmer]

What would this transition mean for HPC clusters? Is there anything I should keep in mind for the installation procedure on clusters?

[connorjward]

The new installation process is:

1. Install system dependencies
2. Install PETSc with the right external packages
3. Install Firedrake

Previously firedrake-install would try and do all 3 of these things for you at once behind the scenes which made debugging a nightmare if any of them failed and also made it very difficult to customise the different steps (in particular PETSc configure). By splitting them out into different parts the hope is that this gets a lot easier to handle. We provide a new script, called firedrake-configure, that will (among other things) print out the different PETSc configure options that we want. It is easy to then take this output, pipe it to a file, and then edit it to respect whatever system packages (e.g. MPI) that exist on the system.

PETSc have some resources and a lot of experience managing HPC installs so by making that step explicit it should also be easier to get help from them.

In terms of stage 3 (installing Firedrake itself) I think the only considerations are that a number of environment variables (e.g. CC, CXX, MPICC, HDF5_DIR) will need setting.

[golowimmer]

I see, thanks!

[dsroberts]

Hi @connorjward. I'm converting our build setup over to this now, and I have a couple of questions. I notice that firedrake-configure --show-env does not set $FC or any similar fortran compiler environment variable. libsupermesh does need a Fortran compiler, is there any risk to firedrake not installing correctly if the $FC environment variable is unset or set to some garbage value? Second, I've noticed the --petsc-int-type flag is no longer used. Was this flag doing anything internal to firedrake that would affect how it behaves with 64-bit PETSc integers, or only checking if PETSc needed to be rebuilt due to int types changing?

[JHopeCollins]

If you are not asking PETSc to build MPI (which we no longer default to), and you don't provide either the --mpicc=... --mpif90=... configure arguments or set $CC, $FC, then PETSc will try and sniff the right compiler. The relevant section from the configure.log of one of my PETSC builds is below.
So yes, if you have multiple mpi compilers available then PETSc will just find the first one in it's list of possible locations. This should mean you get consistent mpicc, mpicxx and mpifort (otherwise your MPI installation is probably broken to begin with), but it may not be the one you expect.

I'm not sure what's best to do here. I'd prefer setting --mpicc=mpicc --mpicxx=mpicxx --mpifort=mpifort configure flags over the environment variables $CC, $CXX, and $FC because it's more self-contained. Setting them without a path should make sure that PETSc uses the same compilers as the user would get if they just called mpicc etc from the command line.
@connorjward what do you think?

I can't see firedrake-install doing anything with --petsc-int-type other than passing PETSc --with-64-bit-indices if it's int64, so it should be fine just to add that to the list of configuration options yourself (Connor will be able to confirm). PETSc will make sure that any packages upstream of it has the right build for the integer type, and anything downstream of PETSc should use PetscInt so won't need the integer type passing separately (although there might be places where we've missed that).

PETSc's process for finding the MPI compilers:

TESTING: checkCCompiler from config.setCompilers(/home/clustor2/ma/j/jth113/petsc/config/BuildSystem/config/setCompilers.py:1457)
    Checking for program /home/clustor2/ma/j/jth113/fdsystem/bin/mpincc...not found
    Checking for program /usr/local/bin/mpincc...not found
    Checking for program /usr/local/sbin/mpincc...not found
    Checking for program /usr/local_machine/bin/mpincc...not found
    Checking for program /usr/local/sbin/mpincc...not found
    Checking for program /usr/local/bin/mpincc...not found
    Checking for program /usr/sbin/mpincc...not found
    Checking for program /usr/bin/mpincc...not found
    Checking for program /sbin/mpincc...not found
    Checking for program /bin/mpincc...not found
    Checking for program /usr/local/basilisk/src/mpincc...not found
    Checking for program /snap/bin/mpincc...not found
    Checking for program /home/ma/j/jth113/bin/mpincc...not found
    Checking for program /home/clustor2/ma/j/jth113/petsc/lib/petsc/bin/win32fe/mpincc...not found
    Checking for program /home/clustor2/ma/j/jth113/fdsystem/bin/mpicc...not found
    Checking for program /usr/local/bin/mpicc...not found
    Checking for program /usr/local/sbin/mpicc...not found
    Checking for program /usr/local_machine/bin/mpicc...not found
    Checking for program /usr/local/sbin/mpicc...not found
    Checking for program /usr/local/bin/mpicc...not found
    Checking for program /usr/sbin/mpicc...not found
    Checking for program /usr/bin/mpicc...found
              Defined make macro "CC" to "mpicc"

[connorjward]

is there any risk to firedrake not installing correctly if the $FC environment variable is unset or set to some garbage value?

We only have to set CC and CXX to work the limitation that setuptools doesn't know anything about MPI so defaults to the wrong compiler to build our extension modules. libsupermesh uses CMake which has a better way to detect MPI compilers so the same issue doesn't appear. Naturally if FC is set to some garbage value then things might well fail, but that is the responsibility of the user.

These environment variables are only set after PETSc configure, and so how PETSc detects the compilers is unrelated.

Second, I've noticed the --petsc-int-type flag is no longer used. Was this flag doing anything internal to firedrake that would affect how it behaves with 64-bit PETSc integers, or only checking if PETSc needed to be rebuilt due to int types changing?

If you want to install Firedrake with 64 bit indices then all you have to do is pass --with-64-bit-indices to PETSc configure. This is included as an example in the install documentation here. We no longer do any sort of detection of changing integer type so if you want to have a 64-bit PETSc then you will have to build it yourself.

In the medium term we would like to support a default-int64 'ARCH' for Firedrake like we have default and complex, but there are still known corners of the test suite that fail with 64 bit indices so I don't want to claim that we fully support it until we definitely do.

Any other questions about porting your build to the new method please let me know! I would be happy to take a look at anything you have done so far.

[dsroberts]

Thanks @JHopeCollins, @connorjward, those were my only questions. The porting process for us was quite smooth, we've been building PETSc separately using (mostly) the args provided by firedrake-install --show-petsc-configure-options and then using --honour-petsc-dir. We're maintaining Firedrake on HPC systems, so we've been building with system-provided MPI/blas/lapack/scalapack/etc. All we had to do to update our build pipeline was add in a couple of new environment variables and swap firedrake-install ... to pip install ....

[connorjward]

Amazing! So glad to hear a success story.

[ddundo]

Hi @connorjward, thanks for doing this. I was wondering why you decided not to install Firedrake in a venv in docker containers?

This seems to cause issues when running github actions jobs within the container, where the github actions environment doesn't seem to inherit or properly set up the environment variables from the container. There's probably a way to debug this, but I'm tempted to just go back to creating a venv - unless there's a reason not to.

[connorjward]

We decided to remove the venv from the Docker containers because it was no longer needed and led to confusion for a large number of users who didn't realise that the extra activation step was necessary.
I am aware that the workflow for the Docker containers on GitHub runners is now quite bizarre. I have an example of how to get things to work here. In particular you have to set --user root and PYTHONPATH.
Other users have already raised this and the right solution is to make the containers run with the root user instead of firedrake. I will make that happen at some point soon.

[ddundo]

Thanks, that fixed it and all looks good! It's really nice how much smaller the containers are now as well.

[ddundo]

Hi again! In a docker container I used to be able to run a script with python3 my_script.py that would contain something like this: subprocess.run(["mpiexec", "-n", "4", "python3", "other_script.py"]. But this no longer works.

Is this due to switching from MPICH to OpenMPI? Could you suggest a fix please?

[connorjward]

Yes that isn't allowed using OpenMPI because it involves calling MPI_Init multiple times. One slightly inconvenient way to avoid this is to take care that you aren't importing mpi4py.MPI, petsc4py.PETSc or firedrake in your outside script as those will initialise MPI.

Can you provide more information about what are you trying to do? In general it's not recommended practice to nest MPI subprocesses (hence why OpenMPI don't support it).

[ddundo]

That makes sense, thanks! I am trying to do something quite messy... at the moment I have a unit test that runs a simulation in serial and then in parallel (using the subprocess call above) and then, in serial, I compare the outputs of both (two CheckpointFiles). So I'll just try to separate them into multiple parts/tests.

[connorjward]

I did something very similar here:

firedrake/tests/firedrake/regression/test_vertex_based_limiter.py

Line 125 in 4b82203

def test_parallel_limiting():

in case it's useful

[ddundo]

Nice, thanks!