tracking issue: support MPI sessions

[garlick]

This is a tracking issue for adding support for MPI sessions

N.B. this is just a sketch as this point. Fill in items with actual issue links as the scope of work becomes clearer:

• support publish/lookup - pset coordination during resource changes
• support PMIX group construct/destruct #122
• register_events/notify event - async resource change signals
• spawn - process addition for job grow
• allocation_request - application -> scheduler grow/shrink signal

[HawkmoonEternal]

Related to this topic, we started analyzing PMIx requirements for Flux in Open-MPI and MPICH, and our current understanding is the following:

Objectives:

• Support basic MPI Sessions model with MPICH and Open-MPI
• Support MPI_Comm_spawn with MPICH and Open-MPI
• Support (PSet-based) allocation changes with Sessions (ongoing work in MPI Sessions WG)
• Support error handling/logging with MPICH and Open-MPI

Required PMIx features:

• MPI Sessions model:

  • groups: OpenMPI -> communicator creation from group

• MPI_Comm_spawn/accept/connect:

  • spawn: OpenMPI/MPICH -> process spawning
  • publish/lookup: OpenMPI -> spawn-intercomm creation & MPI_publish/lookup_name
  • connect/disconnect: OpenMPI -> connect parent and child jobs in spawn

• Allocation changes:

  • allocation_request: OpenMPI/MPICH -> interaction with RM for allocation changes
  • psets: OpenMPI/MPICH -> dynamic PSet creation by RTE to express process changes
  • query: OpenMPI -> fetch (dynamically created) PSets/Memberships from RTE
  • event_codes: MPICH -> receive (dynamically created) PSets/Memberships from RTE

• Error Handling / Logging:

  • abort: OpenMPI/MPICH -> For MPI_ERRORS_ARE_FATAL
  • log: OpenMPI -> error message logging