extending flux-pmix to support MPI sessions

[garlick]

Notes for discussion on adding MPI sessions to flux-pmix.

Background

• flux-core docs may be useful for reference/background
• flux-pmix is a flux-shell plugin
• the shell is started on each node of a job, running as the job owner, and is the process parent of all MPI procs
• the plugin can be enabled by default in a site config or on the command line, flux run -o pmi=pmix ...
• the plugin embeds the openpmix server
• it currently only supports simple MPI bootstrap with KVS and fence
• it is installed in production at LLNL and is actively maintained
• however, no new features have been added since bootstrap was achieved, due to lack of priority

Special Handling of Server Callbacks

• the flux shell is single threaded (event driven), while the openpmix server is designed to be embedded in MT program
• openpmix "thread shifts" calls from the plugin (shell thread) to its server thread, but makes callbacks in the server thread context
• a scheme was devised to turn the parameters of those callbacks into messages that are sent over a channel between threads, to be handled in the shell event loop
• as a consequence registering a new pmix callback requires adding a bit of project specific infrastructure

PMIx namespace

• It was patterned after slurm's pmix plugin, and apparently the source material was wrong/old - See Comments/suggestions for registering an nspace #58
• It represents one flat, statically allocated job
• There is always one application (in the MPMD sense)
• It seems likely that the nspace organization should be overhauled before sessions are implemented

Example of how a job is currently represented

  FLUX JOB (jobid = "ƒABC123")
  └─ PMIx Namespace: "ƒABC123"
     │
     ├─ Job-wide metadata (same on every broker):
     │    PMIX_JOBID      = "ƒABC123"
     │    PMIX_UNIV_SIZE  = 12
     │    PMIX_JOB_SIZE   = 12
     │    PMIX_NODE_MAP   = "host[1-3]"       (PMIx regex)
     │    PMIX_PROC_MAP   = "4,4,4"           (procs per node)
     │
     ├─ Per-rank info (PMIX_PROC_INFO_ARRAY, all 12 ranks, same on every broker):
     │    rank 0:  { nodeid=0, local_rank=0 }
     │    rank 1:  { nodeid=0, local_rank=1 }
     │    rank 2:  { nodeid=0, local_rank=2 }
     │    rank 3:  { nodeid=0, local_rank=3 }
     │    rank 4:  { nodeid=1, local_rank=0 }
     │    ...
     │    rank 11: { nodeid=2, local_rank=3 }
     │
     ├─ BROKER 0 (host1) — local portion
     │    PMIX_LOCAL_SIZE  = 4
     │    PMIX_LOCAL_PEERS = "0,1,2,3"
     │    registered clients: ranks 0–3
     │    ├─ rank 0  (local_rank 0)
     │    ├─ rank 1  (local_rank 1)
     │    ├─ rank 2  (local_rank 2)
     │    └─ rank 3  (local_rank 3)
     │
     ├─ BROKER 1 (host2) — local portion
     │    PMIX_LOCAL_SIZE  = 4
     │    PMIX_LOCAL_PEERS = "4,5,6,7"
     │    registered clients: ranks 4–7
     │    ├─ rank 4  (local_rank 0)
     │    ├─ rank 5  (local_rank 1)
     │    ├─ rank 6  (local_rank 2)
     │    └─ rank 7  (local_rank 3)
     │
     └─ BROKER 2 (host3) — local portion
          PMIX_LOCAL_SIZE  = 4
          PMIX_LOCAL_PEERS = "8,9,10,11"
          registered clients: ranks 8–11
          ├─ rank 8  (local_rank 0)
          ├─ rank 9  (local_rank 1)
          ├─ rank 10 (local_rank 2)
          └─ rank 11 (local_rank 3)

work breakdown

• tracking issue: support MPI sessions #123