PMI2 shim crashes #4
Description
Background information
What version of the PMIx Reference Library are you using? (e.g., v1.0, v2.1, git master @ hash, etc.)
- Open MPI v4.0.x (at open-mpi/ompi@cb3ed47) and PMIx 3.1.4
Describe how PMIx was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
PMIx built with:
./configure --prefix=${PMIX_ROOT} \
--with-hwloc=${HWLOC_INSTALL_PATH} \
--with-libevent=${LIBEVENT_INSTALL_PATH} \
--enable-debug
Open MPI built with:
./configure --prefix=${MPI_ROOT} \
--with-hwloc=${HWLOC_INSTALL_PATH} \
--with-libevent=${LIBEVENT_INSTALL_PATH} \
--with-pmix=${PMIX_ROOT} \
--enable-mpirun-prefix-by-default
Please describe the system on which you are running
- Operating system/version: RHEL 7.6
- Computer hardware: ppc64le
- Network type: localhost
Details of the problem
I was testing PMI2 shim with the pmi2_client.c in the v3.1.4 release.
[mpiuser@3cd5ac47d23f test]$ env | grep PMIX
PMIX_ROOT=/home/mpiuser/local/pmix
[mpiuser@3cd5ac47d23f test]$ mpicc pmi2_client.c -I${PMIX_ROOT}/include -L${PMIX_ROOT}/lib -lpmi2 -Wall -g -O0 -o pmi2_client
If I don't specify a gds component it will crash with the stack below:
Program trace with no gds specified
[mpiuser@3cd5ac47d23f test]$ mpirun -np 2 ./pmi2_client
INFO: spawned=0 size=20 rank=0 appnum=0
INFO: jobid=2406219777
INFO: TI1 : PASS
INFO: TI2 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4 : PASS
INFO: spawned=0 size=20 rank=1 appnum=0
INFO: jobid=2406219777
INFO: TI1 : PASS
INFO: TI2 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4 : PASS
INFO: tkey=sharedFilename tval=pmix-pmi2-check val= found=0
INFO: tkey=sharedFilename tval=pmix-pmi2-check val= found=0
FATAL: PMI2_Info_GetNodeAttr does not find expected key at test_item5:341
FATAL: value does not meet expectation at test_item5:342
FATAL: value does not meet expectation at test_item5:343
INFO: TI5 : FAIL
FATAL: PMI2_Info_GetNodeAttr does not find expected key at test_item5:341
FATAL: value does not meet expectation at test_item5:342
FATAL: value does not meet expectation at test_item5:343
INFO: TI5 : FAIL
INFO: TI6 : PASS
INFO: TI6 : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: TI7 : PASS
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: TI7 : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
[3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00144] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8 : FAIL
INFO: Rank 1 executing Put of key KEY-1
[3cd5ac47d23f:00143] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8 : FAIL
INFO: Rank 0 executing Put of key KEY-0
INFO: Rank 1 executing Fence
INFO: Rank 0 executing Fence
INFO: Rank 0 executing Get of key KEY-0
INFO: Rank 1 executing Get of key KEY-0
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 0 executing Get of key KEY-1
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 1 executing Get of key KEY-1
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 0 executing Get of key KEY-2
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 1 executing Get of key KEY-2
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9 : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=72
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9 : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=72
INFO: 1 : PMI2_KVS_Put [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
[3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-0=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-1=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-2=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-3=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-4=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-5=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-6=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-7=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
[3cd5ac47d23f:00144] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 1 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10 : FAIL
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-8=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-9=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-10=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-11=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-12=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-13=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-14=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-15=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-16=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-17=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-18=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-19=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-20=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-21=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-22=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-23=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-24=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-25=zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-26=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-27=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-28=cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-29=dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-30=eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-31=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-32=gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-33=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-34=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-35=jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-36=kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-37=llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-38=mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-39=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-40=oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-41=pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-42=qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-43=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-44=ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-45=tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-46=uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-47=vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-48=wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-49=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] 0
[3cd5ac47d23f:00138] [[36716,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00143] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 0 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10 : FAIL
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[36716,1],1]
Exit code: 4
--------------------------------------------------------------------------
I noticed a slightly different behavior if I forced the hash component.
Program trace with `hash` gds specified
[mpiuser@3cd5ac47d23f test]$ export PMIX_MCA_gds=hash
[mpiuser@3cd5ac47d23f test]$ mpirun -np 2 ./pmi2_client
INFO: spawned=0 size=20 rank=0 appnum=0
INFO: jobid=2406678529
INFO: TI1 : PASS
INFO: TI2 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4 : PASS
INFO: tkey=sharedFilename tval=pmix-pmi2-check val=pmix-pmi2-check found=1
INFO: TI5 : PASS
INFO: TI6 : PASS
INFO: spawned=0 size=20 rank=1 appnum=0
INFO: jobid=2406678529
INFO: TI1 : PASS
INFO: TI2 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Job Attributes (Do not mark test as failed)
INFO: TI3 : PASS
ERROR: PMIx and SLURM/PMI2 does not set Node Attributes (Do not mark test as failed)
INFO: TI4 : PASS
INFO: tkey=sharedFilename tval=pmix-pmi2-check val=pmix-pmi2-check found=1
INFO: TI5 : PASS
INFO: TI6 : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: TI7 : PASS
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: TI7 : PASS
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
[3cd5ac47d23f:00149] [[36723,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00155] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8 : FAIL
INFO: Rank 1 executing Put of key KEY-1
[3cd5ac47d23f:00154] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI8 : FAIL
INFO: Rank 0 executing Put of key KEY-0
INFO: Rank 1 executing Fence
INFO: Rank 0 executing Fence
INFO: Rank 1 executing Get of key KEY-0
INFO: Rank 0 executing Get of key KEY-0
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 1 executing Get of key KEY-1
INFO: tkey=KEY-0 tval=VALUE-0 val=VALUE-0 len=7
INFO: Rank 0 executing Get of key KEY-1
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 1 executing Get of key KEY-2
INFO: tkey=KEY-1 tval=VALUE-1 val=VALUE-1 len=7
INFO: Rank 0 executing Get of key KEY-2
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9 : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=15
INFO: 1 : PMI2_KVS_Put [RANK1-key-0=aaaaaaaaaaaaaaa] 0
FATAL: PMI2_KVS_Get [KEY-2=?] -1
INFO: TI9 : FAIL
INFO: fence_cnt=0 of fence_num=5 keys_per_fence=50 keys_total=0 val_size=15
INFO: 0 : PMI2_KVS_Put [RANK0-key-0=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-4=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-5=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-6=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-7=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-4=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-5=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-9=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-6=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-10=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-7=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-11=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-9=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-12=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-10=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-13=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-11=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-14=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-12=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-16=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-13=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-17=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-14=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-18=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-19=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-16=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-20=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-17=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-21=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-18=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-19=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-23=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-20=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-24=yyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-21=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-25=zzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-23=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-26=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-24=yyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-27=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-25=zzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-28=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-26=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-29=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-27=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-30=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-28=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-31=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-29=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-32=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-30=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-33=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-31=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-34=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-32=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-35=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-33=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-36=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-34=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-37=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-35=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-38=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-36=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-39=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-37=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-40=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-38=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-39=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-42=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-40=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-43=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-42=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-44=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-43=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-45=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-44=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-46=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-45=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-47=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-46=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-47=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Put [RANK1-key-49=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Put [RANK0-key-49=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-0=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-1=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-2=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-3=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-4=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-5=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-6=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-7=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-9=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-10=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-0=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-11=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-1=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-12=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-2=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-13=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-3=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-14=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-4=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-15=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-5=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-16=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-17=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-6=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-18=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-7=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-19=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-8=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-20=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-9=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-21=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-10=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-11=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-23=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-12=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-24=yyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-13=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-25=zzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-14=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-26=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-15=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-27=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-16=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-28=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-17=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-18=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-29=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-19=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-30=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-31=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-20=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-32=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-21=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-33=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-22=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-34=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-23=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-24=yyyyyyyyyyyyyyy] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-35=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-36=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-25=zzzzzzzzzzzzzzz] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-37=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-26=aaaaaaaaaaaaaaa] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-38=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-27=bbbbbbbbbbbbbbb] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-39=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-28=ccccccccccccccc] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-40=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-29=ddddddddddddddd] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-30=eeeeeeeeeeeeeee] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-42=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-31=fffffffffffffff] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-43=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-32=ggggggggggggggg] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-33=hhhhhhhhhhhhhhh] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-44=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-45=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-34=iiiiiiiiiiiiiii] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-46=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-35=jjjjjjjjjjjjjjj] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-47=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-36=kkkkkkkkkkkkkkk] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-37=lllllllllllllll] 0
INFO: 0 : PMI2_KVS_Get from 0 [RANK0-key-49=xxxxxxxxxxxxxxx] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-38=mmmmmmmmmmmmmmm] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-39=nnnnnnnnnnnnnnn] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-40=ooooooooooooooo] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-41=ppppppppppppppp] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-42=qqqqqqqqqqqqqqq] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-43=rrrrrrrrrrrrrrr] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-44=sssssssssssssss] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-45=ttttttttttttttt] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-46=uuuuuuuuuuuuuuu] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-47=vvvvvvvvvvvvvvv] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-48=wwwwwwwwwwwwwww] 0
INFO: 0 : PMI2_KVS_Get from 1 [RANK0-key-49=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-0=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-0=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-1=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-2=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-3=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-4=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-4=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-5=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-5=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-6=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-6=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-7=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-7=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-8=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-8=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-9=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-9=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-10=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-10=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-11=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-11=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-12=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-12=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-13=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-13=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-14=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-14=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-15=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-16=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-16=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-17=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-17=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-18=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-18=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-19=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-19=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-20=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-20=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-21=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-21=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-22=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-22=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-23=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-23=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-24=yyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-24=yyyyyyyyyyyyyyy] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-25=zzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-25=zzzzzzzzzzzzzzz] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-26=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-26=aaaaaaaaaaaaaaa] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-27=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-27=bbbbbbbbbbbbbbb] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-28=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-28=ccccccccccccccc] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-29=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-29=ddddddddddddddd] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-30=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-30=eeeeeeeeeeeeeee] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-31=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-31=fffffffffffffff] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-32=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-32=ggggggggggggggg] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-33=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-33=hhhhhhhhhhhhhhh] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-34=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-34=iiiiiiiiiiiiiii] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-35=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-35=jjjjjjjjjjjjjjj] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-36=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-37=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-36=kkkkkkkkkkkkkkk] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-38=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-37=lllllllllllllll] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-39=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-38=mmmmmmmmmmmmmmm] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-40=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-39=nnnnnnnnnnnnnnn] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-40=ooooooooooooooo] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-41=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-41=ppppppppppppppp] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-42=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-42=qqqqqqqqqqqqqqq] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-43=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-43=rrrrrrrrrrrrrrr] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-44=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-45=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-44=sssssssssssssss] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-46=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-45=ttttttttttttttt] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-47=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-46=uuuuuuuuuuuuuuu] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-47=vvvvvvvvvvvvvvv] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-48=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-48=wwwwwwwwwwwwwww] 0
INFO: 1 : PMI2_KVS_Get from 0 [RANK1-key-49=xxxxxxxxxxxxxxx] 0
INFO: 1 : PMI2_KVS_Get from 1 [RANK1-key-49=xxxxxxxxxxxxxxx] 0
[3cd5ac47d23f:00149] [[36723,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_fence.c at line 217
[3cd5ac47d23f:00154] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 0 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10 : FAIL
[3cd5ac47d23f:00155] PMIX ERROR: HANDSHAKE-FAILED in file client/pmix_client_get.c at line 334
FATAL: 1 : PMI2_KVS_Get [RANK2-key-0=?] -1
INFO: TI10 : FAIL
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[36723,1],0]
Exit code: 3
--------------------------------------------------------------------------
I'm not sure if this is a PMIx library issue or something with Open MPI as I see a mix of ORTE and PMIx error messages in there.