PMIX_ERR_UNPACK_INADEQUATE_SPACE in file runtime/data_type_support/prte_dt_unpacking_fns.c at line 222

[ashterenli]

                Open MPI: 6.1.0a1
  Open MPI repo revision: v2.x-dev-12210-g448fa674c9

and

+ ucx_info -v
# Library version: 1.20.0
# Library path: /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/ucx-1.20.0/lib/libucs.so.0
# API headers version: 1.20.0
# Git branch '', revision 4b7a6ca
# Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check CC=cc CFLAGS=-Wno-error=unused-command-line-argument -Wno-error=unused-but-set-variable -Wno-error=tautological-constant-compare -qopenmp CXX=CC CXXFLAGS=-Wno-error --without-cuda --without-gdrcopy --without-fuse3 --without-go --without-java --without-rocm --disable-doxygen-doc --with-dm --with-dc --with-rc --with-ud --with-mlx5-dv --with-ib-hw-tm --with-mcpu=native --with-march=native --enable-builtin-memcpy --enable-cma --enable-logging --enable-mt --enable-optimizations --prefix=/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/ucx-1.20.0

submitting a job to 11 nodes with 176 cores each (azure hbv4 instances) with:

mpiexec --display-map --bind-to core \
        -n 704 --map-by ppr:22:numa:pe=2 /usr/bin/env OMP_NUM_THREADS=2 "$exe" \
      : -n  24 --map-by ppr:1:l3cache    /usr/bin/env OMP_NUM_THREADS=4 "$exe" \
      : -n 176 --map-by ppr:22:numa:pe=2 /usr/bin/env OMP_NUM_THREADS=2 "$exe"

and getting

PMIX ERROR: PMIX_ERR_UNPACK_INADEQUATE_SPACE in file runtime/data_type_support/prte_dt_unpacking_fns.c at line 222
PMIX ERROR: PMIX_ERR_UNPACK_INADEQUATE_SPACE in file base/odls_base_default_fns.c at line 558

https://github.com/openpmix/prrte/blob/master/src/runtime/data_type_support/prte_dt_unpacking_fns.c#L222

https://github.com/openpmix/prrte/blob/master/src/mca/odls/base/odls_base_default_fns.c#L558

Is there something wrong with my mpiexec line?

Also, --display-map shows some weird results, I hope just printing glitches.
For the first node in the allocation all is correct:

Data for node: tst5-hbv4-1      Num slots: 176  Max slots: 0    Num procs: 88
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 0 Bound: package[0][core:L0-1]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 1 Bound: package[0][core:L2-3]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 2 Bound: package[0][core:L4-5]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 3 Bound: package[0][core:L6-7]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 4 Bound: package[0][core:L8-9]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 5 Bound: package[0][core:L10-11]
...

but for the second, and all other nodes, the core numbers seem corrupted:

Data for node: tst5-hbv4-2      Num slots: 176  Max slots: 0    Num procs: 88
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 88 Bound: package[0][core:L0-1-175]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 89 Bound: package[0][core:L2-3-175]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 90 Bound: package[0][core:L4-5-175]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 91 Bound: package[0][core:L6-7-175]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 92 Bound: package[0][core:L8-9-175]
        Process jobid: prterun-tst5-hbv4-1-3896068@1 App: 0 Process rank: 93 Bound: package[0][core:L10-1175]
...

Thank you

Anton

[ashterenli]

I attach a slightly sanitised job stdout

job-log.txt

[rhc54]

I don't recognize those version numbers - is this OMPI's "main" branch?? If so, try using the head of the PMIx and PRRTE "master" branches. I can't debug OMPI's fork of prrte. This looks like a problem we've had with Azure before, and it was solved months ago.

[ashterenli]

Hi Ralph. Yes, OMPI main, from a couple of weeks ago: m28d0458.

For PMIx and PRRTE -- I'm not sure exactly what I need to do.... I assumed all necessary bits are included in the OMPI tree, with the right versions.

For the prte_info and pmix_info built as part of my OMPI build, I get:

$ prte_info
                    PRTE: 5.0.0ompi-a1psrvr-v2.0.0rc1-5048-g5ad79eb285
      PRTE repo revision: psrvr-v2.0.0rc1-5048-g5ad79eb285
       PRTE release date: @PMIX_RELEASE_DATE@
                    PMIx: OpenPMIx 7.0.0a1, repo rev: v5.0.10rc1-2-gd30c15e6 (PMIx Standard: 4.1, Stable ABI: 0.0,
                          Provisional ABI: 0.0)
                  Prefix: /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-m28d0458
 Configured architecture: x86_64-pc-linux-gnu
          Configure host: tst5-hbv4-2
           Configured by: anton.shterenlikht
           Configured on: Wed Dec 17 20:50:21 UTC 2025
          Configure host: tst5-hbv4-2
  Configure command line: '--disable-option-checking'
                          '--prefix=/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-m28d0458'
                          '--enable-prte-ft' '--with-proxy-version-string=6.1.0a1' '--with-proxy-package-name=Open MPI'
                          '--with-proxy-bugreport=https://www.open-mpi.org/community/help/' '--disable-devel-check'
                          '--enable-prte-prefix-by-default' '--disable-pmix-lib-checks'
                          '--with-pmix-extra-libs=/dev/shm/SPE/1766004455/ompi/3rd-party/openpmix/src/libpmix.la'
                          '--enable-mpi-ext=affinity,pcollreq' '--enable-mpi-fortran=all' '--enable-mpi1-compatibility'
                          '--enable-static' '--with-cma' '--with-libevent=' '--with-hwloc='
                          '--with-ucx=/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/ucx-1.20.0' '--without-cuda'
                          '--with-slurm' '--without-xpmem' 'CC=icx' 'CFLAGS=-mtune=native -march=native -O3
                          -Wno-error=int-conversion' 'CPPFLAGS= -I/dev/shm/SPE/1766004455/ompi/3rd-party/openpmix/include
                          -I/dev/shm/SPE/1766004455/ompi/3rd-party/openpmix/include
                          -I/dev/shm/SPE/1766004455/ompi/3rd-party/openpmix/
                          -I/dev/shm/SPE/1766004455/ompi/3rd-party/openpmix/' 'CXX=icpx' 'CXXFLAGS=-mtune=native
                          -march=native -O3' 'FC=ifort' 'FCFLAGS=-mtune=native -march=native -O3'
                          'PKG_CONFIG_PATH=/data/ashterenli/spe-tst5/install/intel-2025.1.0.666/compiler/2025.1/lib/pkgconfig:/data/ashterenli/spe-tst5/install/intel-2025.1.0.666/tbb/2022.1/lib/pkgconfig:/data/ashterenli/spe-tst5/install/intel-2024.0.2.29/tbb/2021.11/env/../lib/pkgconfig:/data/ashterenli/spe-tst5/install/intel-2024.0.2.29/dpl/2022.3/lib/pkgconfig:/data/ashterenli/spe-tst5/install/intel-2024.0.2.29/compiler/2024.0/lib/pkgconfig'
                          '--cache-file=/dev/null' '--srcdir=.'
                Built by: anton.shterenlikht
                Built on: Wed Dec 17 20:52:47 UTC 2025
              Built host: tst5-hbv4-2
              C compiler: icx
     C compiler absolute: /data/ashterenli/spe-tst5/install/intel-2025.1.0.666/compiler/2025.1/bin/icx
  C compiler family name: CLANG
      C compiler version: "20.0.0git (icx 2025.1.0.20250317)"

and

 $ pmix_info
                 Package: PMIx anton.shterenlikht@tst5-hbv4-2 Distribution
                    PMIX: 7.0.0a1
      PMIX repo revision: v5.0.10rc1-2-gd30c15e6
       PMIX release date: Unreleased developer copy
           PMIX Standard: 4.1
       PMIX Standard ABI: Stable (0.0), Provisional (0.0)
                  Prefix: /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-m28d0458
 Configured architecture: pmix.arch
          Configure host: tst5-hbv4-2
           Configured by: anton.shterenlikht
           Configured on: Wed Dec 17 20:49:52 UTC 2025
          Configure host: tst5-hbv4-2
  Configure command line: 'dummy' 'perl'
                Built by: anton.shterenlikht
                Built on: Wed Dec 17 20:52:11 UTC 2025
              Built host: tst5-hbv4-2
              C compiler: icx
     C compiler absolute: /data/ashterenli/spe-tst5/install/intel-2025.1.0.666/compiler/2025.1/bin/icx
  C compiler family name: CLANG
      C compiler version: "20.0.0git (icx 2025.1.0.20250317)"

Thank you

[rhc54]

You need to clone and install the PMIx and PRRTE "master" branches from upstream:

https://github.com/openpmix/openpmix
https://github.com/openpmix/prrte

Then reconfigure OMPI --with-pmix <path-to-pmix-install> --with-prrte <path-to-prrte-install>.

I cannot help you with the embedded versions - they are out-of-sync with the upstream and basically unmaintained. Which is why I continue to recommend you use the external support instead.

[rhc54]

Something you can try in the interim: add --hetero-nodes to your cmd line. The root cause of problems on Azure is that they reserve several cores on each node for their hypervisor. This creates an asymmetric topology (i.e., the objects in the cpu chip all have the same number of cores - except for one object that they took their cpus from) that causes HPC systems considerable trouble.

Only way to make it work is to treat every node as a unique topology. We try to automatically detect this scenario, but older runtime versions don't do as well as the latest ones. So if you are using a runtime somewhat out-of-date, we may not be recognizing the situation. Hence my suggestion to update to the latest runtime, or to try the cmd line option that simply tells us "the topologies are all different" (which may not work with an older version).

You will also see trouble if you try, for example, to map a specific number of procs per object - IIRC, Azure takes their cpus from one L3 (at least, for some chips), so mapping N procs by L3 or NUMA can run into trouble as one of those objects has fewer cpus to use. Just a "heads up".

Just FWIW: the need for hypervisors to do this is why HPC has gravitated more towards containers (which don't need to reserve cpus for their overhead) as opposed to VMs.

[ashterenli]

sorry for being so slow. Now all clear. I'll try to rebuild

[ashterenli]

I built pmix mbb34c7c, and trying to build prrte 0996b79:

+ git checkout
Your branch is up to date with 'origin/master'.
+ git rev-parse --short HEAD
0996b79c2b

============================================================================
== Configure PMIx
============================================================================
checking for pmix pkg-config name... /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/lib/pkgconfig/pmix.pc
checking if pmix pkg-config module exists... yes
checking for pmix pkg-config cflags... -I/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/include -I/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/include/pmix -I/data/ashterenli/spe-tst5/install/libevent-2.1.12-stable/include -I/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/hwloc-2.12.2/include -I/usr/include/libxml2
checking for pmix pkg-config ldflags... -Wl,-rpath -Wl,/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/lib -Wl,--enable-new-dtags
checking for pmix pkg-config static ldflags... -Wl,-rpath -Wl,/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/lib -Wl,--enable-new-dtags                                                                                               checking for pmix pkg-config libs... -lpmix                                                                                   checking for pmix pkg-config static libs... -lpmix
checking for pmix.h... yes
checking for PMIx_Init... yes
checking version at or above v6.1.0... yes
checking version below v1000.0.0... yes
checking for src/util/pmix_argv.h... yes
checking for support of version 2 server upcalls... yes
checking for in-memory show-help content compatibility... yes
checking for LTO compatibility... yes


Required Packages
-----------------------
HWLOC: yes (pkg-config: /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/hwloc-2.12.2)
Libevent: yes (/data/ashterenli/spe-tst5/install/libevent-2.1.12-stable)
PMIx: yes (pkg-config: /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c)
Python: yes (3.12)


/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `PMIx_Group_operation_string'
/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `PMIx_Value_get_number'
/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `pmix_show_help_add_data'
/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `PMIx_Info_list_add_value'

am I missing something? Or one master branch is ahead of another?

Thank you

[rhc54]

Hmmm...no, they are in sync. Looks like the PRRTE build is picking up some older PMIx version - what was the PRRTE configure line?

[ashterenli]

./configure \
CC=cc \
--with-libevent=${SPE_LIBEVENT_PATH} \
--with-hwloc=${SPE_HWLOC_PATH} \
--with-pmix=${SPE_PMIX_PATH} \
--prefix=${INSTALL_PKG}
+ ./configure CC=cc --with-libevent=/data/ashterenli/spe-tst5/install/libevent-2.1.12-stable --with-hwloc=/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/hwloc-2.12.2 --with-pmix=/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c --prefix=/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/prrte-m0996b79    

[ashterenli]

attached is the full prrte bld log

prrte-bld-log-1308.txt

[rhc54]

Strange - it all looks correct. I checked here and the symbol is present and visible:

00000000001097b8 T _PMIx_Group_operation_string

Check your LD_LIBRARY_PATH and make sure there isn't an older PMIx version sitting in it. I guess in your script, you might want to reset that envar to the correct thing before doing the builds, just to be safe.

[ashterenli]

seems like a libtool problem. I traced the offending command to:

$ libtool --silent --tag=CC   --mode=link cc -levent_core -levent_pthreads -lpmix -lhwloc -O3 -DNDEBUG  -finline-functions   -L/data/ashterenli/spe-tst5/install/libevent-2.1.12-stable/lib -Wl,-rpath -Wl,/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/lib -Wl,--enable-new-dtags -L/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/hwloc-2.12.2/lib -o prted prted.o    ../../../src/libprrte.la -levent_core -levent_pthreads -lpmix -lhwloc
/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `PMIx_Group_operation_string'
/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `PMIx_Value_get_number'
/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `pmix_show_help_add_data'
/usr/bin/ld: ../../../src/.libs/libprrte.so: undefined reference to `PMIx_Info_list_add_value'
icx: error: linker command failed with exit code 1 (use -v to see invocation)

here the pmix path was correct.

However, on checking ../../../src/libprrte.la, it is missing:

# Libraries that this one depends upon.
dependency_libs=' -L/data/ashterenli/spe-tst5/install/libevent-2.1.12-stable/lib -L/data/ashterenli/spe-tst5/install/u_intel/2
025.1.0.666/hwloc-2.12.2/lib -L/usr/local/cuda/lib64/ -L/usr/local/cuda/lib64/stubs/ -ldl -lnl-route-3 -lnl-3 -levent_core -le
vent_pthreads -lpmix /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/hwloc-2.12.2/lib/libhwloc.la -lm -ludev -lOpenCL -
lxml2'

just -lpmix, which it then pics from /usr/lib.

Let me dig a bit deeper.

[rhc54]

If it would be easier, I can provide you with tarballs - either via email (if you send me an address - rhc at pmix dot org) or I can post them here (about 10MB each).

[rhc54]

Okay, email failed to get through, so let's try posting them here (added .txt suffix so Github will accept them)

pmix-7.0.0a1.tar.bz2.txt

prrte-5.0.0a1.tar.bz2.txt

[ashterenli]

Ralph, thank you.

In the meantime I fixed the problem by passing the path to my pmix lib in LDFLAGS to configure.

Something is going wrong at configure stage, maybe my autotools versions are not tested, not sure.
If I just pass --with-pmix=..., the resulting Makefile has:

LDFLAGS =   -Wl,-rpath -Wl,/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/lib -Wl,--enable-new-dtags

which looks wrong, and eventually leads to the above error.

If I explicitly pass LDFLAGS to configure, pointing to my pmix lib, the resulting Makefile looks better:

LDFLAGS = -L/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/lib  -Wl,-rpath -Wl,/data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/pmix-mbb34c7c/lib -Wl,--enable-new-dtags

and I can link prrte without errors.

I'll now build OMPI with these new libs