ADIOS Spack package broken on Summit

[NAThompson]

To reproduce:

[nthompson@summitdev-login1]~% ml pgi/19.9
[nthompson@summitdev-login1]~% spack install adios
...
==> Installing openmpi
==> No binary for openmpi found: installing from source
==> openmpi: Executing phase: 'autoreconf'
==> openmpi: Executing phase: 'configure'
==> openmpi: Executing phase: 'build'
==> openmpi: Executing phase: 'install'
[+] /autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/openmpi-3.1.6-z36ridnac6bgsinx677u5oooviyrpnd4
==> Installing adios
==> No binary for adios found: installing from source
==> adios: Executing phase: 'autoreconf'
==> adios: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    '/tmp/nthompson/spack-stage/spack-stage-adios-1.13.1-xchp66cjc3oafb2g67pdumtcwbcbqolk/spack-src/configure' '--prefix=/autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/adios-1.13.1-xchp66cjc3oafb2g67pdumtcwbcbqolk' 'CFLAGS=-fPIC' '--enable-shared' '--disable-fortran' '--with-mpi=/autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/openmpi-3.1.6-z36ridnac6bgsinx677u5oooviyrpnd4' '--without-infiniband' '--with-blosc=/autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/c-blosc-1.17.0-efj77ikklysbx5fhah5tajm7uqaeetcr' '--with-zlib=/autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/zlib-1.2.11-tjbynt2kcsdo5ehwlcrxoqhbb7snlqdm' '--without-bzip2' '--without-szip' '--with-zfp=/autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/zfp-0.5.5-oyzh7mkqqnjbwg2dlwxavejrsaeblyty' '--with-sz=/autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/sz-1.4.12.3-apzgwhhqybemdkv6sjoeaozdsjr3blor' '--with-lz4=/autofs/nccs-svm1_home1/nthompson/spack/opt/spack/linux-rhel7-power8le/gcc-4.8.5/lz4-1.9.2-tpkrjicqiew4wkc5t7f6hzibyf4bvm5s' '--without-phdf5' '--without-netcdf' '--without-flexpath' '--without-dataspaces'

1 error found in build log:
     47    checking whether /autofs/nccs-svm1_home1/nthompson/spack/lib/spack/e
           nv/gcc/g++ accepts -g... yes
     48    checking dependency style of /autofs/nccs-svm1_home1/nthompson/spack
           /lib/spack/env/gcc/g++... gcc3
     49    checking for mpicc... /autofs/nccs-svm1_home1/nthompson/spack/opt/sp
           ack/linux-rhel7-power8le/gcc-4.8.5/openmpi-3.1.6-z36ridnac6bgsinx677
           u5oooviyrpnd4/bin/mpicc
     50    checking for MPI_Init... no
     51    checking for MPI_Init in -lmpi... no
     52    checking for MPI_Init in -lmpich... no
  >> 53    configure: error: could not find mpi library for C

See build log for details:
  /tmp/nthompson/spack-stage/spack-stage-adios-1.13.1-xchp66cjc3oafb2g67pdumtcwbcbqolk/spack-build-out.txt

I would attach the build log, but it gives the same information as the console.

[pnorbert]

I am not sure if you are supposed to build your own MPI compiler on summit.
Can you build adios if you force spack to use the system's MPI compiler?

[NAThompson]

@pnorbert : I'm confused; I didn't tell spack to build openmpi; in addition I had an MPI loaded:

ml

Currently Loaded Modules:
  1) hsi/5.0.2.p5   2) xalt/1.1.3   3) lsf-tools/2.0   4) DefApps   5) pgi/19.9   6) spectrum-mpi/10.3.0.1-20190611

Do you mean that I should add a compiler to spack?

Here's what I currently have available:

spack compilers
==> Available compilers
-- gcc rhel7-ppc64le --------------------------------------------
gcc@4.8.5

-- pgi rhel7-ppc64le --------------------------------------------
pgi@19.9

[pnorbert]

You need to point spack to a system MPI and disable building its own. By default ALL dependencies are built from scratch. You only pointed spack to the system compiler.

Here is my setting on Summit (not summitdev):

$ cat /ccs/proj/csc303/adios2-test-suite/summit/spack/etc/spack/packages.yaml
packages:

  # concretization preferences
  all:
    compiler: [gcc/8.1.1]
    providers:
      mpi: [spectrum-mpi]

  # system modules externals. 
  spectrum-mpi:
    modules:
      spectrum-mpi@10.3.1.2-20200121%gcc@8.1.1: spectrum-mpi/10.3.1.2-20200121
    buildable: False
 ...

[NAThompson]

@pnorbert : I get the following when I tried your suggestion:

==> Warning: the attribute "modules" in the "packages" section of the configuration has been deprecated [entry=CommentedMap([('spectrum-mpi@10.3.0.1-20190611%pgi@19.9', 'spectrum-mpi/10.3.0.1-20190611')])]

However, with the following packages.yaml I was able to succeed in getting a successful compilation:

[nthompson@summitdev-login1]~% cat ~/.spack/packages.yaml
packages:
    all:
        compiler: [gcc/4.8.5]
        providers:
            mpi: [spectrum-mpi]
    spectrum-mpi:
        modules:
            spectrum-mpi@10.3.0.1-20190611%gcc@4.8.5: spectrum-mpi/10.3.0.1-20190611
        buildable: False

Strangely, the spack package for adios2 succeeds without this file, even though I was unable to see any significant differences in the spack info log.

[williamfgc]

@NAThompson system MPI libraries are required, as @pnorbert suggested, as your package might get out of sync when they do system updates on Summit.