Spack environments in uberenv: handling of compiler flags

[adrienbernede]

I am trying to switch to Uberenv with Spack environments.
I made the necessary changes in Radiuss Spack Configs, and for the most part, it works seamlessly.

However, there is one corner case I would like to share with you, to see if you know something about it:

I am getting the following error with one of my specs:

==> Added repo with namespace 'llnl.radiuss'.
[adding spack package]
[exe: /dev/shm/lassen646-1247741/spack/bin/spack -D /dev/shm/lassen646-1247741/spack_env add 'umpire@develop~shared +asan +sanitizer_tests +tools tests=basic %clang@12.0.1.gcc.8.3.1 cxxflags==-fsanitize=address']
==> Adding umpire@develop%clang@12.0.1.gcc.8.3.1 cxxflags=="-fsanitize=address" +asan+sanitizer_tests~shared+tools tests=basic to environment /dev/shm/lassen646-1247741/spack_env
[calling spack develop]
[exe: /dev/shm/lassen646-1247741/spack/bin/spack -D /dev/shm/lassen646-1247741/spack_env develop --no-clone --path=/usr/WS2/umdev/gitlab-runner/umdev/builds/T5vctq-D/011/gitlab/radiuss/Umpire 'umpire@develop~shared +asan +sanitizer_tests +tools tests=basic %clang@12.0.1.gcc.8.3.1 cxxflags==-fsanitize=address']
==> Configuring spec umpire@develop%clang@12.0.1.gcc.8.3.1 cxxflags=="-fsanitize=address" +asan+sanitizer_tests~shared+tools tests=basic for development at path /usr/WS2/umdev/gitlab-runner/umdev/builds/T5vctq-D/011/gitlab/radiuss/Umpire
[concretizing spack env]
[exe: /dev/shm/lassen646-1247741/spack/bin/spack -D /dev/shm/lassen646-1247741/spack_env concretize --fresh ]
==> Error: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 -fsanitize=address does not satisfy -fsanitize=address

Here is my interpretation:
The compiler I ask for is defined with cxxflags: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 flags, and in my spec, I ask for cxxflags==-fsanitize=address.
Before switching to environments, both would combine. The consequence being that I could define both generic flags in the compiler definition, and specific ones in the spec.
With environments it looks like I can’t do the same.

Is this a known limitation? Is there a work around?

I found this issue with spack@e4s-23.02, I can see it with spack@v0.20.0 too.

[adrienbernede]

@chapman39 ever seen this before ?

[chapman39]

i've never seen this. so spack doesn't like that theres in inconsistency between spack.yaml cxxflags and the one you put in the spec?

serac gets around it by appending -fsanitize=address in the spack recipe, if +asan. an example in custom mfem package: https://github.com/LLNL/serac/blob/develop/scripts/spack/packages/mfem/package.py#L484-L493

since your compiler is clang@12.0.1.gcc.8.3.1, is it possible you do not need to include gcc-toolchain explicitly?

[adrienbernede]

Thank you for your answer.

I think I understand how Serac (and samrai and mfem) is not affected by this. At least I can see the difference.
However, this won’t work as a workaround for me, since the problem is not only with asan, but other specs in which I try to specify flags in the spec.

I’ll post this in spack community slack.

Note: I am not including the gcc-toolchain explicitly in the spec precisely because I defined it in the compiler, like so

- compiler:
    spec: clang@12.0.1.ibm.gcc.8.3.1
    paths:
      cc: /usr/tce/packages/clang/clang-ibm-12.0.1/bin/clang
      cxx: /usr/tce/packages/clang/clang-ibm-12.0.1/bin/clang++
      fc: /usr/tce/packages/xl/xl-2021.09.22/bin/xlf2003_r
      f77: /usr/tce/packages/xl/xl-2021.09.22/bin/xlf_r
    flags:
      cflags: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1
      cxxflags: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1
      fflags: --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1
    operating_system: rhel7
    target: ppc64le
    modules: []
    environment: {}
    extra_rpaths: []

That’s the only place where the gcc-toolchain is made explicit.