Cannot install packages from spack cache

[daviddavo]

I cant find packages from the build cache, and some packages like llvm take hours to compile

[root@slurmctld data]# spack mirror ls
spack-public [s ] https://mirror.spack.io
[root@slurmctld data]# spack buildcache update-index spack-public
==> Warning: Encountered problem listing packages at https://mirror.spack.io: 'NoneType' object is not iterable
==> Error: Unable to generate package index: Failed to get list of entries from https://mirror.spack.io
[root@slurmctld data]# spack buildcache list --allarch
==> 0 cached builds.

[giovtorres]

Thank you for reporting. There's a couple things going on. First, I don't think this will work: spack buildcache update-index spack-public. I believe that is for indexes that you own, as it tries to write back to the index, so you would get permission denied on a public index. I could be wrong, as I'm not terribly familiar with spack, but it checks out.

I do not have binary caches pre-installed, which explains why llvm takes forever to build, as it is building from source. I'm working on a fix for this.

[daviddavo]

Oh, I see spack was added a few weeks ago, makes sense then that is not fully ready yet.

I never used spack before, so I was confusing terms. I managed to add the correct mirror myself after build, with make shell. But there are no binaries for this architecture in the cache.

[root@slurmctld data]# spack mirror add spack-bin-develop https://binaries.spack.io/develop
[root@slurmctld data]# spack buildcache keys --install --trust
==> Fetching https://binaries.spack.io/develop/blobs/sha256/f6/f6ea4378846cd46ab23fc727c20afc772f03c83a8b3b57b275f511f196d6b9d9
    [100%]   56.00  B @  111.2 KB/s
==> Fetching https://binaries.spack.io/develop/blobs/sha256/e6/e6b951d7df78889b27537f245504b97a6d211a2ae023fe9922a4beb51238a35c
    [100%]   10.43 KB @   17.0 MB/s
remote: Enumerating objects: 19701, done.
remote: Counting objects: 100% (19701/19701), done.
remote: Compressing objects: 100% (10554/10554), done.
remote: Total 19701 (delta 1336), reused 13773 (delta 1223), pack-reused 0 (from 0)
gpg: key A8E0CA3C1C2ADA2F: 7 signatures not checked due to missing keys
gpg: key A8E0CA3C1C2ADA2F: public key "Spack Project Official Binaries <maintainers@spack.io>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: no ultimately trusted keys found
gpg: inserting ownertrust of 6
[root@slurmctld data]# spack arch
linux-rocky9-skylake
[root@slurmctld data]# spack buildcache list
==> Fetching https://binaries.spack.io/develop/blobs/sha256/c0/c038de3923786af7e002a22782394b3ca24ab7dacfbd3e2c08d8afb47a31ca43
    [100%]  108.54 MB @   31.0 MB/s
==> 0 cached builds.
==> You can query all available architectures with:
  spack buildcache list --allarch
[root@slurmctld data]# spack buildcache list --allarch openblas
==> 36 cached builds.
-- darwin-tahoe-m1 / %c,cxx=apple-clang@17.0.0 ------------------
openblas@0.3.32

-- darwin-tahoe-m1 / %c,cxx=apple-clang@21.0.0 ------------------
openblas@0.3.32

-- linux-ubuntu18.04-x86_64_v3 / %c,cxx,fortran=gcc@7.5.0 -------
openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu20.04-x86_64_v3 / %c,cxx,fortran=gcc@11.1.0 ------
openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu22.04-neoverse_v2 / %c,cxx,fortran=gcc@11.4.0 ----
openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx,fortran=gcc@11.4.0 ------
openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx,fortran=gcc@12.5.0 ------
openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu24.04-aarch64 / %c,cxx,fortran=gcc@13.2.0 --------
openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu24.04-aarch64 / %c,cxx,fortran=gcc@13.3.0 --------
openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu24.04-x86_64_v3 / %c,cxx,fortran=gcc@13.2.0 ------
openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu24.04-x86_64_v3 / %c,cxx,fortran=gcc@13.3.0 ------
openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32

-- linux-ubuntu24.04-x86_64_v3 / %c,cxx,fortran=oneapi@2025.3.2 -
openblas@0.3.32  openblas@0.3.32  openblas@0.3.32  openblas@0.3.32
[root@slurmctld data]# spack install --cache-only openblas
==> Compilers have been configured automatically from PATH inspection
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v2.2/blobs/sha256:9cf62674ef82cfa5cbb156bae59f369346da800713d54336956371c80e71c6c3
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v2.2/blobs/sha256:cc238348fb1a3b70351d784ec61f7843d0cc9ca5de745ce98e6bea7cd00787ea
==> Installing "clingo-bootstrap@=spack~apps~docs+ipo+optimized+python+static_libstdcpp build_system=cmake build_type=Release commit=2a025667090d71b2c9dce60fe924feb6bde8f667 generator=make patches:=bebb819,ec99431 platform=linux os=centos7 target=x86_64" from a buildcache
[+] /usr (external glibc-2.34-ppxsf4c5ax7naeut7rm3c2gcwjll2gqi)
==> Installing gcc-runtime-11.5.0-z6pd3fu6tyxvifp5wcof5h5ovvk2xvqd [2/3]
==> Warning: Skipping build of openblas-0.3.30-6hvewr5r2h6nekilx5gsmqq653k7b3fp since gcc-runtime-11.5.0-z6pd3fu6tyxvifp5wcof5h5ovvk2xvqd failed
==> Error: Failed to install gcc-runtime due to InstallError: No binary found when cache-only was specified
==> Error: openblas-0.3.30-6hvewr5r2h6nekilx5gsmqq653k7b3fp: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

Perhaps the image should be based on ubuntu instead of rocky linux

[daviddavo]

I found another build cache called E4S: https://e4s.io/e4s-buildcache/

I installed it and they have packages with rocky9, but they still fail to install because they don't show up when I use spack buildcache list...

[root@slurmctld data]# spack mirror list
spack-public [s ] https://mirror.spack.io
[root@slurmctld data]# spack mirror list
spack-public [s ] https://mirror.spack.io
[root@slurmctld data]# export E4S_REL=25.11
export E4S_MIRROR_URL="https://cache.e4s.io/${E4S_REL}"
[root@slurmctld data]# spack mirror add e4s-${E4S_REL} "${E4S_MIRROR_URL}"
[root@slurmctld data]# spack mirror list
e4s-25.11    [sb] https://cache.e4s.io/25.11
spack-public [s ] https://mirror.spack.io
[root@slurmctld data]# spack buildcache keys --install --trust
==> Fetching https://cache.e4s.io/25.11/blobs/sha256/25/25c19365fe733ae0b99c3fe1676cf2c7dfb0f7e59bd53a9b5f7d8034952140fd
    [100%]   56.00  B @   95.7 KB/s
==> Fetching https://cache.e4s.io/25.11/blobs/sha256/92/920da5133af9bd45ed57f5a7f0a2ac66c1b9580d3a8f1482fbffb2a8d291e0d7
    [100%]    3.13 KB @    4.8 MB/s
gpg: key 4345F04B40005581: public key "University of Oregon - E4S" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: inserting ownertrust of 6
[root@slurmctld data]# spack buildcache list
==> 0 cached builds.
==> You can query all available architectures with:
  spack buildcache list --allarch
[root@slurmctld data]# spack buildcache list --allarch openblas
==> 14 cached builds.
-- linux-rocky9-aarch64 / %c,cxx,fortran=gcc@13.3.1 -------------
openblas@0.3.30  openblas@0.3.30

-- linux-rocky9-x86_64_v3 / %c,cxx,fortran=gcc@13.3.1 -----------
openblas@0.3.30  openblas@0.3.30  openblas@0.3.30  openblas@0.3.30

-- linux-ubuntu20.04-ppc64le / %c,cxx,fortran=gcc@9.4.0 ---------
openblas@0.3.30  openblas@0.3.30

-- linux-ubuntu24.04-aarch64 / %c,cxx,fortran=gcc@13.3.0 --------
openblas@0.3.30  openblas@0.3.30  openblas@0.3.30  openblas@0.3.30

-- linux-ubuntu24.04-x86_64_v3 / %c,cxx,fortran=gcc@13.3.0 ------
openblas@0.3.30  openblas@0.3.30
[root@slurmctld data]# spack arch
linux-rocky9-skylake
[root@slurmctld data]# spack install --cache-only openblas
==> Compilers have been configured automatically from PATH inspection
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v2.2/blobs/sha256:9cf62674ef82cfa5cbb156bae59f369346da800713d54336956371c80e71c6c3
==> Fetching https://ghcr.io/v2/spack/bootstrap-buildcache-v2.2/blobs/sha256:cc238348fb1a3b70351d784ec61f7843d0cc9ca5de745ce98e6bea7cd00787ea
==> Installing "clingo-bootstrap@=spack~apps~docs+ipo+optimized+python+static_libstdcpp build_system=cmake build_type=Release commit=2a025667090d71b2c9dce60fe924feb6bde8f667 generator=make patches:=bebb819,ec99431 platform=linux os=centos7 target=x86_64" from a buildcache
[+] /usr (external glibc-2.34-ppxsf4c5ax7naeut7rm3c2gcwjll2gqi)
==> Installing gcc-runtime-11.5.0-z6pd3fu6tyxvifp5wcof5h5ovvk2xvqd [2/3]
==> Warning: Skipping build of openblas-0.3.30-6hvewr5r2h6nekilx5gsmqq653k7b3fp since gcc-runtime-11.5.0-z6pd3fu6tyxvifp5wcof5h5ovvk2xvqd failed
==> Error: Failed to install gcc-runtime due to InstallError: No binary found when cache-only was specified
==> Error: openblas-0.3.30-6hvewr5r2h6nekilx5gsmqq653k7b3fp: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

Perhaps its because this has "skylake" architecture but it expects "x86_64_v3"? When I try to force the target I get:

[root@slurmctld data]# spack install --cache-only openblas target=x86_64_v3
==> Error: failed to concretize `openblas target=x86_64_v3` for the following reasons:
     1. Cannot select a single "node_target" for package "openblas"

[giovtorres]

Yeah, i’ve been experimenting with E4S. The default mirror that comes with the spack installation points to a source mirror, so all available packages are built from source. The default installation doesn’t seem to come with any binary caches pre-installed.

Spack is installed and available for users to configure any source or binary mirrors, which you are trying to do.

The config in this repo sets the spack arch target automatically to x86_64. I’m not sure if that is the correct thing to do or not. (I don’t use spack much myself) Change the target in /usr/local/spack/etc/spack/packages.yaml to x86_64_v3 to avoid that ‘node_target’ error above.

I was able to install llvm from binary. I did have some juggling to do to get the correct compiler. I think compiler versions have to match or it will attempt to build from source.

When i get something fully working, i can share it here.