branson+rocm `fatal error: 'mpi.h' file not found`

[michaelmckinsey1]

https://lc.llnl.gov/gitlab/benchpark/benchpark/-/jobs/3478454

We noticed this fails

benchpark experiment init --dest=branson tuolumne branson +rocm

while this works (does not incur the build failure linked above)

benchpark experiment init --dest=branson2 tuolumne branson +rocm caliper=time,mpi

The only difference is whether we also add caliper, but the same MPI is used in both cases.

[michaelmckinsey1]

@scheibelp We wanted to know whether this build failure looks like it could be a spack issue (this can be addressed next week).

[scheibelp]

In some sense this could be attributed to spack, because the spack wrapper is bypassed for this build and re-establishing that would -I<the-mpi-dir>. It's interesting that adding the caliper dependency makes it succeed: it could be that the build logic (CMake, for Branson) collects mpi via CMake config files in the caliper install (and thereby adds a -I pointing to the MPI headers).