Ensure transitive dependencies for headers and libraries are always available

[ocaisa]

This addresses #5124 but is quite a major change even if just a bugfix. I believe that this was probably the original intention when introducing the related options.

Requires additional tests (and probably debating) so leaving as a draft PR for now.

[ocaisa]

This may seem quite aggressive, but it does seem to work in the cases I tried.

[ocaisa]

I think this is good enough for a review, it passes existing CI and also fixes #5143

[ocaisa]

This looks odd, but the methods available here are limited. My subdirs are full paths relative to ''

[akesandgren]

Can't we just clear the relevant env variables at start of eb and then just load modules and "all will be fine"?

[ocaisa]

Sure, but that is not what we say we support with the option --search-path-cpp-headers. @lexming can correct me, but I thought the intention there was to exclusively use the indicated approach

[lexming]

When I worked on --search-path-cpp-headers I didn't change the behaviour of EB at build time in that regard. That option only introduces the possibility to set how those paths are injected in the build environment.

The default behaviour of EB (which has not changed in ages AFAIK):

• manually inject search paths in direct dependencies with compiler/linker flags, so include folders are always passed with -I flags independently of what modules set
• the build environment is not reset, any environment variables like CPATH and LIBRARY_PATH will be present in the build environment

Now, I have not yet looked in detail the code in this PR, but it is indeed probably the case that setting --search-path-cpp-headers to an environment variable (instead of flags) is resetting that variable instead of prepending to it.