testsuite: gitlab ci cluster specific tests

[wihobbs]

As @grondo and @garlick pointed out in flux-framework/flux-core#5482, there are some tests we'd like to run in our CI on real clusters in addition to the normal testsuite. Some ideas (opening this issue with the hope others will add to this list):

• Checking that flux uri resolve slurm:jobid is able to properly resolve a Flux URI
• Flux under Slurm works
• Combinatorially testing MPI implementations on our clusters (this should probably be a whole issue by itself)

[chu11]

although lower priority than the above, would we want to test with different python versions that are not the standard installed python? To make sure all the paths and what not are setup correctly? i.e. user uses python 3.X from /usr/tce and we make sure that works with the python 3.X from https://github.com/flux-framework/flux-python?? I think all Python we test in CI is the standard install python. (Edit: and perhaps this also double checks that all python dependencies are installed for each version? )

Note, I'm not sure the flux-python from above is installed yet ... so that maybe be another thing.

[wihobbs]

Snippet of @grondo and my weekly mtg:

1. We're going to set up a new repository for flux-framework testing only. Iterating on core for integration tests might get cumbersome, and we want to test other framework projects.
2. We'll run a nightly test of flux-core (do what was done in testsuite: run entire suite on LC hardware flux-core#5429) and some other stuff, too, that will be discussed in the Issues section of that new repo.
3. There will be a Mattermost channel where the status of the nightly pipelines is reported.