Run BMO E2E with custom IrSO and Ironic

[lentzi90]

We currently run BMO E2E tests in this repo by cloning BMO, patching a few files, and then running the same script that would be use in BMO.
Ref: https://github.com/metal3-io/ironic-image/blob/main/hack/prepare-bmo-tests.sh.

BMO E2E was designed to be re-usable, but not in this way. The go code is the re-usable part. It can be configured/adapted with environment variables or config files.
The kustomizations for deploying IrSO and Ironic are simply what BMO needs for its own tests. The hack script for preparing and running the tests is also that. It builds BMO, loads the container image, etc. It was never meant to be used outside of BMO.

I think it is a bad idea to couple the ironic-image and baremetal-operator repositories by directly executing scripts or relying on kustomizations from the other repository. We get the same pain that we already saw in metal3-dev-env, where any change easily breaks tests across the org, and we constantly need to coordinate PRs.

Instead, we should simply build the scaffolding needed here. It will look similar but it will have a few important differences.

1. We do not need to build the BMO container image. We are testing ironic-image, just pull the latest BMO image (stable release or main branch).
2. We DO need to build ironic-image.
3. We need slightly different kustomizations. Ironic should use the newly built image. Perhaps a newer version of IrSO?
4. We could optimize what tests to run. Some tests may be of less value for ironic-image functionality than for BMO and could be skipped.

[dtantsur]

Most of https://github.com/metal3-io/baremetal-operator/blob/main/hack/ci-e2e.sh is actually relevant here, and I'm quite strongly against just copying it over. It won't save any problems: we'd have to carefully keep two copies in sync.

Also keep in mind that the e2e job runs on Ironic upstream. Please don't break it.

[lentzi90]

It should not be copied over. It should be adapted to what we need here. The main point is that the script in BMO is not some published artifact that we make promises about. The same goes for the kustomizations. In BMO, we should not need to be careful when changing them. If someone depends on hack/test code that is not our problem.

We cannot have these cross-dependencies between the repos main branches. Everything becomes brittle, hard to understand and reason about. It is far better to "carefully keep two copies in sync". We must be able to update in one place without affecting the other.

If the worry is just about duplication, I think we can solve that in better ways. We are already working on it with the vbmctl tool in a way. We can extract more of the common setup. But the fact remains, BMO hack/test code should not be load bearing in other repos or Ironic upstream.

[Rozzii]

/triage accepted
/kind ci