Discussion about merge test strategy

[tuminoid]

I think we need to take the discussion about merge test strategy up again. We get this race condition merges too often.

Originally posted by @tuminoid in #3093

/cc @lentzi90 @kashifest @Rozzii

[tuminoid]

The issue here being, we currently test the PR as is, not cherry-picked on top of the base branch, which often leads to cases where the codebase or tests are broken, after two non-conflicting but breaking changes land without one of them rebasing on top of the other.

Ie. linter breakages when bumping them or adjusting configs and merging an old PR at the same time.

[Rozzii]

/triage accepted
/kind question ci

[metal3-io-bot]

@Rozzii: The label(s) kind/question cannot be applied, because the repository doesn't have them.

Details

In response to this:

/triage accepted
/kind question ci

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tuminoid]

@lentzi90 PTAL, you had some input on this earlier.

[lentzi90]

Yes, let's start with the situation we have today.

• We use Prow to run some tests and to merge when ready.
• We use github workflows for the heavier BMO and IRSO tests
• We use jenkins jobs, handled by prow on other repositories for the heavier tests
• Prow and github's checkout action both checkout the merge commit before running tests
• Jenkins can be configured to checkout the merge commit, but we do not do this consistently.
• Prow can be configured to always re-run jobs before merge if the branch is behind. We accidentally did something like this in Prow: Require manually triggered jobs #992, but reverted because we did not like the extra time and resources it took.
• Github can be configured to require branches to be up to date before merging. We have seen this being used in Kyverno for example. It would protect us from this issue but at tremendous cost of churn and CI resources.
• Alternatively, Github can be configured to use merge queues. This achieves the same but with less churn. However, it seems incompatible with Prow.

TL;DR: We already test merge commits in many cases. Some Jenkins jobs can be improved in this regard.
The real issue is that we do not re-run tests before merge. The simplest config solution is to require branches to be up to date, but this would be an even worse pain IMO. Merge queues are not an option since we use Prow.
Since we use both workflows and ProwJobs, it is hard/impossible to enforce anything other than "require branch up to date".
Prow or merge queues could both on their own enforce that the jobs/workflows are re-run before merge, but since they don't mix we are out of luck.

For ProwJobs we can at least manually re-run them, but for workflows there aren't any other options than merge queues as far as I can see.

[lentzi90]

Looks like someone has been down this road before and ended up re-creating Prow using github actions: https://github.com/jpmcb/prow-github-actions
Not sure if that is what we want. Maybe a discussion in the community around unifying our CI? If we pick one and stick to it, this would make things easier.