Immediately failing PR checks

[roaks3]

Background

The evil build killer will (nearly) immediately cancel PR checks for PRs that make changes to the .ci directory. This is to stop nefarious behavior. However, we currently keep Terraform config for our test environments in that directory, and some contributors include changes to that config along with their feature. The result is a confusing problem where they cannot get PR checks to run.

Potential solution

It would be nice to have a check, probably outside of the evil build killer, that scans for changes to the .ci directory, and somehow alerts the user that they cannot make changes to that directory. This would at least tell the user why checks aren't running, and how they can resolve.

[melinath]

Could we move the terraform configuration out of that directory? We will never apply the terraform config as part of a PR, and definitely not without approval, so it should be safe to keep elsewhere.

[roaks3]

We can, but that does not solve the issue for other files in the .ci directory. There could also be other things added to that directory in the future, and contributors would most likely not be aware of this behavior.

I think the behavior being raised here (a PR check inexplicably failing after a few seconds) should be improved either way, and separately, we can consider improvements to how Terraform config changes are made. Moving those files out of the .ci directory seems fine, or maybe we could just omit .ci/infra/terraform from triggering the evil build killer.

[melinath]

We should ideally have this show up as a failed check rather than a comment.