Bulldozer is not auto merging my PRs

[llopes-webera]

Good afternoon, team. Thanks for your help in advance.
I'm figuring out an issue with Bulldozer that stopped to auto-merge my organization PRs around three weeks ago. I noted is kind of some changes by Engineers in the Helm Chart, like:

• Ingress

FROM: apiVersion: extensions/v1beta1
TO: apiVersion: networking.k8s.io/v1
---
FROM: - path: {{ $ingressPath }}
TO: - path: {{ $ingressPath }}
            pathType: ImplementationSpecific

• Deployment

FROM: apiVersion: apps/v1beta2
TO: apiVersion: apps/v1

Also, the specification looks like some old one. Instead of Trigger and Ignore, there are using Whitelist and Blacklist. Is it some older version, please?

version: 1
merge:
  whitelist:
    labels:
    - ready for merge
    - merge
  blacklist:
    labels:
    - do not merge
    - wip
    - testing
  method: squash
  options:
    squash:
      body: pull_request_body
  delete_after_merge: true
update:
  whitelist:
    labels:
    - wip
    - update
    - ready for merge
    - merge
    - ready for review

My apologies for any dumb question, but I'm not much familiar with Bulldozer. Thank you very much for your support.

[bluekeyes]

Hey @llopes-webera, sorry you are having trouble with Bulldozer. It's kind of hard to debug problems like this without access to the environment or additional details about errors that are happening, but here's some information that might help:

• We don't currently support an official Helm chart for Bulldozer, so I'm guessing the changes you found in your Helm chart are necessary to work with your particular Kubernetes deployment. They shouldn't impact the functionality of Bulldozer.
• We changed the configuration fields two years ago in v1.10.0, but the latest version of Bulldozer should still read and process old configuration files. I recommend using the new trigger and ignore fields, but they are not be required.

To debug further, you might want to check:

• What version of Bulldozer are you running? Are you using it against GitHub.com or GitHub Enterprise? If GitHub Enterprise, was there a recent GitHub upgrade? Did the version of Bulldozer change?
• If you have access to the Bulldozer logs, do you see any events for the repositories where merging stopped working? Are there any errors? Bulldozer should have reasonable logs about why it is or is not merging a pull request.
• If there are no events in the logs, is the Bulldozer app still installed on your organization and enabled on the right repositories? If you have access to the GitHub App page in GitHub, are recent webhook deliveries successful or are they failing?

I think the main thing that will help is determining if GitHub webhooks are making it to Bulldozer or not. If they are, the logs should explain what is happening. If they are not, the problem is probably due to configuration in GitHub or your Kubernetes cluster, rather than a problem with Bulldozer.

[llopes-webera]

Perfect @bluekeyes. I so much appreciated the tricks. I found that someone else deleted all the deployment (resources) from bulldozer in our cluster. I'm figuring this out and trying to redeploy the latest docker image version. We are good for now, and I'll let you know about anything else.

Thank you very much for your answer.

[llopes-webera]

That is a log error that I'm getting @bluekeyes.

ERR Update merge failed unexpectedly error="POST https://api.github.com/repos/VibrentHealth/mission-control-scheduler-ui/merges: 409 Merge conflict []\ngithub.com/palantir/bulldozer/bulldozer.UpdatePR\n\t/home/runner/work/bulldozer/bulldozer/bulldozer/update.go:61\ngithub.com/palantir/bulldozer/server/handler.(*Base).UpdatePullRequest\n\t/home/runner/work/bulldozer/bulldozer/server/handler/base.go:107\ngithub.com/palantir/bulldozer/server/handler.(*Push).Handle\n\t/home/runner/work/bulldozer/bulldozer/server/handler/push.go:89\ngithub.com/palantir/go-githubapp/githubapp.Dispatch.Execute\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:55\ngithub.com/palantir/go-githubapp/githubapp.(*scheduler).safeExecute\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:183\ngithub.com/palantir/go-githubapp/githubapp.QueueAsyncScheduler.func1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:257\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.19.3/x64/src/runtime/asm_amd64.s:1594" github_delivery_id=63e699be-ae1a-11ed-9733-91a2630b26c8 github_event_type=push github_installation_id=524133 github_pr_num=2256 github_repository_name=mission-control-scheduler-ui github_repository_owner=VibrentHealth rid=cfn5utdnj6iglgm25qfg

Another one:

DBG Received status error event from VibrentHealth/data-lake-charts github_delivery_id=8741641a-ae1b-11ed-89c1-b4503ff41033 github_event_type=status github_installation_id=524133 github_repository_name=data-lake-charts github_repository_owner=VibrentHealth rid=cfn62nlnj6iglgm25su0

I deployed the latest image from palantirtechnologies/bulldozer. Should I update the specifications for Trigger and Ignore? Apart from deployment, also we have to install the GitHub App?
Thank you very much.

[bluekeyes]

Apart from deployment, also we have to install the GitHub App?

Yes, the GitHub App must be configured to point at your server and must be installed on the repositories where you'd like to use Bulldozer. If this was working before and you restored the application in your K8s cluster using the same URLs and configuration, I expect the GitHub App still works.

If the log messages you shared are recent, that's also good evidence that the app is configured and delivering webhooks to the Bulldozer server.

Both of those log messages seem fine to me:

1. This shows that when Bulldozer tried to merge PR 2256 in the repository, it failed because there was a merge conflict. In this case, GitHub should prevent merging manually as well, until the conflict is resolved.
2. This shows that Bulldozer received a status check event with type error for a repository. This was maybe because a CI build failed or some other check was failing for this PR. Bulldozer listens for (successful) status events so that it knows when required checks have passed and it can try to merge the pull request.

Most of the log messages include the repository name and pull request number in the github_repository_name and github_pr_num fields. If you have a specific PR that you expected Bulldozer to merge and it did not, I recommend searching the logs for that PR number to see what happened.

[llopes-webera]

Pretty perfect. I'm going to check that.
The only thing that looks weird to me is the principal repository we were using bulldozer doesn't get any information on the logs. And there, the .bulldozer.yml was configured and worked before deleting the deployment. I double-checked the GitHub app, and there are set up for "all repositories" as expected. Should I be able to check something specific?

I huge much appreciate your support, @bluekeyes.

[bluekeyes]

If you have access to the GitHub App management UI (see Maintaining GitHub Apps), you can go to the "Advanced" tab to see a record of recent webhook deliveries from GitHub's point of view. This shows events for all repositories, so you'll want to check if immediately after performing some action in your principal repository. Expand the events to see the payload that GitHub sent. This view will show you that (a) GitHub actually generated an event for the repository and (b) if Bulldozer processed the event or returned an error.

One time, I did see GitHub "forget" about the repositories an app was installed on, even though it showed as enabled for all repositories. I was able to fix this by uninstalling the app from the organization and installing it again. You might want to try this if you don't see any events for your repository in the recent webhook delivery view in GitHub.