CLOUDP-420746: automate RedHat operator bundle publishing by josvazg · Pull Request #1365 · mongodb/mongodb-kubernetes

josvazg · 2026-07-09T13:27:06Z

Summary

Add scripts/release/publish-openshift-bundles-to-rh.sh:

Downloads the certified OLM bundle from S3
Prepares/pushes a branch on the mongodb-forks certified-operators and community-operators repos
Prints the upstream PR links (PR creation stays manual).

Wire it into Evergreen via a new publish_openshift_bundles_to_rh func, task, and variant using github.generate_token.

Proof of Work

CI must pass.

✅ Local run passes

% KEEP_WORKDIR=true VERSION=1.8.1 RH_DRYRUN=true scripts/release/publish-openshift-bundles-to-rh.sh
...
Click the links below to open the upstream PR (mark it as a DRAFT), then ask the team
for a quick look; certified merges on green CI, community may need a close/re-open:
  https://github.com/redhat-openshift-ecosystem/certified-operators/compare/main...mongodb-forks:certified-operators:mongodb-kubernetes-1.8.1?expand=1
  https://github.com/k8s-operatorhub/community-operators/compare/main...mongodb-forks:community-operators:mongodb-kubernetes-1.8.1?expand=1
Workdir kept for inspection: /var/folders/8g/h28ztw4j64x4msqcmg4p9t0h0000gp/T/tmp.GbC0n9rGUb

Checklist

Have you added changelog file?
- use skip-changelog label if not needed

github-actions · 2026-07-09T13:28:25Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.10.0 Release Notes

New Features

MongoDBOpsManager, AppDB: The mongodb-agent-monitoring sidecar container has been merged into the main
mongodb-agent container. Both automation and monitoring now run as a single process in a single container. The
spec.appDB.monitoringAgent field is now deprecated and has no effect; Monitoring Agent options for the AppDB —
including log level and rotation — are configured via spec.applicationDatabase.agent and
spec.applicationDatabase.agent.monitoringAgent, the same fields used by MongoDB resources. Enabling AppDB
monitoring still results in rolling restarts of AppDB pods.
MongoDBSearch: Added support to show cluster level status of Search, Managed LoadBalancer and MetricsForwarder in the MongoDBSearch resource's status.clusters field.
MongoDBSearch: Added support to configure the node affinity of the MongoDB Search (mongot) pods using the MongoDBSearch CR fields spec.clusters[].nodeAffinity or spec.clusters[].shardOverrides[].nodeAffinity.

Bug Fixes

MongoDBUser: Fixed a bug where the connection string Secret created in the central cluster did not carry a controller owner reference to the MongoDBUser CR. The missing reference prevented Kubernetes garbage collection from cleaning up the Secret when the MongoDBUser was deleted.
MongoDBUser: Fixed a bug where the ownership guard on the connection string Secret was always satisfied regardless of the actual owner, allowing the operator to silently overwrite a Secret controlled by a different resource.
Fixed a bug where all MongoDBCommunity deployment telemetry rows incorrectly reported IsRunningEnterpriseImage = true. The field was being evaluated against the operator-level enterprise image rather than the image configured in each CR's spec, causing a misleading ~100% enterprise rate for the Community deployment type.
IsRunningEnterpriseImage for Community deployments is now derived from the mongod container image override in spec.statefulSet.spec.template.spec.containers, if present. When no override is set, the field correctly defaults to false, reflecting that Community CRs use the community MongoDB server image by default.
MongoDBOpsManager, AppDB: Fix for spec.applicationDatabase.agent.monitoringAgent.logRotate field not being
handled properly. Additionally, added defaults for
spec.applicationDatabase.agent.monitoringAgent.logRotate.sizeThresholdMB to 1000 and
spec.applicationDatabase.agent.monitoringAgent.logRotate.timeThresholdHrs to 24, which are same defaults Ops Manager
would set.
MongoDBSearch: The default JVM heap size (half of the memory request) is now capped at 30GB, following the mongot sizing guidance. Heap sizes above ~30GB prevent the JVM from using compressed object pointers and degrade performance. User-provided heap flags are not affected, if more than 30GB heap is required, we recommend using jvmFlags.
MongoDB: Fixed a bug where switching a MongoDB ReplicaSet or ShardedCluster to a new Ops Manager project could cause the automation agent to generate a random keyfile in the empty target project, breaking internal cluster authentication.

viveksinghggits

Looks pretty good.

viveksinghggits · 2026-07-16T15:01:50Z

+# It downloads the unified certified bundle tarball (produced and uploaded by the
+# prepare_and_upload_openshift_bundles Evergreen task) and, for each RedHat fork,


the bundle is not already certified right? or is it?

What do you lean exactly?
Just to double check, we create both PRs to redHat from a single tarball, the certified one. That is what I did for 1.8.1 AFAICR and it worked.

What I wanted to highlight is, is the bundle that is uploaded to AWS bucket by our evergreen pipeline already certified? If not then the comment is wrong and we should change that.

I believe Red Hat's certification is happening when we open a PR against their certified operator repo. I suspect "certified bundle tarball" here means something else.

I would change wording to avoid confusion like this ^

This all comes from the wording on the current release document:

OLM bundles and RedHat Pull Requests
Download created the unified OLM bundle from S3 and open a PR:
Download the bundle (it’s named mck-operator-certified-x.y.z.tgz, untar it) from S3 link that is logged in the prepare_and_upload_openshift_bundles evg task from release.
We have to create PRs for our OLM bundle in the repositories https://github.com/k8s-operatorhub/community-operators and https://github.com/redhat-openshift-ecosystem/certified-operators both of these repositories are forked already at the https://github.com/mongodb-forks/community-operators and https://github.com/mongodb-forks/certified-operators respectively.

AFAIU we create a single unified bundle tarball, but we call it certified. I have no problem changing the wording here, but the terms were confusing beforehand. I was also surprised the same tarball works in both repos as is, but yeah, that is the process we seem to follow and I completed for 1.8.1.

As for whether it was certified or not, I think the same as @m1kola here, the repo in RedHat's side is called certified-operators, I assume that they are considered certified once they are merged there, never before.

To be clear, I will be changing wording here, but will not rename the tarball which predates this change.

viveksinghggits · 2026-07-16T15:02:55Z

+# It intentionally does NOT open the pull requests: it prints the "create PR" URLs and
+# the manual review checklist so a human can raise the DRAFT PRs and drive the review.


Why do we not open the PR? Are we not able to using the app account that we have or there are any other reasons?

It was a guardrail to avoid unwanted PRs when testing or by mistake. I can reconsider though, as we could draft the PR instead, without actually putting it for review. That would solve the the issue that the evergreen job does not have a good way to give the PR opening links, other that checking the build logs.

Actually I do not see a proper way to drop the releaser into the "open PR page", because I do not really want to create PRs in RH without human users being aware of them. That has to be a conscious decision and tests or mistakes could pollute RedHat without us noticing until it is too late.

because I do not really want to create PRs in RH without human users being aware of them.

If someone is starting the release, they know that two PRs are going to be raised for OLM bundles as part of the release process.

That has to be a conscious decision and tests or mistakes could pollute RedHat without us noticing until it is too late

Is this really possible? This workflow would ONLY be executed once the release is triggered, is there any other way to execute this workflow.

I think we should just create the PR in draft because that's what we are recommended to do in our release guide. The PRs would not even be in ready for review, once the PRs are raised in draft status we can ask for reviews and then we can make them ready for review manually.

that the evergreen job does not have a good way to give the PR opening links, other that checking the build logs

I am not sure what exactly you meant here but if you mean you are not easily able to create the draft PR from evergreen, we do that already for the helm release step. To release helm we create a PR to the helm-chart repo automatically.

+1 for a draft PR

If someone is starting the release, they know that two PRs are going to be raised for OLM bundles as part of the release process.

The release process is done by evg in the background, and we will also be testing it. It is easy to trigger by mistake, by a test or forget about cleaning up.

I think we should just create the PR in draft because that's what we are recommended to do in our release guide. The PRs would not even be in ready for review, once the PRs are raised in draft status we can ask for reviews and then we can make them ready for review manually.

I was almost convinced to go the PR draft way, until I checked with gemini and confirmed that even a PR draft kicks things on the RedHat side. We should not be doing that:

Tekton/Prow CI Pipelines Will Still Trigger
Red Hat’s automated certification and community pipelines (operator-pipelines) listen to GitHub webhook events. When a PR is created on their repos—even as a draft—it triggers webhook events that kick off their Tekton/Prow validation pipelines, run bundle checks, and reserve test cluster resources.

It Hits Red Hat’s "One Open PR" Limit
Red Hat enforces a strict rule across certified-operators, community-operators-prod, and related repos: you may only have one open pull request at a time per package/project. A draft PR counts as an open PR. If your automation opens a draft PR and it sits waiting for human approval, your release pipeline is locked. If another automated build runs in the meantime, it will fail outright because an open PR already exists.

Upstream Noise and Artifact Pollution
Creating a draft PR publishes a public PR under the redhat-openshift-ecosystem organization. If your internal human review decides not to release or needs to regenerate the bundle from scratch, you have to close or abandon that draft PR, leaving unnecessary clutter and failed pipeline records on Red Hat’s upstream repositories.

I do not think we need to care for point 2, that is on us. But points 1 and 3 makes us a noisy contributor and we do not want that.

I am not sure what exactly you meant here but if you mean you are not easily able to create the draft PR from evergreen, we do that already for the helm release step. To release helm we create a PR to the helm-chart repo automatically.

I mean evergreen is a job runner, it will create the PR, but it has no interactive way (AFAIK) to tell you "here are the links"... which is precisely the same limitation to the DX for just the links to create the PRs instead.

I think we should leverage the already interactive release workflow in GitHub to "show the create PR links" once evergreen is done creating the branches required. That makes that part of the process totally synchronous and manual for the releaser, which usually only means approve in a couple of clicks per PR.

Even if we are worried about CI getting called for testing, I think we won't need more than 1 PR to test whether we are able to open the PR in that repo or not and I am pretty sure that 1 PR is not going to be a problem to anyone and wouldn't make un noisy contributor 😄. And after that one PR we know that the PR will only be raised for releases.

it will create the PR, but it has no interactive way (AFAIK) to tell you "here are the links"... which is precisely the same limitation to the DX for just the links to create the PRs instead.

We can log the link in the evergreen patch and that link can be clicked on. An example from the PR that we create to release the chart

So I am not really sure what you meant by interactive way.

But it's ok, I think you are inclined towards just showing the link that can be used to open the PR, which is fine as well, it's just that if we are automating the things I didn't see a downside completely automating this one as well.

viveksinghggits · 2026-07-16T15:09:01Z

are we using these Go files by the script publish-openshift-bundles-to-rh.sh?

we don't seem to be using this, so where are they used exactly?

Good catch! The publishing code requires a GitHub App token, evergreen only allows to wire one app by project, and MCK already had one set up. This new app was created adhoc for releases. The go code here takes a github app credential and mints a session token to use a GH_TOKEN before the release process can proceed.

This was not surfaced on local tests as I was minting the token manually and I had not managed to test this in CI as there the tests were blocked by the preflight failing, as I was testing an old image that broke the preflight test.

All this to say you catched a bug here. I need to make sure CI wires the GH_TOKEN minting before calling scripts/release/publish-openshift-bundles-to-rh.sh

will fix

This is now fixed

m1kola · 2026-07-20T12:36:03Z

+    tags: [ "release" ]
+    run_on:
+      - ubuntu2404-small
+    allowed_requesters: [ "patch", "github_tag" ]


Do we need it on patch?

maybe not, will double check

thanks for the heads up

You were right, seems better to remove. We should not kick this off manually

Signed-off-by: Jose Vazquez <jose.vazquez@mongodb.com>

Add scripts/release/publish-openshift-bundles-to-rh.sh: - Downloads the certified OLM bundle from S3 - Prepares/pushes a branch on the mongodb-forks certified-operators and community-operators repos - Prints the upstream PR links (PR creation stays manual). Wired into Evergreen via a new publish_openshift_bundles_to_rh func, task, and variant using a GH_TOKEN minted by a new mckci github option. Signed-off-by: Jose Vazquez <jose.vazquez@mongodb.com>

josvazg requested a review from a team as a code owner July 9, 2026 13:27

josvazg requested review from fealebenpae and lsierant July 9, 2026 13:27

josvazg marked this pull request as draft July 9, 2026 13:27

josvazg added the skip-changelog Use this label in Pull Request to not require new changelog entry file label Jul 9, 2026

josvazg force-pushed the CLOUDP-420746/publish-to-rh branch 3 times, most recently from 54e9a62 to cb773cf Compare July 16, 2026 09:34

josvazg marked this pull request as ready for review July 16, 2026 09:40

josvazg requested review from m1kola and mircea-cosbuc July 16, 2026 09:41

viveksinghggits reviewed Jul 16, 2026

View reviewed changes

josvazg force-pushed the CLOUDP-420746/publish-to-rh branch from cb773cf to 6b0633b Compare July 17, 2026 07:57

josvazg requested a review from viveksinghggits July 17, 2026 08:45

josvazg force-pushed the CLOUDP-420746/publish-to-rh branch from 6b0633b to cf6df00 Compare July 17, 2026 09:38

m1kola reviewed Jul 20, 2026

View reviewed changes

josvazg added 4 commits July 20, 2026 17:02

Use a custom GitHub App for RH publishing

a06ab88

Signed-off-by: Jose Vazquez <jose.vazquez@mongodb.com>

Fix wording around unified vs certified tarball naming

9ec30c4

Removed the manual patch trigger for RH releases

995c763

josvazg force-pushed the CLOUDP-420746/publish-to-rh branch from cf6df00 to 995c763 Compare July 20, 2026 15:04

josvazg requested a review from m1kola July 20, 2026 15:05

		# It downloads the unified certified bundle tarball (produced and uploaded by the
		# prepare_and_upload_openshift_bundles Evergreen task) and, for each RedHat fork,

		# It intentionally does NOT open the pull requests: it prints the "create PR" URLs and
		# the manual review checklist so a human can raise the DRAFT PRs and drive the review.

Uh oh!

Conversation

josvazg commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of Work

Checklist

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCK 1.10.0 Release Notes

New Features

Bug Fixes

Uh oh!

viveksinghggits left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josvazg Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josvazg Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

josvazg commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

josvazg Jul 20, 2026 •

edited

Loading

josvazg Jul 17, 2026 •

edited

Loading