Automatic release of patches for buildpacks

[loewenstein]

I just had a brief conversation with @fg-j about the degree of patch and release automation for stacks and I was wondering where we are with regards to this for buildpacks.

My understanding is that dependencies get tracked for updates automatically, but this has multiple manual interaction points with maintainers.
Once to merge a dependency bump PR and once more to cut a release. Is that understanding correct?

I can see this as required and even valuable for bumps that provide larger changes, i.e. major or minor bumps or depending on the dependencies and their history of breaking changes even patch bumps of specific dependencies.
However, for “usually well behaving” dependencies, users would greatly benefit from fast and fully automated delivery of fixes, in particular security patches.

What do you all think?

[robdimsdale]

@loewenstein - yes that's consistent with my understanding, too. However there are more steps than you highlight. The full process is as follows:

1. The automation creates PR to bump dependency in buildpack (e.g. go in go-dist buildpack).
2. A human (buildpack maintainer) adds a semver label to the PR in the dependency buildpack.
   • Typically this is the same level as the semver of the dependency as most dependencies follow semver.
   • This step could potentially be automated.
3. The automation merges the dependency-bump PR once a semver label is added and all checks pass (e.g. integration tests, linter, etc).
4. A human creates a new release of dependency buildpack.
5. The automation opens a PR in the language family buildpack to bump the newly-released dependency buildpack (e.g. go buildpack, in our example).
6. A human adds a semver label to the PR in the language-family buildpack.
   • Again this could potentially be automated, as all our dependency buildpacks follow semver.
7. The automation merges the PR in language-family buildpack once there is a semver label and all tests pass
8. A human creates a new release of the language family buildpack.

So we see there are two types of manual steps - labeling the PR and creating a release. This is true for both the language-family buildpack (e.g. go) and the dependency buildpack (e.g. go-dist).

We potentially could automate the semver labeling - even just on a per-dependency basis if we wanted to be conservative. I'm a little more hesitant to automate release creation because we would have to account for the fact that dependency bumps might get merged in the middle of an active track of work, and we currently do not use branches or feature flags for active tracks of work.

Taking a step back from a potential solution, what is the underlying issue you are trying to solve? Are you feeling pain around the release frequency of upstream dependencies in the buildpack ecosystem?

[loewenstein]

I have not seen a concrete issue with the release frequency yet, however I believe that we can’t deliver security patches fast enough.
I hope that we will not see many issues that need immediate action if any at all, but a potential RCE vulnerability like log4shell or spring4shell that doesn’t get patched after office hours or because of unfinished work on the main branch sounds like something to be avoided.

I’d say an automated semver labeling for well-behaving dependencies following semver or another reliable versioning scheme sounds like a first step.

Releasing with a similar rigor might be a bigger step, needing more and longer consideration.
We could figure out how to distinguish normal fixes from security fixes - if upstream for example has security notices like Ubuntu has for the stacks.
We could apply fixes to a separate branch based off of the last release and even automatically merge back those changes into the main branch.
Or find some other means to balance security and development ergonomics.

[loewenstein]

Also, wondering if the process is the same for all buildpacks...
The semver/patch label seems to be set by the paketo bot for Java buildpacks.

[robdimsdale]

Yes - there are some inconsistencies between the Java buildpacks and the rest of the paketo buildpacks. We aim to converge these buildpacks over time.

RE branches: I agree that using branches allows for quick and easy merges of patch releases onto main. It is a pattern many of us have used before with success on other teams. The main benefit that this model provides - the ability to prioritize security fixes over incomplete feature work - we get for free as we are using a PR model. I don't think there is enough benefit to switching to another branching model (e.g. feature-PR->"feature-branch"->"main" and security-PR -> "main").

Also, as an open-source project we don't provide any support for backporting security fixes to older versions. Nor do we have any particular SLA around fixing vulnerabilities. Commercial consumers of the Paketo Buildpacks can (and do) provide that level of support at their own cost. Our goal as an open-source team is to incorporate security fixes onto the main branch and release them in a reasonable timeframe. Practically, I have observed that we tend to release security fixes within a day or two of them being made available upstream.

Bringing this together - my personal opinion is that we should do the following:

• explore automating merging of dependency PRs across all the paketo buildpacks (similar to how Java does this for the dependency and language-family buildpacks)
• explore automatically releasing buildpacks (perhaps the Java buildpacks already do this too - I'm not sure)
• continue development with a PR-to-main model: do not introduce additional branches or process at this time.
• potentially improve documentation around the support model (e.g. no SLAs, no backports, etc)

What do you think?

[loewenstein]

I agree that additional branches do add an additional cost in terms of complexity. I just brought them up because the PR model of course doesn't guarantee that nothing lands on main that maintainers do not want to release yet.

So, if that wish to decide on the time to release would get in the way of timely security fixes, I think short lived "let us quickly fix the last release and ship the next release in a week" branches do not impose a massive cost, especially when controlled by automation.

Also, as an open-source project we don't provide any support for backporting security fixes to older versions.

I wasn't talking about older releases, but rather fixing the current release - if need be without the feature work that will be in the next release.

Nor do we have any particular SLA around fixing vulnerabilities. Commercial consumers of the Paketo Buildpacks can (and do) provide that level of support at their own cost.

I wonder why we don't though? Like, we could setup forks of the Paketo buildpacks and work on automation for providing fixes quicker (if that's possible at all - note: I am not at all complaining about Paketo delivering fixes slowly) and more automated or for older relases if we see a demand for this. But why would we not share those efforts and the cost involved with other vendors using Paketo buildpacks?

Your summary sounds right. Thanks. I don't know if there's anything missing in the documentation actually though. Of course an open source project has no service level agreement.

[robdimsdale]

It sounds like we're broadly on the same page.

Your proposition for sharing the burden of backports across vendors is interesting. This might be worth raising at a working group meeting - I know those meetings are attended by representatives from VMware and sometimes other vendors.

Let's see if we encounter pain around security fixes not being released in a timely manner, and, if so, we can evaluate a change in process. Related - if you want to perform some analysis on how long it takes to release security fixes across the paketo buildpacks that could make for a really interesting conversation.

[fg-j]

Something I've been thinking about:

We could set up component (e.g. go-dist) and language family (e.g. go) buildpacks to auto-release when the release would be a patch version; this would avoid automatically releasing features/breaking changes, but would take some manual work out of patching CVEs in buildpacks.

In my opinion, this is a lower lift than implementing semver autolabeling for dependency bumps because it avoids having to parse git diffs.

I think more semver auto-labeling would certainly make my life as a maintainer easier, but it's not the lowest hanging fruit right now.

[robdimsdale]

I am in favor of auto-merging and auto-release (see my comment above), but I think we should:

• be consistent with the java buildpacks
• only auto-release a "patch" when there is a dependency bump (vs e.g. a bump to github-config or tests). This may be what the java buildpacks already do - I'm not sure.

To elaborate on the second point - if we explore additional automation around releasing I would want to ensure that if we auto-release when a semver:patch PR is merged we only do so when there is a dependency bump. There are other types of PRs with semver:patch that should not constitute a release. Examples to this include:

• tests, and bumps to testing libraries
• github-config
• README changes

While it is technically not a problem to release patch releases for all of these non-functional changes, it would cause a significant increase in automation (and we already have issues with github actions struggling to handle the level of automation we have) and it would be noisy for consumers of the buildpacks; it would not be easy to determine if the patch release actually offered any bug fixes or was simply a non-functional change.

It's possible the Java buildpack automation already handles this case.

@paketo-buildpacks/java-buildpacks can you summarize how the auto-labelling and auto-releasing works for the Java buildpacks?

[sophiewigmore]

Agreed Frankie. My thoughts are that we do a pretty good job as a project merging in dependency-update PRs in a timely manner as-is, and the release process is the part that more often gets left out anyways.

[loewenstein]

I like the idea of auto-bumping across the project and auto-releasing those bumps only that at least could include security fixes. I guess without better knowledge that's going to be those that bumped buildpack dependencies.

[loewenstein]

There are other types of PRs with semver:patch that should not constitute a release.

If I am not mistaken, there is an elstablished label for type:dependency-upgrade as well, so this should be easy to identify. I suppose that label is also already used for release note generation as e.g. in https://github.com/paketo-buildpacks/apache-tomcat/releases/tag/v7.3.4 showing bumps of Tomcat 9 and 10 dependencies.

[dmikusa]

Sorry, all. I've been OOO so just getting to this conversation now.

On the Java side we...

• check for dependency updates once per day with the upstream source, this is really the most frequent that Github actions can accommodate or we will hit limits with action limits
• if there's an update a PR is automatically submitted, the PR is tagged based on the upstream version (i.e. patch -> patch, minor -> minor, major -> major).
• We review the PR and merge. Most of the time this is straightforward, but we do occasionally need to manually adjust the PR. Examples would be: a.) if the label isn't right, which usually happens cause not all projects follow semver yet we are forced to, and b.) odd behavior from upstream like they changed an already published dependency.
• We typically cut component buildpack releases on Thu and we cut releases of composite buildpacks on Friday. This gives us a weekly release cadence. We are striving for a weekly release cadence for consistency and to not overload users with too many updates.

If there is a critical security issue, we'll cut an immediate release of the affected component buildpack & likely also the composite buildpack, depending on how things are impacted.

I have no interest in automating the release cycle more than it is presently at this time. The manual inspection of dependency PRs and manually determining when we are ready to release allows us to catch issues that could be problematic for the community.

As I've said in other discussions, I would much rather get dependencies out of the buildpacks. Then we can fully automate the availability of dependencies for buildpack users and get into a saner release cycle for buildpacks that is largely feature driven.