Internal: implement tool to construct changelog

[mwichmann]

At the moment each pull request is expected to have its own edit of CHANGES.txt - the PR template encourages this. The problem is after a PR sits for a while, there are inevitably conflicts to resolve in that file. This hits you when you need to rebase, for example - you have to perform a relatively useless merge edit because two PRs each added a line in a similar place and git sees a conflict. We would like to implement something to handle keeping these separately and producing a changelog for a release at release time.

For illustration, the CPython project has a system where with each merge, a new file is created with the potential changelog entry, these are then stitched together by a homegrown blurb tool when it comes time to make a release (and the directory of changes-since-last-release is cleaned out). You can see this in action at https://github.com/python/cpython. Look in Misc/NEWS.d. That directory contains past version-specific changelogs, while the subdirectory next containss categorized snips that are candidates for inclusion. The needs of CPython are not the same as SCons, so directly lifting it doesn't entirely make sense, but it serves to illustrate the idea.

[bdbaddog]

https://github.com/twisted/towncrier also looks pretty good.
Buildbot project uses that. among others.

[mwichmann]

I don't have any biases here, just pointed to python.org as a sample of what could be done, to get this recorded.

[mwichmann]

Okay, towncrier seems fine for this purpose. To put it into use, I guess we need some dev guidelines, to answer these questions plus I'm sure ones I didn't think of.

1. Who: does a PR submitter write the entry? If so, when? The "convention" suggested by towncrier is to use an issue number, we don't always have an issue for every PR and you don't know the PR number until you've submitted it, which makes a little bit of a chicken and egg if you say "created with the PR". Or does a project maintainer create it when merging?
2. Categories: predefined are .feature, .bugfix, .doc, .removal, .misc but you can also create your own.
3. Content: is the policy "only interesting to users"? So less detailed than CHANGES.txt? Would it replace or complement CHANGES.txt. Do you make entries that aren't necessarily interesting to users? (.misc is defined for that in the project page).

[bdbaddog]

Here's buildbot's README.txt on such..
https://github.com/buildbot/buildbot/tree/master/master/buildbot/newsfragments

1. Yes. PR submitter
2. maybe add .deprecation ?
3. hmm. not sure. let me ponder and respond later.

[mwichmann]

thanks... maybe I should have made item 1 two questions, as there is an interesting chicken-and-egg problem (I don't know how Python handles it with their setup): how to use the issue for the news clip name if you don't have the number until you submit?

[bdbaddog]

O. IC. No need to have PR # IMHO.. Only an issue # if the PR's fixing it..

[mwichmann]

So towncrier (and the Python blurb system, and one other I looked at) don't have a way to order entries. They do use the issue # in the fragment name. Python uses the date as well, for example NEWS.d/Library/2021-01-16-18-36-00.bpo-33809.BiMK6V.rst to fix bugs.python.org #33809.

So in the absence of mandating an issue number as the fragment name (or part of name), we'd need a "naming convention". Which could just be "make up something unique", or maybe we could provide a tool that adds a random number to the chose name as apparently Python is doing.

We don't have a way to associate the PR submitter (or author if someone else submits the PR) with the fragment contents like the current CHANGES.txt does - but RELEASE.txt does not.

Do we still need CHANGES.txt? we do have git log though it tends to be noisier than what's gone into CHANGES.txt. On the whole, this seems better suited to the model that towncrier seems written to: "news for users".

[mwichmann]

So with 4.2 released, what do we do here? Keep the status quo? Implement this to replace RELEASE.txt? Or CHANGES.txt? Or both? If both, do we need to keep two separate ones?

[bdbaddog]

Interesting.. so you can specify alternative output template files. (jinja2 format?)
From their git repo:
https://raw.githubusercontent.com/twisted/towncrier/master/src/towncrier/templates/hr-between-versions.rst

[mwichmann]

In the example repo I did up, I had to comment out the template thing because I couldn't figure out how to make it work. But possibly that is part of the answer.