Time to first comment metrics

[GregSutcliffe]

This should probably be restricted to maintainers - i.e. time to first comment from a maintainer. However, that could be tricky in bot-controlled collections community.general

@gundalow is community.general the only repo that uses a bot to do merges in lieu of just having maintainers?

Failing that, it could instead be "time-to-first-comment" from a past committer...

[GregSutcliffe]

THis gets complex fast, the only way to identify a maintainer in a BOTMETA-controller repo is by parsing the botmeta file and comparing to the paths touched in the PR.

For simplicity, I'll start by looking at the "author_association" field in the GitHub api for comments. This can have the "maintainer" value, so we'll look for that. This won't work for bot repos like community.general as only the bot is an official maintainer, but we can revisit that.

[GregSutcliffe]

First version add in fa8c148 - for now this is just a comment from any non-bot person other than the author. Will refine further later.

[gundalow]

@gundalow is community.general the only repo that uses a bot to do merges in lieu of just having maintainers?

The bot will be running on other repos, such as AWS, Network & Windows

For collections using Zuul-CI (mainly network) all the merges are done by the Zuul bot when the gate label has been added AND CI is green. Example: ansible-collections/cisco.nxos#146

This should probably be restricted to maintainers - i.e. time to first comment from a maintainer. However, that could be tricky in bot-controlled collections community.general

What are your thoughts around "Time to first comment by maintainer" vs "Time to first comment by some human"

[GregSutcliffe]

My gut feeling is that the only case where they'd be very different is when there's a lot of "+1 I have this issue" going on. Otherwise, I feel that most of the time, maintainers or keen contributors would be the first to reply.

Which makes a good point. If a maintainer is burning out, do we filter to just comments from maintainers (so we see the worsening time-to-comment) or include other humans (so that we see people are still getting replies)...

[GregSutcliffe]

Detecting maintainers is starting to look hard in a variety of contexts, but if we can do it, then I guess we'd like both?

[gundalow]

I'm OK with time to comment from first human (that isn't the creator) to start with.

For repos not using the Bot, maintainers are defined as those with write (or above) privileges, which I assume GitHub can give us.

[GregSutcliffe]

The API returns an "AuthorAssociation" field for each comment. https://docs.github.com/en/graphql/reference/enums#commentauthorassociation suggests that MEMBER or OWNER should tell us who can commit, normally. At least I think so - one has to be in the org to have rights granted, yes? But those rights might be "readonly"? Hmmmm.

[GregSutcliffe]

Ah, I think https://docs.github.com/en/graphql/reference/enums#repositorypermission will work. Will test later on.