Copilot, thoughts?

[dopry]

In a pull request @lullis recently said

I understand it might help you to deal with some of the maintenance load, but to be 100% honest dealing with Copilot is quite demotivating, At least for me, it makes me less interested in becoming a more active contributor to the project.

and @Natureshadow expressed a desire not to deal with copilot.

I'd love to hear more about various peoples experiences with copilot.

As for myself, I find copilot to be extremely helpful. As a maintainer I am nervous every time I'm reviewing and merging a PR. I'm always concerned I've overlooked something important in code review that will adversely impact the users of DOT. I'm worried about security issues, performance bugs, standards conformance, architecture and design, testing, and overall quality... Frankly it's a lot.

I have blind spots, some I am aware of, some I am not. Typescript is my daily driver language. Django is a tool I only use for one client. I spend a lot of time verifying what the current best practices are for Python, Django, and Django package development. I also have limited time.

So far my experience with copilot code review as both a maintainer and developer has been that it tends to catch a lot of quality and conformance issues when they are present. When there aren't a lot of major issues it will nitpick the crap out of minor code style issues. Regardless it brings issues to light for me to consider and consider them I do. I am happy for the bugs and quality issues it finds and simply disregard the nitpicking.

I feel it improves my overall code quality. I do verify it's findings and refute its suggestions when I disagree. I do find the nitpicking frustrating at times when I just want something done, but maintaining and contributing to DOT isn't just about putting things out the door. It's a security related module. It's critical infrastructure for its consumers. I feel quality and conformance should be our priority, not simply getting features shipped.

I'd love to hear from others about their experience both positive and negative. I'd ask the we focus on sharing actual experiences and not cathartic expression and externalities.

[n2ygk]

I find the verbosity of the copilot comments to be difficult to wade through.

[dopry]

I wonder if that is something we can tune the AGENTS.md to improve. Do you have specific examples of comments that were particularly verbose?

[n2ygk]

No, I don't have specific examples. Just when I get an email from github for a PR update, it's followed by a wall of copilot text. I actually haven't drilled down into any of the text to assess the value added. I will try to make time to actually read the comments and provide more useful feedback.

[dopry]

Maybe we can find a way to keep the copilot stuff out of the email if it is overwhelming. I stopped using email as a trigger for review, generally my work as spread out over various PM tools and calendars. I miss the day when I could work from my Inbox. I tend to use the Github Notifications view on the website and have calendar reminders for maintainer work. I will occasionally peek when I am not buried in other work.

Take a look, so far the copilot feedback I've seen seems to be largely valid. Some of it may be minor... but things like catching the replay attack exposure in the backchannel logout PR that I would most likely have overlooked are what I find valuable.

[lullis]

I'd avoid Copilot purely for its ethical (Copilot being trained on tainted data, opens the possibility of people arguing for any code touched by Copilot to be public domain) and systemic (do you really want to have your worflows even more dependent on Microsoft?) issues, but even if I turn a blind eye for that: the demotivating part of the current flow is that the maintainer is asking for assistance, but the onus of dealing with its requests falls on the developer creating the PR.

I see the comments from Copilot and my first instinct is to ignore most of it and only take action if the comment shows a critical mistake, but then I am second-guessing myself regarding (1) what should be considered "critical" and (2) how likely the PR is going to continue to be ignored because I didn't work on the "minor" stuff.

So, to sum up: it's a weird dynamic and I worry that the way that it's implemented will lead to a toxic environment where the AI becomes the ultimate authority.

[dopry]

I'm not putting the onus on you of reviewing copilot. I'm assessing it's comments for merit first before asking you to address. them. How is regurgitating the comments any better than just reviewing the copilot comments before asking you to address them? For me it doesn't save as much time. I have RSI in my hands and wrist that makes typing accuracy difficult, along with the progressive near field vision issues that come with middle age. It doesn't save me as much time and increases my typing load if I have to copy it all around by hand. I also don't want to tie up local resources on my machine for the code review, I prefer GH actions doing it in the background so I can trigger a batch of copilot reviews at the start of a work session, then come through and review them before asking people to address issues that I don't dismiss. Are you opposed to people using assistive tools? Or do just that you find the personal attention more rewarding?

[lullis]

I am not opposed to assistive tools. I am opposed to tools that become surrogates for personal responsibilities and blurry the lines of "who made what call".

Case in point: the RP-initiated logout is sitting there for over a month, without any "request for changes", just a bunch of comments from Copilot. Do you think I should address those or should I keep waiting for further feedback from you?

[dopry]

this batch from last week, #1573 (review)? If you are eager to get it the PR in, then yes you should address them.

[lullis]

I was thinking about RP-initiated registration, which I think I addressed the things you raised.

But in regards to the one related to backchannel logout, this works as a very good example of why this flow with Copilot is not working out. I thought you said you were going to take care of the last round of comments from Copilot, now you are telling me that I am supposed to look into it...

[dopry]

I was going to, but I don't have time... That has nothing to do with copilot, that has to do with my own habit of overcommitment.

[Natureshadow]

As I already told the maintainer in private mail, I consider the use of Copilot highly disrespectful towards contributors. All the PRs here where our team is involved took a lot of time to work out. We spent many days and several thousand Euros to improve django-oauth-toolkit. Reacting to that with a pile of LLM vomit is not acceptable.

[dopry]

@Natureshadow nothing is being 'rejected'. It's just an extra review step. A lot of the comments are accurate and catching errors and bugs in people's code. Your statements doesn't sound experiential. It sounds more like an emotional reaction.

[Natureshadow]

Your comment reads like you read my remarks filtered through an "AI" assistant. That would explain why you make up quotes I never said.

[dopry]

Not filtered through an AI assist. I'm just a human being who makes errors. I think I misread Reacting as Rejecting . (my close vision isn't always so good these days). So did you actually read the comments? Were they wrong? Is your problem with the content of the comments or how they were generated?