LLM-wielding students seem to be a new "attack vector" in this and many other repos

[nickchomey]

It is abundantly clear to me that this repo (and seemingly many others) are being spammed by LLM-wielding students who are eager to put something on their CV.

There's been some seemingly helpful PR that make small (eg http->https) changes, but there's a lot of very concerning stuff going on that I dont think the maintainers are at all aware of yet.

#1927 is the most egregious example of this, but #1937 is completely of this nature, I opened #1936 but the only two responses I got were clearly like this as well. Even after I rejected the terrible proposal there, they went ahead and submitted a PR (#1949) anyway, which @jmanico started treating seriously. I've noticed other maintainers (eg @szh in #1937) doing the same.

You created an AI-disclosure policy and PR template (which only seems to be sporadically used), but there's clearly nothing stopping these people from flouting it all. Moreover, even if they say they "manually reviewed" the AI output, what value does that have when they are self-interested ESL students with limited knowledge/experience?

Given that this is a premier source of security guidance, it seems like it would be prudent for your team to give serious thought towards how to be significantly more vigilant going-forward with guarding against this issue.

edit: To be clear. I say this all with significant gratitude for the work you have done. It is an alert and call to action, rather than any sort of castigation. I dont really know what can be done about it, but I am sure that all of open source is suffering from the same problem.

Ghostty terminal seems to have a comprehensive approach to all of this: https://github.com/ghostty-org/ghostty/blob/main/CONTRIBUTING.md

They dont even allow anyone to open issues (and presumably PRs must stem from an approaved issue) - you have to start a discussion first.

I hope this helps

[szh]

Agreed. I for one have been trying not to be too heavy handed or confrontational in order not to turn away potential new open source contributors but I acknowledge that this tendency has serious downsides.
Do you have ideas for how we can improve processes to handle this new onslaught - without throwing out any and all AI usage?

[nickchomey]

I just ninja-edited the OP at the same time you posted - i added a link to Ghostty terminal's contribution guidelines. It talks a lot about AI, but also they dont even let people open Issues. You have to start a discussion first, and then issues graduate from there (and presumably PRs must have a pre-approved tracking issue).

A lot of overhead, but it might be less work overall - especially since running the guantlet would probably largely deter these people altogether. People who genuinely care (eg how I and others did about the Sec-Fetch CSRF guidance) would be happy to go through the motions.

[szh]

Thank you. We'll have to take a look. It's a hard balance, trying to be open to contributors (which is our lifeblood), while not allowing the flood of low quality slop.

[mackowski]

Thanks @nickchomey for referencing Ghostty terminal's contribution guidelines they looks solid.

Also I think we can stop PR without assigned issue — it is much simpler to review issue than PR.

[nickchomey]

Yeah that seems like an easy first step to improve things. The common pattern I've seen from these student spammers across repos is look at issues, post some Ai slop about "this is a crucial issue that I see come up all the time, please assign this to me and I will do 5 vague bullet points" and then often submit a PR regardless of response.

If PRs are explicitly prevented/closed without some initial meaningful discussion and explicit approval for PR (eg what happened with the sec fetch csrf guidance, and surely lots of other places), it'll surely cut back the noise.

Though, I still think that there will be a ton of slop in issues, as evidenced by the few that I linked above (and were thankfully closed).

The discussion-> issue->PR gauntlet might be sufficiently onerous to discourage low effort sloppers, while not being a problem for people who care.

[jmanico]

What I suggest is that we do not force a formal process or template on contributors. We should accept contributions in whatever form folks can provide, and then politely work with them to get it into shape. When someone is far off-base, we can point them to our contribution guidelines. More than once if needed.

Recently we’ve also seen a rash of low-effort AI-generated submissions. In the last few days we’ve been more aggressive about closing issues that are in the "AI Slop" category. I think we should keep this up, to protect the time of leadership.

Here’s the behavior I’m proposing (and I am open to other suggestions):

1. Be kind and respectful to all contributors. Avoid inflammatory or accusatory language. This is open source, and we should model that. ABN: Always Be Nice.
2. When a contribution is off-base, guide folks to our contribution guidelines and help them course-correct kindly.
3. Be quicker to close unactionable / low-effort submissions so they don’t consume maintainer time. Close them promptly, and do it politely. Or give folks a deadline when the content is on the border like "fix this in 2 weeks or we politely close it out".

This is just my 2 cents. I know your time is valuable as leads for contributions and I want to respect it. I’m hoping that being quicker to close non-actionable items addresses some of your concerns.