Consider adding an `AGENTS.md` to reduce AI slop

[notatallshaw]

Well, the AI slop era is upon us, many recent PRs have been created by people submitting hundreds of PRs in a short time to hundreds of projects.

Like many of the recent innovations by technological companies, they are willing to sell you the solution to the problem they are enabling, as such dealing with AI slop may require adding repo documentation for AI tools.

The Apache Airflow project is taking this very seriously:

apache/airflow#62575
apache/airflow#62443
apache/airflow#62442

I suggest we create an AGENTS.md that instructs something like:

1. Fully read the contributor guide and follow it
2. Linting and Tests must pass
3. Aim to minimize reviewer time by being targeted, minimal, and complete
4. Code must match the project's existing code style
5. Test must match the project's existing test style
6. Must include meaningful unit tests
7. Must include functional tests for user facing behavior changes
8. Must be minimal in adding functional tests as they are expensive on CI resources
9. PR's that change only trivial things like fix typos or links, or add redundant comments or doc strings will not be accepted

The hope is that bots that have been set to open PRs en-masse against any open issue they can find will realize that their PR most likely doesn't meet these standards and not open a slop PR to pip.

Very open to ideas here.

[pfmoore]

I strongly suggest that we adopt the attrs AI policy: https://github.com/python-attrs/attrs/blob/main/.github/AI_POLICY.md

Maybe we need an AGENTS.md file as well, but I don’t think attrs has one, so I’m not sure if it would be worth it alongside their policy.

[notatallshaw]

I'm open to adding a policy document, this is not intended to be that.

This is intended to be a practical solution to the fact users are setting ai bots to automatically search repos, find issues, and open PRs without ever reading the projects policies or contribution guide. A lot of bot services are converging around reading the AGENTS.md file first.

One thing we could do if we had both is instruct the bot in the AGENTS.md to print out the AI policy document to the user and instruct the bot that the user must agree to the policy.

[notatallshaw]

@pfmoore if you want to open an issue or a PR separately for a policy document like that I would be supportive and review.

I think some of the sections are unclear/don't make sense for pip and should be slimmed down or removed, and I would add some clarifying language to other parts.

[pfmoore]

See #13827 (and the associated PR #13826). I am extremely limited on time at the moment, for personal reasons, so I'm happy to see reviews, and will attempt to respond to them in a timely manner, in particular from the other @pypa/pip-committers, but I won't get involved in meta-debates around the merits or otherwise of AI in general.

As far as this issue (AGENTS.md) is concerned, I don't know enough about how to control/manage LLM agents to have anything meaningful to say. If an LLM tool is given access to the pip project, including all of our contributing documentation, I'm unclear as to why that's not sufficient and an LLM-specific file of instructions would fare any better. But if a committer who is skilled in writing such instructions wants to add one, I'm fine with that. I don't think this is something we should accept external contributions on, at least initially - it would be too easy to end up with a file that seems to say what we want it to, but doesn't actually have the effect we want. It's a chicken and egg situation - if the maintainers don't have the skills to maintain a valid AGENTS.md file, how can we support a contributed one? How can we even know if someone submits an AI-generated one?

As an alternative (or additional) way of minimising AI slop, could we simply reject any PRs that say they were authored by (or co-authored by) an LLM? Requiring the contributor to take an explicit step of asserting ownership might be enough to at least remove the more harmful cases (the pure automated LLM generated PRs).

[notatallshaw]

. If an LLM tool is given access to the pip project, including all of our contributing documentation, I'm unclear as to why that's not sufficient and an LLM-specific file of instructions would fare any better

This is a side effect of the mathematics of LLMs and the consequences on engineering of agents. Most agents are looking to minimize the amount of information in their context window as it is a fixed size, and agents are engineered to summarize the information in their context window when they go over a certain size, this summarization process leads to noticeably worse results.

As a result there is a lot of "context engineering" to try and get LLMs to consume a minimal amount and correct level of granularity of information for what the the LLM is being tasked to do. The industry is converging around AGENTS.md files, where when it is placed at the root level of a project it is instructed to immediately read it to understand the rules of interacting with that project.

But if a committer who is skilled in writing such instructions wants to add one, I'm fine with that.

I have experience in my job, including being used with a wide number of different models, however in that role I only have to deal with a handful of contributors and none of them are attempting to write PRs en-masse, so I'm not sure how "skilled" I am. But I am also following along with the progress of Apache Airflow to reduce the amount of maintainer time spent on AI slop.

I don't think this is something we should accept external contributions on

Agreed, my intention is to reduce the amount of maintainer review times by reducing chance of receiving AI slop PRs, I think it makes sense to even clearly state that's the intention in the file, I think external contributors might have very different ideas on the purpose of such a file.

As an alternative (or additional) way of minimising AI slop, could we simply reject any PRs that say they were authored by (or co-authored by) an LLM?

I am 100% onboard with this rule and think it should be in the AI policy, I have already made such points in PR reviews, e.g. #13801 (review).

Requiring the contributor to take an explicit step of asserting ownership might be enough to at least remove the more harmful cases (the pure automated LLM generated PRs).

I don't think it will, as my understanding is some users are doing "agentic" workflows with no human interaction until after the PR has been opened, e.g. Search for popular open source products -> Find open issues that are awaiting PR -> Write code to implement-> Open PR. They may not even be aware of what projects their bot has contributed to.

[pfmoore]

They may not even be aware of what projects their bot has contributed to.

Excuse my ignorance here, but how are such PRs being opened then? Are they using the bot owner's github account without that owner's knowledge? Or are they using bot-owned accounts with no actual human owner? Either of those situations seem like they should be violations of github's terms and conditions¹, and regardless seem like something we could legitimately ban the user from contributing to pip immediately. (Which begs the question of how we identify such cases without consuming excessive maintainer time, of course 🙁).

Footnotes

1. It would be interesting to report such accounts to github, and see what happens... ↩

[notatallshaw]

Are they using the bot owner's github account without that owner's knowledge?

This is my understanding, bearing in mind I have not done this myself and do not work with anyone having done this. But I do observe "human" accounts who have opened many hundreds of PRs against many hundreds of projects in a few days, when they previously had almost no contributions to any project.

Either of those situations seem like they should be violations of github's terms and conditions1, and regardless seem like something we could legitimately ban the user from contributing to pip immediately.

I do think this is something github as an organization needs to rapidly start tackling if it is interested in preserving the open source ecosystem as a viable model. I have reported some accounts with the comment that they are spam and opening AI slop PRs, I have not received a verdict from github's support team on any of them yet.

[sbidoul]

Maybe a stupid question, but is there a possibility that the mere presence of an AGENT.md would actually attract LLM contributions?

[notatallshaw]

but is there a possibility that the mere presence of an AGENT.md would actually attract LLM contributions?

That's very a reasonable question, I'll do some research if there's any evidence on this.