AGENDA TOPIC: AI code assistant policy

[lgritz]

Please share any additional details on this topic

"AI" code assistants are becoming commonly used. We should decide on what our policies should be about it. While the details are probably up to each individual project, I think it would be good for the TAC to provide overall guidance and a sound policy for projects to use as a default if they don't have reason to create a custom policy.

Detail what actions or feedback you would like from the TAC

We want to get a clear idea of what benefits, protections, and risk tolerance levels the member companies and projects expect, as producers and consumers of software, and produce a set of written guidelines letting developers know where and how AI coding assistants may be used related to the projects.

Some things to consider:

• For what tasks and under what circumstances are LLMs permissible? Glorified autocorrect / typing suggestions? Modifying existing code? Refactoring? Debugging? Generating test cases? Writing entire functions or more?
• Are AI code reviews of PRs ok? If it makes a suggestion, is it ok to incorporate it into the PR?
• Is it ok to use LLMs to generating documentation? Proofread?
• What disclosures must a code author make about what AI tools were used and how?
• Who is the author of generated code, who takes responsibility for it?
• Do the DCOs and CLAs we abide by allow you to submit a PR that contains code you didn't really write yourself?
• What should projects do about PRs that are low quality or unacceptable because they appear to be LLM generated? What should they do about authors who habitually submit poor quality LLM code?

How much time do you need for this topic?

At least 30 minutes

[jmertic]

Good topic @lgritz - I'll state that we have https://www.linuxfoundation.org/legal/generative-ai to build from, and there is the availability of Copilot for ASWF projects ( @tykeal can confirm ). Hoping this can help!

[shidarin]

Just to get started, you need good solid foundations for AI assisted coding to be successful:

• Test suites with decent coverage and descriptive test names
• Usage documentation describing how the various parts of a project should be used
• Docstrings and API documentation
• Code comments describing why you're doing something in the code a certain way.
• A carefully crafted, minimal agents.md file providing general project guidance
• Additional steering

You should be using an agentic code agent of some kind which takes rules/steering and allows for planning/specification such as Cursor or AWS's Kiro. Here's Cursor's plan mode introduction. Here's an overview of Kiro's specification mode. I have an obvious bias but I've really been enjoying Kiro.

AI assisted changes are only as good as the engineer reviewing them, and the engineer's proficiency in the project and project code language. AI assisted coding is like supervising a genius child gardening prodigy. They can do amazing work but you're still letting a 3 year old run around with 24" sheers- supervise them unless you want your prize winning roses scattered across the ground.

For what tasks and under what circumstances are LLMs permissible? Glorified autocorrect / typing suggestions? Modifying existing code? Refactoring? Debugging? Generating test cases? Writing entire functions or more?

At this point, all of the above.

Are AI code reviews of PRs ok? If it makes a suggestion, is it ok to incorporate it into the PR?

A human needs to read the suggestion before implementing. AI assisted code reviews may lack context.

Is it ok to use LLMs to generating documentation? Proofread?

AIs familiar with the codebase generally write good, but excessive and pompous documentation. Generate than have a human trim.

What disclosures must a code author make about what AI tools were used and how?

I suspect this is going to be a bit of spitting into the wind. We're going to be getting AI generated submissions no matter what. It might be a good recommendation to ask submitters to check a box if it includes AI code?

Who is the author of generated code, who takes responsibility for it?

According to US copyright law, no one is the author of the generated code, but the PR submitter must take responsibility.

What should projects do about PRs that are low quality or unacceptable because they appear to be LLM generated? What should they do about authors who habitually submit poor quality LLM code?

Close the PRs, warn and then ban the author. Our expert's time is too short to spend it on low quality submissions (famous last words before all my PRs are closed.

-Sean