LLM-generated content policy

[twpayne]

Sadly, I'm starting to observe users posting unreviewed LLM-generated content in my repos (not just chezmoi). Specifically, I have seen:

• Users posting LLM-generated answers to other users' questions.
• Users posting LLM-generated code to "help".
• Users posting LLM-generated code that doesn't work and demanding that I fix the code for them.

In all cases, the LLM-generated content is superficially plausible but fundamentally incorrect.

It takes more effort to understand and fix the LLM-generated content than it takes to write correct content in the first place. Posting unreviewed LLM-generated content is extremely disrespectful of my, and the other maintainers', time.

Note that I don't care if people use an LLM to help them generate content, but I do expect them to review it for correctness before posting it here.

Going forward, I propose the following policy:

• Users posting unreviewed LLM-generated content with the admission that they do not understand the code will receive a single warning. If they continue to post unreviewed LLM-generated content then I will ban them.
• Users posting unreviewed LLM-generated content without any admission will be immediately be banned without recourse.

I plan to update the developer guide and issue templates with clear warnings, but would like to hear feedback from others first.

[Lockszmith-GH]

I agree with your reasoning and choices.
Bullshit generators^* are useful tools, but are not a replacement for a brain.

I love LLMs when they are useful, the same way I love search engines when they get me to what I was looking for.

Thank you for not calling this AI - as that term really annoys the hell out of me.

^* I've adopted Richard Stallman's term for a recent talk of his, where he addresses Stochastic Parrots / LLMs as Bullshit generators. I hope more people would use it as well, it describes these tools very well. However, if you feel the word bullshit is too harsh, referring to them as Stochastic Parrots can do as well.

[twpayne]

Latest terrible LLM interaction:

LLM bot @jacks-sam1010 from latta.ai tries to generate a PR in response to an issue in #4157 (comment). The generated code does not even compile, and is also wrong on every other level. I copied the change into #4158 so people can see just how bad latta.ai is.

[arran4]

Hrm I feel like I should have written better documentation cross referencing the functions.

[twpayne]

I stumbled across an LLM-generated podcast about chezmoi today. It was bland, impersonal, dull, and un-insightful, just like every LLM-generated contribution so far.

I will update chezmoi's contribution guide for LLM-generated content to say simply "no LLM-generated content is allowed and if you submit anything that looks even slightly LLM-generated then you will be immediately be banned."

[arran4]

Was the postcast a NotebookLM produced one?

Seems a bit harsh considering it's stemming from a bad podcast rather than bad actors on the repo (contributors are unlikely to be the people that create the podcast too.) The way I see it there is a lot of low quality content out there AI or not, but AI just enable a floodgate of low effort (and resultingly quality) content. While I'm not a "authenticity-ist" (as an opposite to "result driven") I do think that human effort is what is missing in a lot of AI generated content. It's good for bulk filler, ie explanations that you need as part of your writing to make things more approachable, rather than the core of the writing itself. I think your previous policy was good.

[singiamtel]

Does this mean Copilot tab complete is banned too?

What about asking an LLM for advice and then writing all the code yourself?

[twpayne]

As long as I can't tell that you used an LLM then it's OK ;)

[norbertkeri]

All in favor of this policy, and I'm glad you are taking a stance.

[twpayne]

Recently, there have been several issues that have triggered this policy (#4893, #4924, #4927). This has led to unhappy users/contributors. I want to explain my position as a maintainer.

• LLM-generated contributions tend to come from people who either don't understand the project or don't understand the programming language, or both. Consequently, the PRs tend to be drive-by "bolted on" features that don't integrate well with the project's existing code, architecture, or philosophy. For example, they often add new functions where existing equivalent functions already exist, make the minimal change that allows the "happy path" to run but fail on obvious edge cases, or implement features that should not be part of the project at all.

• I don't mind coaching human contributors who demonstrate the potential to contribute more in the future, but I have no interest in doing "LLM coding by proxy", where they take my review comments, copy them into the LLM, and then paste the output back. I understand that people contributing LLM-generated code are genuinely trying to help, but they're really just adding a slow level of abstraction. If I wanted to use an LLM, I can drive it directly myself.

• LLM-generated code tends to be plausible but fundamentally incorrect. On the surface, it often looks like it might be a reasonable solution, but on deeper inspection it is usually completely misguided. Such code is extremely hard to review as you literally have to reverse-engineer code generated by an alien brain. Unfortunately, as LLMs are getting better, the code they generate tends to be increasingly plausible and increasingly subtly flawed, and even harder to review. When reviewing human-written code, the human at least has a mental model that you can analyse and correct. The LLM has no mental model of the code to correct and does not learn.

• LLM-generated PRs push the mental load from the PR author (who gets the credit) to the PR reviewer who now has to do the actual work of making the code work and maintaining it forever.

Although I know that I come across as strongly anti-LLM, I am trying to maintain an open mind and I remain keen to embrace powerful tools. chezmoi is, in part, a personal testing ground for me where I can experiment with ideas and technologies, and where I can transfer the best practices to other projects and my professional life.

LLMs are certainly useful in many cases, particularly in greenfield projects where you have limited understanding of the technology. Many engineers who I deeply respect are using them very effectively in their own projects. In my case of an established open source project where everything is already well understood by the maintainers, they are currently very net negative.

Ultimately, open source projects only survive as long as their maintainers' mental health, and LLMs in their current form only destroy maintainers.

I'm open to changing this strict no-LLM policy in the future, but, for now, I choose to protect my mental health and prioritize chezmoi's long term viability.

[KapJI]

Thanks for sharing this, Tom. I agree with your point about protecting maintainers' mental health, that's essential for a project's long-term success.

That said, I feel the current policy is a bit too harsh. The previous version seemed better balanced to me: giving a warning first, closing clearly AI-generated PRs, and removing generated comments where needed. That felt like a better way to handle it without escalating things unnecessarily.

I understand why you've chosen this direction, but I hope you might reconsider the current approach at some point.

[halostatue]

@KapJI there's no change between the two policies.

It has always been zero tolerance (at Tom's discretion). The only changes made recently were to make the policy more obvious because one of the folks so banned asked someone to post a follow-up saying that they weren't aware of the policy in advance.

I won't speak for Tom, but I think that there's some room for tolerance but the rules around that are slippery and may lead to worse outcomes all around if people argue this. Tom has already said that using LLMs to smooth over ESL stumbling blocks is semi-acceptable, but even there it's a hard problem as the discussion on that random CI/CD comment shows.

If this were my project, I would be explicit: any command that the LLM says should be run to test something must be run on your local system first. If you don't show evidence of that, then you get your first warning—or banned if you've done this once before.

I think that vouch is VERY interesting, but I'm not sure how it will scale for a relatively small community like that of chezmoi.

[KapJI]

@halostatue By previous version I mean this: #4010 (comment)

[twpayne]

I'm open to reviewing this policy in a few months, but right now I'm getting multiple AI slop PRs (not just in chezmoi, in my other projects too) and I want to stop the onslaught before it becomes overwhelming.

There used to be a saying "talk is cheap, show me the code". Nowadays, I think this needs to be updated to "code is cheap, show me the understanding". This article on comprehension debt gives a good rationale for why LLM-generated code is long-term harmful. In the long term, I, personally, as the primary author of chezmoi, need to understand each change, and being assaulted by thousands of lines of code that the PR author doesn't understand is damaging both in the short and long terms.

[PallasAthenaie]

From @Blacksmoke16 again:

I just noticed this updated thread and first off I'm glad to see someone took my suggestion and created #4956. That definitely would have saved me 😅. All of your points are totally valid and I understand where you're coming from. However as I read it, it seems like they're all pretty focused on the LLM-generated PR side of things. Whereas in my case the it was an issue I opened, on my own comment with no expectation of review.

After debugging a bit of the original problem with you I decided to have Claude look into the secondary issue and thought its output was genuinely useful information that you, me, or some other contributor could use as a starting point, so I edited my post with the output and fully intended to leave things there. I explicitly didn't want to propose a fix because Go isn't a language I'm familiar with and wouldn't be able to vouch for the validity of it. Now in hindsight I probably could have handled it better than a literal copy/paste, but at the time I didn't think anything would have came from it without opening another issue dedicated to it given edits don't trigger notifications or anything.

I do agree with some other commenters in this thread in that, at least in my particular case, the response was a bit too harsh. But it's ultimately your decision so is what it is. I still imagine clemency isn't an option? 😅 I can promise you it wouldn't happen again haha.

[twpayne]

This is an AI slop comment: 1323292#commitcomment-179839683

I'm very tempted to use https://github.com/mitchellh/vouch.

[arran4]

The comment looks organic. Network of trust is interesting not sure if it's going to filter out AI but could work at reducing PR load.

[twpayne]

In what way is this comment "organic"? The comment talks about "CI/CD stuff" but there's nothing about the commit related to CI/CD. The comment has zero insight and is only noise. What makes you think this comment was written by a human?

[arran4]

Humans tend to make mistakes such as posting something to the wrong location, misunderstanding etc. Especially when tired, stressed, not paying attention etc. Plus there were plenty of mistakes in the comment itself. Doesn't mean it can't be, but in this case I think these are human errors.

[mackenzieclark]

Vouch is a cool project, I think it's a great idea. I hope it takes off and gets widespread adoption.

[twpayne]

For the record, I quite like rust-lang/rust's LLM policy, and would consider such a policy to be a possible good fit for chezmoi. There's no immediate change to chezmoi's current LLM policy however: I want to see how rust-lang/rust's policy (and other LLM policies) work out over the next few months.

[halostatue]

I'm generally happy with the GhosTTY policy. It's more open to accepting LLM output, but it's also clear that it's the human that has to take ownership.

[arran4]

GhosTTY seems to have a measure built in to it too.