Every OSS project maintainer I hear from is like "we can't ban AI because it's an unenforceable policy. There's literally no way of knowing whether something is written by AI" and every OSS project contributor I hear from is like "yeah, contributing is really miserable because I'm so busy reviewing this PRs and every single one is so obviously AI"
4 Boosts
3 Likes
@mcc *offers bat* There seems like a simple resolution to the second one!
This federated comment is not yet available here.
@aeva@mastodon.gamedev.place @moody@hj.9fs.net @mcc@mastodon.social I think a lot of people don't want the responsibility of judging "does this code smell like it was written by an LLM? Am I being paranoid and is rejecting this going to upset someone because I'm wrong? " plus questions like "we accidentally accepted a bunch of LLM authored commits 6 months ago, what do we do?"
Especially if in the latter case you might end up with an angry mob equipped with pitchforks whatever you do "It's slop now!" vs "You removed that feature I really loved"
@erincandescent @moody @aeva @mcc in practical terms, if we ignore the purity test part (which gets icky), a lot of maintainer burden can be eased by a blanket policy of "giant drive-by PR" -> instant reject
@erincandescent @moody @aeva @mcc it used to be that I wanted to work with people on this, now it's just "if reviewing this is a chore, and it wasn't explicitly asked for, it's an instant no".
re: AI policy, again I'm not super-interested in the "purity test" angle of it. I don't actually care if a 2-line patch was written by a LLM. If it's a 2-line patch and the PR desc doesn't make me want to kill myself, that's fine.
@erincandescent @moody @aeva @mcc in the other direction, if the PR desc is a 3-page wall of text full of emojis, I'm not even gonna read it fully before I reject it at this point
@rygorous @erincandescent @moody @aeva @mcc half of these are also very „this kind of terrible patch could have been a pretty decent bug report and that *would have been way less work*“ ime. Like instead of a bad fix just tell me what’s wrong then I can write a good fix based on that instead of first reverse engineering what the problem actually was
@halcy @erincandescent @moody @aeva @mcc oh yeah +100, I just wrote basically that.
even before LLMs, most PRs I got were "I wish you'd given me the bug report instead, because this fix sucks"
@rygorous@mastodon.gamedev.place @halcy@icosahedron.website @moody@hj.9fs.net @aeva@mastodon.gamedev.place @mcc@mastodon.social though sometimes you're dealing with e.g. the Linux kernel where there is no bug tracker and the most effective way to get your bug fixed is to send whatever patch you can cobble together. there's probably a sweet spot there where it's good enough to get looked at but bad enough to get rewritten to be better
@mcc What happened is that AI boosters brainstormed arguments that were strong enough to manipulate some people, and now everyone repeats it.
@mcc
you're listening to the wrong OSS project maintainers - there's also many who are rejecting AI and acting like everyone has already given up makes it harder for them to defend their position
@Doomed_Daniel i want an operating system and a web browser and a text editor. i want to use the "rust" and "python" programming languages. of these the only thing i can have right now is classic vim (?), but i haven't had time to install it on my computer as i've been too busy trying to replace my mail provider and maps provider and my translations app which all got replaced with gemini. then i'm gonna be real busy trying to figure out how many years until servo and redox are daily driver ready
@mcc lets just have no rules for anything since people can always lie and commit fraud! this is a very sensible standpoint
@eniko "no no it's important we continue to have other rules we're just going to draw a dotted line around this one rule i want to break and say that's impossible"
@mcc @eniko our current company policy is that an engineer is the author and responsible person for the code that they submit for review, no matter how it was created. So if an engineer submits a big steaming pile of crap, we refuse to review it, and ask them why they think it's acceptable to treat their peers in such a disrespectful way, and why they think this mess is something we as a team want to maintain over the next years.
Other engineers use LLMs but adapt the code and commits into readable and idiomatic changes. That's fine. Those reviews are smooth.
@mcc Ultimately an admin shouldn't care about AI, only code quality and manageable submissions both issues perfectly solvable. The fact that solving them also takes care of AI slop is just "free" bonus. Making the issue strictly about AI is the wrong way to go even if we agree that AI is definitely the main cause of the problem. If a human post code produced by an AI, you can tell the if code is bad but never that an AI wrote it with any certainty. Sloppy developers are not something new.
@jeantranscene i think ai should be banned even if it hypothetically produces human-indistinguishable code because there are moral dimensions and externalities.
i also think this rule is probably procedurally needed because slopmongers tend to post slop and say "you should accept it because it's indistinguishable from human code! i reviewed it after all"
@mcc@mastodon.social
loved reading that coming from the most ai-in-pfp person i have yet to see on mastodon who is probably writing all their posts with ai. so cool! love that. so normal
@mcc That's a political statement and you are perfectly entitled to believe it but this have nothing to do with code repo management efficiency which is the point I was trying to make.
@mcc Ok, as a maintainer I can attest that I have started ti deny. I have accepted some ( claude ) in the past. I'm still cleaning up. Good experience, but not one I wish to repeat. I maintain 50+ programs & libs and at the moment, I just doing crack to catch up.
@mcc I think banning AI is the wrong way. We should just ban idiots. Unfortunately chatGPT makes idiots think that they know something.
@mcc
Maybe an unusual POV, but I'm fine with these PRs. I can review them only as much as I wish, and I can ignore or reject them at a whim. In the current state of affairs any formal policy looks like an invitation to fight over it.
I'm trying to be nice though.
@mcc@mastodon.social @iorsh@kishkush.net
>
any formal policy looks like an invitation to fight over it.
I mean, you can't fix stupid. If someone is dumb enough to willingly get onto the trebuchet set to launch directly into the fucking Sun, that's their decision but they shouldn't be shocked when they end up launched into the fucking Sun.
@daemonspudguy @iorsh i appreciate you're on my side on this one, but it seems like max is trying to be polite and you're responding pretty abrasively? like i disagree with max (and said so) but your reply seems unlikely to convince him to change his mind
@iorsh I would simply ban anyone who wants to have a conversation about this. They've just raised their hand as someone who is likely to try to sneak around the policy
@mcc
My feeling is that mass bans very much invite for more whining and brigading. A dozen of overblown PRs quietly rotting in the queue - much less so.
@iorsh As a contributor to open source projects, when I see a lot of ignored open PRs I worry that the maintainer is checked out. I think this is overthinking it. I've never had a high-engagement github project, but in other online situations I find people get engaged when you *argue back*. But block and delete and brickwall them and they get bored at the lack of stimulus and troll someone else.
This federated comment is not yet available here.
@mcc cowards! I make at least 4 unenforceable policies a day!
/me is a libre/free open source maintainer who is so old, he barely survived the last so-called "A'I' Winter" and was tasked with writing anti-A"I" code on others' systems in the 1980s. That it has reared its ugly head again, just speaks volumes about the collective memory chasm in this field and how the rich and powerful and greedy punch down relentlessly even when some survivors, do know better already; and avoid it like the plague.
sigh the distressing part about people having a choice is: invariably, many choose poorly.
@mcc fundamentally it feels like the problem is that most programmers view themselves as inherently logical but this response is emotional. most maintainers just want to avoid any personal friction, and refusing ai contributions is pure friction— it’s just too easy for advocates to spam and brigade any sufficiently high profile projects that deign to take a stand. but it’s embarrassing to publicly, or even consciously, admit you’re afraid of a personal cost
@mcc
Ohhh I'm so sick of seeing this one. Like, yeah, you're right. It could slip in. But you know, maybe *at the very least* reducing 98% of it and shaming the remaining 2% is infact a good idea? At that point it makes dealing with the rest of the problem a whole lot easier.
@mcc The same idiots that spend years arguing with people online about software licenses also somehow simultaneously believe that banning slop is unenforceable. It's truly a mystery.
@mcc this is changing quickly, but you can still "sense" whether something was created by an LLM or not: they completely lack a sense of "elegance". In general terms, their output feels brutish. That's the reason why enforcing such a policy is hard: "the contribution must be elegant" does not translate well.
There are some papers on using deep learning models to output a probability that a piece of code comes from an LLM. I don't know if people who do not want those kinds of contributions are willing to use models to detect them.
@mem a good thing about your criteria is you don't have to worry about false positives. you can just reject stuff like "sorry, this patch is big and the code is not good and you can't clearly answer questions about it"
@mcc paraphrasing:
"we can't have a policy against trolling, derogatory comments, public or private harassment, and other inappropriate conduct[0], because it's an unenforceable policy. there's literally no way of knowing whether soming is inappropriate"
also
"we can't require attribution in our license, because it's unenforceable. there's literally no way to know if someone used our code without attribution, and no way we can stop them from doing it"
[0] contributor covenant code of conduct a.k.a. the "we have a coc" checkbox
@mcc@mastodon.social ah yes, the two genders
@lumi possibly literally accurate
@mcc part of me feels like this is just the extremely perfect-as-the-enemy-of-the-good Software Engineer way of looking at problems. like the VLC guy saying "seeking by one frame backwards is difficult or even intractable on some worst-case video formats, so until we have a solution that always works all of the time in all situations, we will never support the feature ever"
@jk @mcc or like, "actually this is NP hard so there's no reason to even try". Look, that's not what that means, you almost never need an ideal solution!
"Oh no that's basically travelling salesman"
Hey dumbass it's like traveling salesman but the salesman is clark kent and it doesn't matter if they retrace their steps, greedy solutions are fine, just try it.
@jk i would be ok with a back one frame that took a second or so to work. or that printed an error if it detected it was going to take 30 seconds
@mcc I maintain HardenedBSD. I also maintain that HardenedBSD will not accept AI/LLM generated patches.
The only problem we have is that we maintain our patches on top of our upstream (FreeBSD). It's possible that an AI/LLM generated patch would get accepted there and be brought into HardenedBSD that way.
I'm under the (currently unfounded) suspicion that OpenZFS might have already accepted at least one AI/LLM generated patch. Those patches could be brought into FreeBSD, too, which then obviously dominoes into HardenedBSD.
It really is a tough problem. I myself don't want code that is potentially tainted with regards to licensing and IP rights.
@lattera@bsd.network @mcc@mastodon.social scrolling through the history. most of the ones i am seeing only touch code related to tests. i know some people consider that acceptable.
@lattera Yeah, this is why i'm looking at Redox. There is a long list of reasons why Redox is not an adequate replacement for FreeBSD, but for my purposes maybe I can get away with it.
@mcc "Oh, it's not a perfectly enforceable policy so we can't really take a stance on it"
In other news, thieves still exist even though we have laws against theft.