I wonder if at this exact moment one could succeed in filing a issue against an open source project for an entirely ficticious vulnerability, get it assigned a CVE and a github score, and then get the upstream project to accept your patch for the vulnerability, with all these things presented as the issue, the CVE, and the PR being written by generative AI, but in fact you wrote them substantially by hand and the patch for your nonexistent vuln is actually a backdoor
2 Boosts
1 Like
@mcc Do submitters of vulns include patches in their submissions? Don’t projects test patches before building packages?
And the real question: I feel it would take one maintainer saying “I received this patch from a reporter, is it a minimal fix and sound and safe? Kick the tires” to an agent and it would immediately find the backdoor?
@jason Right. So all you need to do is find a project where the maintainers are so overwhelmed by LLM patches they have given up on paying attention and just ask their LLM to check the patches in the queue, or else bomb a project with mashed potatoes LLM-generated patches until the project you're interested in becomes such a project
@jason As for the submitter submitting the patch— I don't know what would be most realistic (remember, we're not looking for good practice, we're looking for what would fool a tired maintainer) but presumably the attacker could submit the vuln report from one account and the patch from another. It's even easier than it used to be to build up the dummy account as they can use LLMs to give it a history of mashed potatoes patches
I'm pretty sure one of the "four freedoms" was the ability to read and inspect the source code. You'd think obfuscated source code, which as far as I'm concerned LLMs produce, is incompatible with that.
https://social.chinwag.org/@FediThing/117571935630906057
It's hard to tell of course, because if you Google "four freedoms" you get dozens of results about a speech FDR made in 1941. Open source people just say "four freedoms" all the time as if they expect you to already know what that is. I think it's some kind of shibboleth
I'm so sick of people trying to fear-monger me about the torment nexus.
@mcc One of my issues with the four freedoms is that they outline a situation that only ever need be hypothetical. Sure, you're free to read and modify the source code, but people mostly don't. The bigger and more complicated the code gets, the bigger the lift for exercising those freedoms. That was true long before anyone put LLMs into the mix, but LLMs intensify the negative pressures that discourage people from applying those freedoms as more than merely hypothetical.
@lrhodes I would go in the opposite direction, and conclude that if the goal of open source is to make it possible to inspect and modify the source code, then *open source is failing at its ideals* unless the code is well-commented, has included structural documentation that allows a third party to find the relevant section of code for a particular task quickly, and has an easy-to-use, cross-platform build/deploy script included which itself is well documented.
@mcc Ideally, I think that would be the case. But freedom is written as a sort of negative ideal in the document. It guarantees that you're free to do those things, but doesn't guarantee that you're able. And the fact of the matter is that a movement based around the freedom to modify code will always be limited to those who are able to code. FOSS isn't really built for everyone. Code that is open but inaccessible only narrows an already narrowly limited scope.
@lrhodes I believe it is possible to structure programming in such a way that anyone or nearly anyone is able to do it. I think that LLMs move us in the opposite direction from that ideal by introducing something that is not programming but does some of the same things, if you keep pumping quarters into the slot machine
@mcc at least one paper published last year in a computer science conference could assess that LLM evaluation can easily be gamed if you use LLM yourself.
Is it open source any more if no one reads and/or understands the source code? 🤔
@mcc "well, i can't find this vuln, but AI is so much better at finding them these days..."
@mcc I imagine if such a submission were to exist, it'd focus on very esoteric code, like out of memory handling and longjmp, to introduce the trojan vulnerability.
Very probably the more your submission appeared to be written by generative AI, the more likely it would be to be accepted. If some person came in with an ungainly, unreadable patch and was aggressively pushing a project to accept it you'd get suspicious and wonder why but if an AI did that you'd be oh well AIs are just like that and AI written code is just like that