I wonder if at this exact moment one could succeed in filing a issue against an open source project for an entirely ficticious vulnerability, get it assigned a CVE and a github score, and then get the upstream project to accept your patch for the vulnerability, with all these things presented as the issue, the CVE, and the PR being written by generative AI, but in fact you wrote them substantially by hand and the patch for your nonexistent vuln is actually a backdoor
1 Boost
1 Like
@mcc Do submitters of vulns include patches in their submissions? Don’t projects test patches before building packages?
And the real question: I feel it would take one maintainer saying “I received this patch from a reporter, is it a minimal fix and sound and safe? Kick the tires” to an agent and it would immediately find the backdoor?
@jason Right. So all you need to do is find a project where the maintainers are so overwhelmed by LLM patches they have given up on paying attention and just ask their LLM to check the patches in the queue, or else bomb a project with mashed potatoes LLM-generated patches until the project you're interested in becomes such a project
@jason As for the submitter submitting the patch— I don't know what would be most realistic (remember, we're not looking for good practice, we're looking for what would fool a tired maintainer) but presumably the attacker could submit the vuln report from one account and the patch from another. It's even easier than it used to be to build up the dummy account as they can use LLMs to give it a history of mashed potatoes patches
I'm pretty sure one of the "four freedoms" was the ability to read and inspect the source code. You'd think obfuscated source code, which as far as I'm concerned LLMs produce, is incompatible with that.
https://social.chinwag.org/@FediThing/117571935630906057
It's hard to tell of course, because if you Google "four freedoms" you get dozens of results about a speech FDR made in 1941. Open source people just say "four freedoms" all the time as if they expect you to already know what that is. I think it's some kind of shibboleth
I'm so sick of people trying to fear-monger me about the torment nexus.
@mcc at least one paper published last year in a computer science conference could assess that LLM evaluation can easily be gamed if you use LLM yourself.
Is it open source any more if no one reads and/or understands the source code? 🤔
@mcc "well, i can't find this vuln, but AI is so much better at finding them these days..."
@mcc I imagine if such a submission were to exist, it'd focus on very esoteric code, like out of memory handling and longjmp, to introduce the trojan vulnerability.
Very probably the more your submission appeared to be written by generative AI, the more likely it would be to be accepted. If some person came in with an ungainly, unreadable patch and was aggressively pushing a project to accept it you'd get suspicious and wonder why but if an AI did that you'd be oh well AIs are just like that and AI written code is just like that