I wonder if at this exact moment one could succeed in filing a issue against an open source project for an entirely ficticious vulnerability, get it assigned a CVE and a github score, and then get the upstream project to accept your patch for the vulnerability, with all these things presented as the issue, the CVE, and the PR being written by generative AI, but in fact you wrote them substantially by hand and the patch for your nonexistent vuln is actually a backdoor
5 Boosts
1 Like
@mcc Possibly, although it would probably take some underhanded coding. I would, potentially, recommend studying former IOCCC submissions for ideas, should one want to go that way.
@mcc i could see that. The upstream repo maintainers might let an llm tell them that the patch is good, so the patch should include an assurance that this is not a backdoor
@mcc it would be really interesting to actually try it into several projects then tell ask them to remove it in case they accepted. After getting the responses from all tried projects, we could work with that data to see how vulnerable projects are on average, how many accepted it, etc. It would b ea nice paper
@perdidonavida well, I'm not doing it because it would probably involve at some point running an LLM. I would consider saying "you should try it!" but I think that would constitute in writing encouraging someone to commit a crime so I'm not going to say that
@mcc Is it ethical to use llms to prove there are problems with the use of llms? That's a great question. I'm inclined to try such things, but I admire your position.
Anyway, if someone reading this thinks it's worth trying and wants some help, maybe we can work together. I will say in advanced that there's a serious stretch of scientific ethics here that should be well evaluated before, that might make the idea not viable at all
Very probably the more your submission appeared to be written by generative AI, the more likely it would be to be accepted. If some person came in with an ungainly, unreadable patch and was aggressively pushing a project to accept it you'd get suspicious and wonder why but if an AI did that you'd be oh well AIs are just like that and AI written code is just like that
@mcc Do submitters of vulns include patches in their submissions? Don’t projects test patches before building packages?
And the real question: I feel it would take one maintainer saying “I received this patch from a reporter, is it a minimal fix and sound and safe? Kick the tires” to an agent and it would immediately find the backdoor?
@jason Right. So all you need to do is find a project where the maintainers are so overwhelmed by LLM patches they have given up on paying attention and just ask their LLM to check the patches in the queue, or else bomb a project with mashed potatoes LLM-generated patches until the project you're interested in becomes such a project
@jason As for the submitter submitting the patch— I don't know what would be most realistic (remember, we're not looking for good practice, we're looking for what would fool a tired maintainer) but presumably the attacker could submit the vuln report from one account and the patch from another. It's even easier than it used to be to build up the dummy account as they can use LLMs to give it a history of mashed potatoes patches
I'm pretty sure one of the "four freedoms" was the ability to read and inspect the source code. You'd think obfuscated source code, which as far as I'm concerned LLMs produce, is incompatible with that.
https://social.chinwag.org/@FediThing/117571935630906057
It's hard to tell of course, because if you Google "four freedoms" you get dozens of results about a speech FDR made in 1941. Open source people just say "four freedoms" all the time as if they expect you to already know what that is. I think it's some kind of shibboleth
@mcc Yes. #SALAMI cannot be called "open" (or, better, free) without the availability of the training data. Here is why: https://garymarcus.substack.com/p/open-source-is-not-the-same-as-open
@mcc
They must have been intentionally parodying FDR when they invented them. The four freedoms were a huge deal. FDR put them in at the end of a state of the union and was shocked at how much they took off. They're considered the core jumping off point for the process that led to modern international human rights law and institutions. People seriously talked about adding freedom from want and freedom from fear to the constitution (the other two were, freedom of speech and freedom of worship).
@mcc as someone suggested (cannot find the toot, sorry), the C or Python or whatever is not the relevant thing. The relevant thing is the "preferred form" for working on the program. So that's the prompts, including all files that were included in the context.
I'm so sick of people trying to fear-monger me about the torment nexus.
@mcc One of my issues with the four freedoms is that they outline a situation that only ever need be hypothetical. Sure, you're free to read and modify the source code, but people mostly don't. The bigger and more complicated the code gets, the bigger the lift for exercising those freedoms. That was true long before anyone put LLMs into the mix, but LLMs intensify the negative pressures that discourage people from applying those freedoms as more than merely hypothetical.
@lrhodes @mcc i agree with this but yet can also imagine that people hold the opposite pov at the same time. That while it was pretty difficult to contribute something to a project you loved and used before if it was nontrivial in size, now you can “just” vibe code up a fix for that annoying bug that never gets fixed or that feature you want. So I can imagine people feel LLMs make the source code *more* open not less. Not that I want their patches that they don’t even understand but yeah.
@lrhodes I would go in the opposite direction, and conclude that if the goal of open source is to make it possible to inspect and modify the source code, then *open source is failing at its ideals* unless the code is well-commented, has included structural documentation that allows a third party to find the relevant section of code for a particular task quickly, and has an easy-to-use, cross-platform build/deploy script included which itself is well documented.
@mcc at least one paper published last year in a computer science conference could assess that LLM evaluation can easily be gamed if you use LLM yourself.
Is it open source any more if no one reads and/or understands the source code? 🤔
@mcc "well, i can't find this vuln, but AI is so much better at finding them these days..."
@mcc I imagine if such a submission were to exist, it'd focus on very esoteric code, like out of memory handling and longjmp, to introduce the trojan vulnerability.
@mcc kiiinda related thought:
https://chaos.social/@dngrs/117515470819786409