"LLMs are not useful for finding vulnerabilities": well...

i do tend to agree with the argument that if we'd spend this much money on static analysers and fuzzers, we'd probably have a much more useful set of vulnerability detection tools that don't require entire datacentres to run.

but that doesn't mean LLMs are useless, just inefficient. i've been using an LLM to audit some of my own (non-FreeBSD) code this week and while doing that, it found two separate, real vulnerabilities in FreeBSD; one had previously been reported but not fixed yet, the other was novel.

i wasn't even trying to audit FreeBSD, it just found these issues because the actual behaviour of my code didn't match the expected behaviour, and it determined that the bug wasn't in my code.

obviously, as a FreeBSD developer i was able to analyse these and confirm they are real bugs before submitting them, and i don't think people should be spamming maintainers with ignorant and useless bug reports. i wonder if static analysers would be any better there, though... Coverity reports a lot of false positives.