lolll you know how the new line is "but the bobot is super good at refactoring actually"
WELL IT FUCKING TURNS OUT,
1 Boost
0 Likes
@chaekyung @davidgerard I think it's a nice study design, but it's limited to a single language, to the specific case of upgrading language version compatibility, and also the paper appears to be heavily AI-written. I, too, wonder how reliable these results will be.
@chaekyung @davidgerard A point worth noting is that the setting they describe, i.e. copy pasting a single file with no context into a chatbox and then the model having no access to an execution environment, is not representative anymore. Even if you don't use their "coding" environments, these days both OpenAI and Anthropic's offerings will spin up a VM and let their models test their code even if you prompt them through the web chat box interface.
Note that the limited selection of (smaller) models they chose might be a resource limitation. All the authors are from Bangladesh. Even here in Germany I can say that from experience, getting a university to pay for LLM access is difficult to impossible.
@jaseg @chaekyung you're coming up with a list of objecitions all of which are obviously answered "so run it yourself with all that, the design's in front of you"
@chaekyung i'm really not accepting "but old models" unless someone repeats the work with the new ones and demonstrates it makes a dot of difference
@davidgerard@circumstances.run @chaekyung@ieji.de These observations are exactly what you'd expect given how LLMs work. David's should be the default hypothesis. Any argument to the contrary needs to marshal serious evidence in favor, which so far has been lacking as far as I can tell.
The bumper sticker version is "extraordinary claims demand extraordinary evidence". The claim that stochastic next token predictors can successfully refactor rules-driven source code better than a rules-driven tool is extraordinary.
The bumper sticker version is "extraordinary claims demand extraordinary evidence". The claim that stochastic next token predictors can successfully refactor rules-driven source code better than a rules-driven tool is extraordinary.
@davidgerard it will only get better though, as they cut up more old books of php code
@davidgerard I'm sure they're just prompting it wrong and that's the issue.
@davidgerard The models they tested are Claude Sonnet 4, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-5 Codex, Llama 3.3 70B. All of them are made by the Americans, and all of them are old and outdated. It's kind of odd that the good people of Bangladesh, who did this research, did not include a single one of the much smaller and vastly superior French and Chinese models or newer versions of the overly bloated models made by fat lazy Americans. That aside, there seems to be a very important lesson here:
Rector, a very specialized tool specifically created to refactor older PHP code, did a _much_ better job than all the "general-purpose" large language models. And not only did Rector do a much better job, it did a better job while being magnitudes smaller and magnitudes more efficient. It's like comparing a screwdriver and a 9 axis CNC machine the size of a small house: The LLMs in this test really do require a huge data center larger with a power draw bigger than your average town.
I've never actually used a software as a service LLM (or anything else "AI" for that matter) so there's a slim chance I'm wrong here, but based on this study: if there's existing normal software designed to do a job then it'll probably do the job much better than modern LLMs can do.