it's time to play everyone's favorite game, "Allison Once Again Tries To Use An LLM For Research Because The World Is Constantly Gaslighting Her Into Thinking That LLMs Are Suited To That Purpose, Only To (Once Again!) Have The LLM Confidently Spew Misinformation Until Explicitly Called Out"
0 Boosts
0 Likes
i really, really don't get it. the topic i'm researching here isn't *all* that obscure—it's like a semi-expert level topic, not that far from the kinds of questions i'd expect academics to research in their own fields—and not only is the LLM incapable of producing an accurate answer to the question, it generates one confident lie after another in what looks like an attempt to cover up its tracks. how can so many otherwise reasonable-seeming people use tools like this for research? it's baffling
@aparrish it's nuts to me too
(this is the "gemini 3.6 thinking" model btw, which i have access to through nyu, for some god forsaken reason)
i am being told that if i use a newer model then the meaning will, somehow, magically inhere in the text, and facts that have yet to be uncovered will spontaneously burst into the light, propelled by magnificent statistical weights alone
@aparrish My read is that the “trick” here is what makes newer models “more accurate” is mostly that they’re being fed more comprehensive training data, making the statistical inference more likely to result in a correct response. Asking anything outside of the “comfort zone” of any models training still is going to result in confident guessing.
Agreed that spending trillions to pretend this isn’t the case seems pretty foolish.
@njc i just can't imagine a situation where having a response that looks correct from a statistical model actually obviates the work of finding out whether or not the response is true. and honestly i can't be dissuaded from the belief that any question, as actually asked by an individual person, is necessarily embedded in contexts (physical, social) that are not just outside the "comfort zone" of statistical models, but impossible to capture with statistical models at all
@aparrish I’m not really defending the approach here, but I agree that a statistically likely but inaccurate answer to a question, presented as fact is significantly worse than an “I don’t know” (though tbf you’re a lot more likely to get an “I don’t know” from newer models now)
But also, LLMs do carry the concept of context forward both between chats (if enabled) and within chats, and while it’s not comparable to like.. a mentor or peer, it’s a lot more context than a stranger or search result would typically carry. And I do think there are a decent amount of questions for which a statistically likely answer is essentially identical to a correct one.
I think for any sufficiently advanced subject, where training data is likely to be limited, and nuance is likely to be important, LLMs lose a lot of their efficacy, but for “build me a function/query that does X” in some well trained / established language (and where you can verify the output), they tend to be fine, and typically better than a lot of alternatives.
I don’t personally think that justifies the ongoing investment, and certainly doesn’t earn many of the other costs that coincide with the aggressive rollout of AI (ecological, labor, etc.) But I also don’t think people that are finding them useful are necessarily dealing with (or not noticing) frequent hallucinations anymore.
@njc in any of those cases, at some level, for someone to believe that LLMs reduce the amount of work involved in a task, they have to trust the output of the LLM to be correct, or apply some criteria for "correctness" that aren't as time-consuming as their typical criteria. that's the gap that I don't understand. a researcher *knows* what it means to verify whether or not a claim is factual. they just *choose* to suspend that rigor, and roll the dice instead, when it comes to LLM output
@aparrish which environment is this in? i don't use gemini much but this seems off as far as current model generations go. I'm happy to give it a go if you give me your starting prompt / intent. contrary to (im)popular opinion, there is a (more or less steep) learning curve.
@aparrish 3.6 is fairly old. I recommend trying 3.8-flash in Google AI Studio or something like deepseek. Or limited models like Anthro/OAI/etc
@aparrish it's so weird that technology has advanced to the point where we can make a word prediction algorithm that talks about lying and to no other point
(i'm trying to figure out whether resetting the div counter on the game boy also resets the phase of the serial clock pin, and each time i asked it to give me an actual source for its conclusions, it just dug itself deeper and deeper into made up stuff)