LLM question: is the result deterministic with the same model and prompt?
I guess this also assumes they haven't changed any of the system prompts between prompts. There is probably no way of knowing that, is there?
0 Boosts
0 Likes
@Scmbradley @jaranta Yeah, that's what I understand as well--and about the same time, too. But it also makes sense if you're trying to make a chatbot sound more human. Ask a human the exact same question, you'll get answers that are worded differently each time.
Without a randomizing element (called "temperature" IIRC), the illusion of humanness would be broken because you'd get exactly the same response all the time with the same prompt. And folks won't pay for that.
@Scmbradley @jaranta Found this page which is a reasonably intro-friendly example (you can skip the code parts if you're not into that and just look at the text and examples). https://dylancastillo.co/posts/seed-temperature-llms.html
But the tl;dr is that lower temperature settings produce more deterministic output, and higher temperature produce more random outputs. And there's a "seed" which, like in many games, impacts the overall pattern of randomization.
@jaranta @michaelc @Scmbradley Right and if you are either self hosting / at least have API access to an LLM inference engine you can set temperature to zero, to make it deterministic for a given model.
Since I am doing a workshop on this, I just tried thrice with the same prompt and got the same response back.
@Scmbradley @jaranta The answer is no. Models are inherently stochastic. A parameter called temperature adjusts the degree of randomness, afaict it's never zero.
@jaranta I'm pretty sure the answer is no. It's not a case of always picking the highest scoring token, but some kind of random selection from high scoring tokens. But my knowledge dates from 2023ish, so take it with a pinch of salt.