I mean this in a very specific, philosophical way.

In 2005, prominent philosopher Harry G Frankfurt published On Bullshit, which establishes a theory of, well, bullshit.

To Frankfurt, truth and falsehood, and therefore truth telling and lying, are on the same axis: a liar presumes to know what the truth is, and endeavors to replace it with a falsehood.

Bullshit, on the other hand, is a completely different dimension. The bullshitter is completely unconcerned with truth or falsehood. Bullshit can be truthful, or it can be false, but this is of no consequence to the bullshitter. The bullshitter is not trying to convey facts nor intentionally spread falsehood; their project is completely different: it aims to create a certain reaction or belief in the listener, and truth is completely orthogonal to this project.

LLMs do not, by their nature, "know" truth or falsehood. They contain (extremely sophisticated) statistical models of what tokens are likely to follow other tokens. Sometimes, this produces true statements. Sometimes, it produces false statements. Sometimes, it produces code and proofs and other things that have some sort of practical value.

But, no matter how large you grow them, how well you tune them to improve the probability of true statements, they are still, fundamentally, producing bullshit.

https://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_bullshit.pdf