Why Do They Lie?

You've probably noticed that the language models you use sometimes produce faulty outputs. To put it more simply: that they make things up, that they lie. The technical term for this is hallucination.

A paper by Adam Tauman Kalai from OpenAI, published on arXiv, explains beautifully why this happens and why they can't be free of lying. In short:

Language models don't actually know what is what; they work on the logic of predicting the next token in the answer they'll produce. In the process, if they're answering about a topic they saw little of in the training data, they start making errors on every “next word/token” prediction, and it carries on that way. If it's a topic they never saw, they can make it up entirely.

If we ask for an answer about something it saw little of, or not at all, it is forced to guess — and hallucination becomes inevitable.

The “Missing Mass” problem

The paper describes this as the “Missing Mass” problem. It explains that expecting the model to produce the answer everyone thinks of first — “I don't know” (IDK for short) — isn't a definitive solution either. Because the model can't be perfect at knowing “whether it knows something.” So adding an IDK mechanism to the model isn't a definitive fix.

It doesn't look possible for RAG and similar approaches, different and better model architectures, or a reward/penalty-based application of IDK — any one of these or all of them together — to reduce hallucination to zero.

So language models aren't systems that produce hallucination — they're systems that are compelled to produce it.

Which means it falls to the user

However well the models are engineered, an important part of the job falls to the users. Designing prompts and system messages correctly, and not falling into “blind approval syndrome” with model outputs, matters.

At the user level, anyone dealing with these models would do well to give some thought to this thing called “prompt engineering.”