Why LLMs Hallucinate (and What Actually Helps)

Ask a language model for a paper on a niche topic, and it may hand you a title, three authors, a journal and a year. All of it sounds perfect. None of it exists. This is hallucination, and it is not a rare glitch. It falls out of how these models are built.

• Fluent
• Specific
• Completely invented

Last episode we saw why decoding is memory-bound and how the KV cache keeps generation fast. Today we ask a different question: why does a fast, fluent model say false things? We will cover the causes, work a numerical example, look at the fixes that actually help, and write a small Python check.

• Last time: the KV cache
• Today: why models make things up
• Then: what helps

By hallucination, we mean output that reads fluently but is false, unsupported, or unfaithful to the given source. The key feature is that nothing in the tone warns you. A correct answer and a fabricated one can look identical on the page, and that is exactly what makes the problem dangerous.

• Fluent but unfaithful or false
• Stated with confidence

Here is the core intuition. Imagine a student who has read an enormous library, and who is always graded on how natural their next word sounds. Never on whether it is correct. When asked something they do not know, the student keeps talking smoothly, because smooth is what earned the reward.

• Rewarded for plausible text
• Not rewarded for truth

Recall from the first episode that pretraining minimizes cross-entropy on next tokens. Equivalently, it maximizes the log probability the model assigns to the real next token in human text. Notice what that objective measures: how well the model matches the text distribution. Whether a sentence is true appears nowhere in the loss.

• Minimize cross-entropy
• Maximize likelihood of real text

Truth does get in, but only indirectly. In human writing, true statements tend to be more common than false ones, so a model that imitates text well often reproduces facts. But that is a correlation, not a guarantee. When the data is thin on a topic, the model still produces something shaped like an answer.

• True text is common in data
• So truth is learned indirectly

A note on rigor. It is well supported empirically that models make more errors on rare facts, and that retrieval and verification reduce errors. What is still partly hypothesis is the precise internal mechanism, for example why a model fails to refuse in one case and succeeds in another. I will flag which is which as we go.

• Supported: models err on rare facts
• Hypothesized: exact internal mechanisms

Let us make this concrete with numbers. Suppose we ask: the lead author of an obscure 2009 paper on soil bacteria is Doctor, and the model must predict the surname. The model has seen this paper rarely or never. Yet it must still output a probability distribution over the next token.

• Prompt: lead author of an obscure 2009 paper
• Model must pick a surname

Here is a plausible distribution. Anderson gets zero point two two. Chen gets zero point one eight. Martin, point one five. Patel, point one two. Kim, point one. The true author, Okafor, gets only point zero four, and all other names share the remaining point one nine. These sum to one.

• Anderson 0.22, Chen 0.18
• Martin 0.15, Patel 0.12, Kim 0.10
• Correct name Okafor 0.04
• Everything else 0.19

Greedy decoding picks the highest bar, Anderson, which is wrong. And notice something subtle: no wrong name is individually likely, but together the wrong names hold ninety six percent of the mass. The model is uncertain, yet the output is one clean name, with no hint of that uncertainty.

• Greedy picks Anderson
• Correct mass is only 4 percent
• Wrong mass is 96 percent

Now sample at temperature one. Each run draws from this distribution, so Okafor appears about four times in a hundred, and Anderson about twenty two. Run the same prompt twice and you can get two different confident names. That instability across runs is itself a clue, and we will exploit it later.

• Temperature 1: sample from p
• Okafor appears 4 times in 100
• Different answer each run

What does lower temperature do? At temperature zero point five, probabilities are effectively squared and renormalized. Among the six named candidates, Anderson rises to about thirty seven percent, while Okafor falls to about one point two percent. Lower temperature makes outputs more consistent, but here it makes the wrong answer more consistent, not right.

• Temperature 0.5 squares the probabilities
• Renormalize among the six names
• Anderson 37 percent, Okafor 1.2 percent

The first cause is the long tail. Facts about famous people appear thousands of times in training data, so the model learns them robustly. Facts about obscure people, papers and local events appear once or never. Research has found that factual accuracy tracks how often a fact shows up in the data, so rare facts are where hallucinations cluster.

• Rare facts appear few times
• Little signal to memorize

The second is calibration. A calibrated model that says it is seventy percent sure should be right about seventy percent of the time. Studies have reported that pretrained models are often reasonably calibrated on some tasks, while later tuning for style and helpfulness can make them sound more certain than they should. The exact extent varies by model and task.

• Calibrated: 70 percent confident, right 70 percent
• Post-training can distort this

The third cause is exposure and snowballing. Generation is autoregressive: every new token conditions on all the tokens before it, including the model's own earlier mistakes. Once a wrong claim is in the context, the most plausible continuation is to stay consistent with it. One early slip then pulls the rest of the answer further from the truth.

• Each token conditions on the last
• Early error becomes context
• Model defends its own mistake

The fourth cause is what we reward. Think of a multiple-choice exam with no penalty for wrong answers: guessing always beats leaving it blank. Many benchmarks and preference signals work this way, and researchers have argued this pushes models to answer rather than abstain. This is a plausible, partly supported explanation, not a fully settled one.

• Benchmarks grade only accuracy
• Guessing beats abstaining
• Tuning may favor confident style

The fifth cause is stale knowledge. A model learns from data up to a cutoff date, then is used for months or years afterwards. Ask about a recent release, a new law, or a current price, and it may answer from outdated patterns, with the same confident tone. Weights are a snapshot, not a live feed.

• Training data has a cutoff
• World keeps changing
• Model may still answer as if current

Researchers separate two types. Intrinsic hallucination contradicts the material you gave the model, for example summarizing a report that says revenue fell as saying it rose. Extrinsic hallucination adds content that the source never mentioned and cannot verify. Telling these apart matters, because the fixes differ.

• Intrinsic: contradicts the source
• Extrinsic: adds unsupported content

Fabricated citations are the famous case. A citation has a very regular shape: surname, initial, year, title, journal. The model has seen millions of them, so it can generate the shape perfectly, while the specific content is a blend of real fragments. People have been embarrassed by submitting such references, including in legal filings.

• Plausible authors, titles, years
• Real format, fake paper

Why are citations so prone to this? The format tokens are highly predictable, so the model is fluent and confident throughout. The specific details, though, are exactly the rare long-tail facts from earlier. Fluency on the surface hides uncertainty underneath, and that is why you should always verify references by looking them up.

• Format is highly predictable
• Content is rare and specific
• Looks right, checks wrong

Now the good news: several techniques measurably reduce the problem. They fall into three jobs: give the model better information, change what it is allowed to say, and check what it produced. None is a silver bullet, so production systems usually stack several of them.

• Reduce, detect, contain
• No single silver bullet

The most important fix is grounding. Instead of asking the model to recall a fact from its weights, retrieve relevant documents and put them in the prompt, then instruct it to answer only from that text. This moves the task from memory to reading comprehension. We will build this fully in the RAG episodes, but note it also reduces staleness.

• Fetch relevant documents
• Put them in the prompt
• Answer from the evidence

Second, make abstaining acceptable. Tell the model it may say it does not know when the context lacks the answer, and give it a clear way to do so. At training time, evaluations that score honest abstention above a confident wrong answer push models in the right direction. The trade-off is that over-cautious models refuse questions they could have answered.

• Instruct abstention explicitly
• Offer an unknown option
• Reward abstaining over guessing

Third, lower the temperature. This reduces the random draws that land on rare wrong tokens, and it is a sensible default for factual tasks. But as our example showed, it cannot supply knowledge the model lacks. If the most likely answer is wrong, a lower temperature simply makes that wrong answer more stable.

• Reduces random wandering
• Does not add knowledge
• Wrong answers become repeatable

Fourth, add verification. Have the model draft an answer, list the factual claims in it, and check each claim against sources, ideally in a separate call. Require citations that point to the retrieved passages, and then verify in code that each quoted span really appears in the source. Checking a quote is far easier than recalling a fact.

• Draft, then check claims
• Ask for quotes from sources
• Verify links and quotes in code

Fifth, use tools. Models are poor at exact arithmetic, current prices, and database lookups, but they are good at deciding to call a calculator, a search engine, or an SQL query. Delegating exact work to a tool replaces a guess with a computed result. We will return to this in the tool calling episode.

• Calculator for math
• Search for current facts
• Code or database for exact data

Now a practical detector. Recall that wrong answers from guessing tend to vary between samples, while well-known facts come back the same each time. So ask the same question several times at a nonzero temperature and measure how often the answers agree. High agreement is a hint of reliability, and low agreement is a warning.

• Sample the same question many times
• Stable answers suggest knowledge
• Scattered answers suggest guessing

Here is the code. We assume a function called llm that takes a prompt and a temperature and returns a string, so you can wrap any provider. We call it n times, normalize each answer by lowercasing and stripping punctuation, count them with a Counter, and return the most common answer plus its share of the votes. If the share is below a threshold, such as zero point six, we flag the answer as uncertain.

• Provider-neutral llm function
• Sample n answers
• Return top answer and agreement

Run it on the capital of France, and ten of ten samples say Paris, so agreement is one point zero. Run it on our obscure paper, and say the samples give Anderson three times, Chen twice, Martin twice, and three singletons. The top answer has agreement zero point three, below our threshold, so we flag it instead of trusting it.

• Paris: 10 of 10, agreement 1.0
• Okafor question: top answer 3 of 10
• Agreement 0.3, flag it

Be honest about the limits. A model can be consistently wrong, as in our low-temperature example, and then the check passes a false answer. Comparing long free-form answers needs semantic matching rather than exact strings. And you pay for n calls. Treat agreement as a useful uncertainty signal, never as proof of truth.

• Consistent errors pass the check
• Free-form text is hard to compare
• Costs n times more calls

The biggest mistake is believing hallucination can be fully eliminated. Because the model is a probabilistic generator trained on likelihood, some chance of a plausible wrong output always remains. Grounding, checks and tools shrink the rate, but they do not make it zero. Design the product so that residual errors are caught, disclosed, or low-stakes.

• Reduce, do not eliminate
• Task design sets the risk
• Plan for residual errors

A few more traps. Trusting a confident tone, when fluency tells you nothing about truth. Treating temperature zero as a cure. And assuming retrieval solves everything, when the model can still ignore or misread the retrieved passages, or retrieval can return the wrong documents. Always evaluate end to end on your own questions.

• Trusting confident tone
• Using temperature zero as a fix
• RAG without checking the answer

Let us recap. Models are trained to be plausible, and truth enters only indirectly. Hallucinations come from rare facts, imperfect calibration, snowballing, rewards that favor guessing, sampling randomness, and stale knowledge. What helps is grounding, permission to abstain, lower temperature, verification, citations, tools, and agreement checks. And it can be reduced but never eliminated.

• Objective: likelihood, not truth
• Causes: tail, calibration, snowball, rewards, staleness
• Fixes: ground, abstain, verify, tools

That closes our section on LLM foundations. From training to scaling, sampling, context, the KV cache, and now hallucination, you know how these models work and where they fail. Next episode starts prompting: prompt patterns and system prompts, the tools you control directly.

• Section 1 complete
• Next: prompt patterns and system prompts
• Subscribe for the series