Chain-of-Thought and Reasoning Models: How LLMs Learn to Think

Ask a language model what seventeen times twenty-four is, and it may blurt out a wrong number. Add the words, let's think step by step, and suddenly it gets it right. Nothing about the model changed. Not a single weight moved. So what actually happened in that extra sentence?

• Same model, same question
• One extra sentence

In the last episode we looked at few-shot prompting and in-context learning, where examples steer the model. Today we go one step further and steer how the model computes. We will cover chain-of-thought prompting, self-consistency voting, a glimpse of tree search, and then reasoning models, the systems trained to think at length before answering.

• Chain-of-thought and voting
• Reasoning models
• Python and pitfalls

Here is the big idea to hold onto. A model's thinking is made of tokens, and every token is a unit of computation you can spend. Writing out reasoning is a way of buying more compute for a hard question. By the end, you will know when that purchase is worth it and when it is wasted money.

• Thinking costs tokens
• Tokens are compute

Recall from the ML course's Transformer lesson that producing one token takes one forward pass through a fixed number of layers. That is a fixed amount of computation, whether the question is trivial or brutally hard. If the answer needs ten sequential steps of logic, but the network only has so many layers, the model has to somehow squeeze them all into a single pass.

• Fixed layers per token
• Hard problems need more

Think about how you multiply seventeen by twenty-four. You do not do it instantly in your head; you grab paper. The paper stores intermediate results so each step is easy. A language model has the same option, because every token it writes is fed back in as input for the next one. The generated text is its scratch paper.

• Paper for the model
• Output feeds back as input

Take seventeen times twenty-four. In direct mode, the model must produce the first digit of the answer immediately. To get that digit right, it has to have already worked out the whole product internally. The correct answer is four hundred and eight, but a single pass often lands on something close and wrong, like four hundred and eighteen.

• Answer emitted immediately
• Must compute in one pass

Now let the model write steps. Seventeen times twenty is three hundred and forty. Seventeen times four is sixty-eight. Three hundred and forty plus sixty-eight is four hundred and eight. Each line is an easy step the model can do in one pass, and each result is now written down where the next step can read it.

• 17 x 20 = 340
• 17 x 4 = 68
• 340 + 68 = 408

The original chain-of-thought idea used few-shot examples. Instead of showing question and answer pairs, you show question, worked reasoning, then answer. The model imitates the pattern and writes its own reasoning before committing to a final answer. This built directly on the in-context learning we saw last time.

• Examples include reasoning
• Model imitates the format

Researchers then found that you did not even need examples. Appending a simple phrase such as, let's think step by step, often nudges a model to produce reasoning on its own. This is zero-shot chain-of-thought. It works because the training data is full of explanations that walk through steps before reaching a conclusion.

• No examples needed
• One trigger sentence

Why does it help? Three reasons. The problem is decomposed into smaller steps that each fit in one pass. Intermediate results are stored in the text, so nothing has to be held internally. And the model gets a longer path in which to correct itself. It is a boost in effective serial computation, and that matters most for multi-step arithmetic, logic and planning.

• Decomposition into easy steps
• Intermediate results stored
• Errors become visible

There is a catch. A chain is only as strong as its weakest link. If each step is ninety-five percent reliable and you have ten steps, the chance that every step is right is about point nine five to the power ten, which is roughly sixty percent. One wrong intermediate result and the final answer is wrong, however confident the prose sounds.

• One bad step ruins the answer
• Longer chains, more risk

Remember from the sampling episode that generation is random when temperature is above zero. So each run produces a different chain of thought, and different chains can end at different answers. That randomness looks like a problem, but it is also an opportunity. If we sample several chains, maybe the answers can vote.

• Sampling gives different chains
• Different chains, different answers

Self-consistency does exactly that. Sample, say, five reasoning chains at a nonzero temperature. Extract each final answer. Then return the answer that appears most often. The intermediate reasoning is thrown away; only the votes count. It trades extra compute for reliability.

• Sample many chains
• Take the majority answer

Let's do the numbers. Suppose each sample is right with probability zero point six, and samples are independent. With five samples, the majority is correct if at least three are right. Exactly three right: ten ways, times zero point six cubed, times zero point four squared, which is zero point three four five six. Exactly four: five times zero point six to the fourth times zero point four, which is zero point two five nine two. All five: zero point zero seven seven six.

• p correct = 0.6 per sample
• 5 independent samples
• Majority needs 3 or more

Add them: about zero point six eight three. So voting over five samples lifts accuracy from sixty percent to roughly sixty-eight percent. With eleven samples, it rises to around seventy-five percent. Notice the diminishing returns: each extra vote helps less than the one before, while the cost grows linearly.

• 0.3456 + 0.2592 + 0.0778
• About 0.683
• Up from 0.6

Here is the fine print. That math assumed independent errors. If the model has a systematic misconception, every sample makes the same mistake, and voting just confirms it with great confidence. Voting helps most when wrong answers are scattered across many values while the right answer keeps recurring. Real samples from one model are only partly independent, so real gains are smaller than the ideal calculation.

• Errors must be independent-ish
• Shared bias defeats voting

Voting also needs answers that can be compared. A number, a multiple-choice letter, or a short label is easy to tally. A long essay is not. For free-form output you need tricks, like asking another model call to pick the most consistent response, but that adds its own noise and cost.

• Works for short final answers
• Hard for free-form text

Self-consistency samples whole chains independently. Tree-style methods go further and branch inside a chain. At each step the model proposes several candidate next thoughts, something evaluates how promising each looks, and weak branches are pruned. It is search, like a chess engine, with the language model proposing moves and judging positions.

• Branch at each step
• Score partial progress
• Prune bad branches

Tree search shines on puzzles and planning, where early choices can doom a path and backtracking matters. The price is many model calls per question and a need for a decent way to score partial progress. It is a powerful idea, and as we will see next, it set the stage for models that learn this kind of exploration on their own.

• Good for puzzles and planning
• Many model calls per question

Chain-of-thought as a prompt is a trick applied from outside. Reasoning models take the opposite approach: they are trained so that long, careful thinking is simply how they answer. Before the visible reply, they produce a thinking trace, often thousands of tokens long, where they plan, try approaches, check work and backtrack.

• Prompted thinking is optional
• Trained thinking is built in

How are they trained? With reinforcement learning on verifiable rewards. Take problems whose answers can be checked automatically, like math with known solutions or code with unit tests. The model samples many attempts. Correct final answers earn reward one, wrong ones earn zero, and the training update makes the traces that led to success more likely. No human has to grade each step.

• Sample many attempts
• Reward correct final answers
• Reinforce the winning traces

A notable finding from this style of training is that useful habits tend to emerge: thinking gets longer on hard problems, the model double-checks its arithmetic, and it abandons dead ends. These behaviours are rewarded only indirectly, because they raise the chance of a correct final answer. We will cover the policy optimisation machinery itself in the RLHF and DPO episodes later in the course.

• Longer thinking
• Self-checking
• Backtracking

Remember the scaling laws episode: more parameters, data and training compute improve a model. Reasoning models add a second axis, test-time compute. For many verifiable tasks, letting the model think longer tends to improve accuracy, at least up to a point. Instead of only building a bigger model, you can spend more at inference on the problems that need it.

• More thinking tokens, better answers
• A second scaling axis

In practice, providers expose a control for this, often called a thinking budget or reasoning effort. A low setting caps the trace, a high setting lets the model roam. The knob varies by provider, so check your API's documentation. Thinking tokens are typically billed as output tokens, and they are often not shown in full, so your bill can be much larger than the visible reply suggests.

• Cap or set effort level
• Trade quality for cost

Here is a simple illustration with made-up round numbers. A standard model answers in five hundred output tokens. A reasoning model thinks for seven thousand five hundred tokens, then answers in five hundred: eight thousand total. That is sixteen times the output tokens. Since decoding is memory-bound and sequential, as the KV cache episode showed, latency grows roughly in proportion too. Seconds can become minutes.

• 500 vs 8,000 output tokens
• Time scales with tokens

So when should you use which? Reasoning models earn their cost on multi-step math, tricky debugging, planning, and problems with a checkable answer. Standard models are better for classification, extraction, summarising, rewriting, simple question answering and anything interactive where users are waiting. A good rule: start with the cheap model, and escalate only where it fails.

• Reasoning: math, code, planning
• Standard: lookup, rewriting, chat

Let's test this ourselves in Python. We write one wrapper, call_llm, that takes a prompt and a temperature and returns text. Inside, you plug in whichever provider you use. Keeping the rest of the code provider-neutral means the experiment is easy to rerun on different models.

• One function, provider-neutral
• Swap any API inside

We build a small set of arithmetic problems, such as multi-digit products and sums with several terms, each with a known answer. The direct prompt says: answer with only the number. The chain-of-thought prompt says: think step by step, then finish with a line that starts with Answer colon. A small parser pulls out the number after that marker, and we compare it with the truth.

• Small arithmetic set
• Direct: answer only
• CoT: reason, then 'Answer:' line

For self-consistency, we call the chain-of-thought prompt, say, five times at a temperature around point eight, parse each answer, and use a Counter to find the most common one. Ties can be broken by picking the first seen or by sampling more. Always set a cap on samples, because every extra call costs money and time.

• n samples at temperature 0.8
• Counter for majority vote

What will you see? It depends on the model. A strong modern model may already ace small arithmetic without any chain-of-thought, and a reasoning model thinks internally either way. Weaker models often benefit more. Results vary between runs and tasks, so I am not claiming any benchmark figures here. Run it on your own model, with enough problems to see past the noise.

• Results vary by model and run
• No benchmark numbers claimed

The first common mistake is adding chain-of-thought to everything. For a simple lookup or a sentiment label, extra reasoning adds tokens, latency and cost, and sometimes talks the model out of a correct instinct. Worse, if you pair a reasoning model with a prompt that says think step by step, you may be paying twice for the same thing. Test whether it helps before you ship it.

• Simple tasks gain nothing
• Costs tokens and time
• Can add errors

The second mistake is treating the written reasoning as a faithful explanation of what the model actually did. Research on this has found cases where stated reasoning omits factors that influenced the answer, or is a polished story built after the fact. The text is generated like any other text. It can be useful, but it is not a window into the network's true computation.

• Trace may not reflect true computation
• Plausible is not faithful

So what should you do? Verify outputs independently. Run the code against tests, recompute the numbers with a calculator tool, and check claims against sources. Be careful about presenting a reasoning trace to users as proof of correctness. A confident chain of logic can still end in a wrong place, and traces can also contain content that was never meant for end users.

• Check the final answer
• Use tests and tools
• Don't show traces as proof

Let's recap. Generated tokens act as scratch space, giving a fixed-depth model more serial computation. Chain-of-thought prompting, few-shot or zero-shot, taps that. Self-consistency votes across chains and helps when errors are scattered. Reasoning models are trained with reinforcement learning on verifiable rewards and scale with test-time compute, at a real cost in tokens and latency. And never confuse a trace with a faithful explanation.

• Tokens are scratch space
• Voting reduces random errors
• RL trains long thinking
• Spend compute where it pays

Next time, we make model output dependable for software. We will cover structured output, forcing JSON that matches a schema, and then how to evaluate prompts systematically, so that claims like this prompt works better are backed by data rather than vibes.

• JSON and schemas
• Evaluating prompts systematically