Reasoning Models

Background

Chain-of-thought prompting (Wei et al. 2022) showed that reasoning emerges from scale + the right prompting
OpenAI o1 (2024) extended this by training models to use internal chain-of-thought before answering
Reasoning models demonstrate emergent mathematical and logical capabilities above ~100B parameters

The Concept

Reasoning models are LLMs trained or prompted to produce intermediate reasoning steps (chain-of-thought) before giving a final answer. The model essentially "thinks out loud", which improves accuracy on complex problems by decomposing them into smaller steps.

How it Links

Emergent abilities from scaling laws explain why reasoning appears above a size threshold
Process reward models (PRMs) connect to RLHF - rewarding correct reasoning steps rather than just correct final answers
Tree-of-thought and graph-of-thought extend linear chains to branching search over reasoning paths

Concrete Example

Solving: "If Alice has 3 apples and gives half to Bob and a third to Carol, how many does she have?" A base LLM might guess. A reasoning model works: "Alice starts with 3. Half is 1.5 -> to Bob. A third is 1 -> to Carol. 3 - 1.5 - 1 = 0.5. Alice has 0.5 apples." The answer emerges from the steps.

Prerequisites

Transformers, Scaling Laws & Training, Fine-tuning & PEFT

Learning outcomes

Explain chain-of-thought, self-consistency, and least-to-most prompting and state the evidence for when each helps.
Distinguish process reward models (PRMs) from outcome reward models (ORMs) and articulate their trade-offs.
Describe the test-time compute scaling insight: why spending more inference FLOPs improves accuracy on multi-step problems.
Explain how tree-of-thought and MCTS generalize CoT to a search problem over reasoning steps.
Critically evaluate the open debate about whether LLMs "reason" or perform sophisticated pattern matching.

TL;DR

Chain-of-thought prompting [1] unlocks multi-step problem solving by having the model write intermediate steps before giving a final answer. Scaling inference compute further — via self-consistency sampling [2], tree-of-thought search [3], and reinforcement-learned scratchpads (OpenAI o1 [5], DeepSeek-R1 [4]) — turns a fixed model into a stronger problem-solver at the cost of more tokens, not more parameters.

Intuition

Traditional LLMs answer questions in one shot: prompt in, token stream out. This works well for tasks that fit in a single pattern-match but breaks down for problems that require several intermediate steps — multi-digit arithmetic, logical proofs, algorithm design. Humans naturally write scratch work; chain-of-thought simply asks the model to do the same.

The deeper insight from the 2022—2025 wave of reasoning research is that test-time compute is a new axis of scaling. Pre-2022, scaling meant training more parameters on more tokens. Post-2022, a model with fixed weights can solve harder problems simply by spending more inference tokens on planning, exploration, or verification. OpenAI o1 [5] and DeepSeek-R1 [4] are the clearest existence proofs.

Test-time compute is a new scaling axis

Training-time compute determines the model’s knowledge and base capabilities. Test-time compute determines how carefully that knowledge is applied to a specific problem. Reasoning models trade off latency for accuracy: asking for a 2000-token scratchpad before the final answer costs more inference tokens but substantially improves accuracy on math olympiad problems, code generation, and scientific reasoning.

Chain-of-thought prompting

Chain-of-thought (CoT) prompting [1] is the practice of including reasoning steps in the model’s output before the final answer. The simplest form is few-shot: include worked examples where the solution includes intermediate steps, and the model generalises to new problems with similar step-by-step outputs.

Wei et al. found that CoT only helps reliably in models above roughly 100 billion parameters. Below that threshold, intermediate steps often contain errors that compound. The paper also introduced the now-standard “Let’s think step by step” zero-shot CoT variant (Kojima et al. 2022, building on Wei et al.).

Why CoT helps on multi-step but not single-step tasks

For a single-step lookup (“What is the capital of France?”), writing intermediate steps adds noise without signal. The model already has the answer as a direct pattern. CoT helps when the solution requires serial computation: each step narrows the problem before the next step. For tasks where the answer is not a stored pattern but must be derived — arithmetic, symbolic reasoning, planning — the scratchpad lets the model represent intermediate state that does not fit in one forward pass.

Least-to-most prompting

Least-to-most prompting [7] addresses compositional generalisation: first ask the model to decompose the problem into easier subproblems, then solve each subproblem in sequence, using prior answers as context. This outperforms standard CoT on tasks requiring systematic decomposition, such as solving a harder version of a problem by building on the easier version.

Scratchpad and scratch tokens

“Scratchpad” (Nye et al. 2021) extends CoT to arbitrary intermediate computation: the model writes tokens that are not part of the final answer and are masked from the loss (or used as a separate supervision target). OpenAI o1 trains a model to use hidden reasoning traces via RL, with the chain never shown to users in its raw form. DeepSeek-R1 [4] demonstrates that RL with group relative policy optimisation (GRPO) over outcome reward suffices to produce long, coherent reasoning chains without supervised chain-of-thought data.

Self-consistency decoding

Self-consistency [2] observes that sampling $k$ independent chains of thought and taking a majority vote over final answers is strictly better than greedy decoding with a single chain. The intuition: different chains may make different errors, but correct answers cluster.

Self-consistency: the sampling-and-voting argument

Let $p_c$ be the probability that a single greedy chain gives the correct answer. Under the idealised assumption that $k$ chains are independent and each correct with probability $p_c$ , the majority-vote probability of correctness is

P(\text{majority correct}) = \sum_{j > k/2}^{k} \binom{k}{j} p_c^j (1 - p_c)^{k-j}.

For $p_c = 0.6$ and $k = 40$ , this is $P \approx 0.98$ , compared to $0.6$ for a single chain. In practice chains are not independent (they share the same model weights and prompt), but empirically marginal gains continue out to $k \approx 40$ before plateauing. Wang et al. 2022 report consistent gains of 10—20 percentage points on GSM8K, MATH, and StrategyQA with $k = 40$ versus greedy.

The cost is $k \times$ the inference budget. Self-consistency is therefore a trade-off: more inference compute for higher accuracy, with the majority-vote oracle setting the ceiling.

Process reward models vs outcome reward models

When training a model to reason via reinforcement learning, the reward signal can be applied at different granularities.

Outcome reward models (ORMs) give a single reward at the end of a generation: correct final answer = +1, wrong = 0 (or -1). ORMs are cheap to obtain (just check the final answer) but provide sparse credit assignment. A model might reach a correct answer via a flawed chain and still receive full reward.

Process reward models (PRMs) assign reward to each reasoning step individually, judging whether the step is correct and relevant to solving the problem. Lightman et al. [6] trained a PRM on Grade School Math by having human annotators label each step as correct, neutral, or incorrect. They found that PRM-guided beam search substantially outperformed ORM-guided search at the same number of samples.

Dimension	ORM	PRM
Reward granularity	Final answer only	Per reasoning step
Label cost	Low (auto-check final answer)	High (human or model labels per step)
Credit assignment	Sparse (entire chain gets same signal)	Dense (each step rewarded individually)
Search guidance	Can be used for reranking final answers	Can prune search tree at each step
Risk	Reward hacking via shortcut answers	Noisy step-level labels degrade training
Example uses	DeepSeek-R1 (GRPO on final correctness)	OpenAI process supervision (Lightman 2023)

Tree-of-thought and MCTS for reasoning

Tree-of-thought (ToT) [3] frames reasoning as a search problem over a tree of partial solutions. At each step, the model generates $k$ candidate next-thoughts, evaluates each (either via a separate value model or via self-evaluation prompting), and selects the most promising branches to expand. This generalises chain-of-thought from a linear sequence to a branching search.

Monte Carlo Tree Search applied to LLM reasoning

Monte Carlo Tree Search (MCTS) consists of four steps applied repeatedly:

Selection: traverse the current tree from the root, selecting child nodes via the UCB1 rule $\text{UCB1}(n) = \bar{v}_n + c \sqrt{\ln N_{\text{parent}} / N_n}$ until a leaf is reached.
Expansion: generate one or more child nodes by sampling next-thoughts from the LLM.
Simulation (rollout): from the new node, run a fast rollout to the end of the reasoning chain and evaluate the final answer.
Backpropagation: update the win-rate estimates $\bar{v}_n$ for all nodes on the path.

The LLM serves simultaneously as the policy (generating candidate next-thoughts) and the value function (scoring intermediate states). At modest tree sizes ( $k = 5$ , depth $= 4$ ) this multiplies inference cost by roughly $5^4 = 625\times$ over greedy decoding, but achieves accuracy approaching the oracle on 24-game and symbolic reasoning benchmarks where greedy CoT fails.

Method	Structure	Inference cost	Best for	Limitations
Greedy decoding	Single linear chain	1x	Simple factual queries	No error recovery
Chain-of-thought	Single linear chain with steps	2-5x tokens	Multi-step arithmetic, logic	Errors compound linearly
Self-consistency	k independent chains + vote	k x	Any CoT task with answer space	No structural search; cost grows linearly
Least-to-most	Sequential subproblem decomposition	2-3x	Compositional generalisation	Requires problem decomposability
Tree-of-thought	Branching tree with evaluation	Exponential in depth	Combinatorial planning, puzzles	Very expensive; value function quality critical
MCTS + LLM	Full search tree with UCB1	Very high (100-1000x)	Hardest math, formal proofs	Impractical for many production settings

The “reasoning” debate

Whether LLMs genuinely “reason” or perform sophisticated statistical pattern matching is one of the most contested questions in the field. Key positions:

Optimistic view: CoT, ToT, and RL-trained scratchpads enable systematic compositional generalisation that was absent in pure next-token prediction. DeepSeek-R1 [4] exhibits self-verification, backtracking, and error correction in its reasoning traces without explicit supervision for these behaviors. This looks like emergent reasoning.

Sceptical view: models trained on billions of math problems can approximate correct multi-step patterns without representing the underlying computational structure. Failure modes — wrong answers on trivially modified problems, inconsistent step generation across samples — suggest brittle retrieval rather than robust inference.

Working consensus: the debate is partly definitional. If “reasoning” means exhibiting consistent, generalisable, step-by-step derivation of novel conclusions, then current models do this imperfectly but measurably better than random. Formal verification of reasoning chains (against a proof checker) is the gold standard but currently limited to narrow domains.

CoT does not fix hallucination

Chain-of-thought reduces errors on problems where intermediate steps are verifiable (math, code). On factual claims, a confident-sounding reasoning chain can still hallucinate premises and reach plausible-looking but wrong conclusions. The intermediate steps are not independently verified during generation; they are just more tokens that the model treats as context for the next token.

Real-world applications

Math olympiad: OpenAI o1 scored in the 83rd percentile on the 2024 American Mathematics Competition AMC 12, and achieved gold-medal equivalent performance on IMO 2024 problems (informal evaluation). AlphaProof and AlphaGeometry 2 (Google DeepMind, 2024) solve formal IMO geometry proofs by combining neural guidance with a symbolic proof engine.

Code generation: AlphaCode 2 (DeepMind, 2024) reaches the 85th percentile of competitive programmers on Codeforces by generating and filtering large numbers of candidate solutions — a form of self-consistency at the program level.

Scientific discovery: reasoning models are being applied to hypothesis generation in drug discovery (protein-protein interaction prediction) and materials science, where the intermediate reasoning trace can be audited by domain experts.

Check yourself

Score: 0 / 5(0 answered)

1. What is the key difference between a process reward model and an outcome reward model?
2. Why does chain-of-thought prompting help on multi-step arithmetic but not single-step lookups?
3. Self-consistency decoding samples k independent reasoning chains and then:
4. The "test-time compute scaling" insight means that:
5. In Tree-of-Thought, what role does the LLM play beyond generating candidate next-thoughts?

What is the difference between a process reward model and an outcome reward model, and when would you prefer each?onsite

An outcome reward model (ORM) gives a binary or scalar reward based solely on whether the final answer is correct. It is cheap: for math problems, correctness can be verified programmatically. The downside is sparse credit assignment — a chain with ten steps all gets the same reward regardless of which steps were sound and which were lucky guesses.

A process reward model (PRM) evaluates each reasoning step independently: is this step logically valid and relevant? PRMs provide dense supervision, enabling finer-grained RL training and better search guidance (you can prune the tree at each step). The cost is high: step-level labels require human annotators or a separately trained step-verifier.

When to prefer ORM: tasks with automatically verifiable final answers (math, code execution), when label budget is limited, or when the model is already strong enough that sparse rewards suffice (as in DeepSeek-R1’s GRPO setup).

When to prefer PRM: tasks requiring search over long reasoning chains, when you want to identify exactly which step introduces an error, or when the training signal from final answers alone is too sparse (e.g., very long proofs with rare correct completions). Lightman et al. 2023 showed PRM-guided beam search substantially outperforms ORM reranking on GSM8K at the same sample budget.

Why does test-time compute help reasoning? Is there a theoretical justification?staff

The informal argument is that reasoning is serial computation, and serial computation cannot always be parallelised into a single forward pass. A Transformer with fixed depth $L$ can simulate $O(L)$ serial computation steps (one step per layer). If a problem requires $k \gg L$ serial steps, a single forward pass cannot express the solution. Generating $T$ tokens before the final answer effectively extends the serial depth to $L \cdot T$ (each token gets $L$ layers of computation, and later tokens can condition on earlier ones). This is the “scratchpad as extended computation” view from Nye et al. 2021 and formalised in circuit-complexity arguments by Merrill and Sabharwal (2023).

From an RL perspective, o1-style models learn a search policy: the model is trained to try approaches, detect errors, backtrack, and retry. This search is implemented entirely in the token stream, so more tokens mean more search budget. The PRM or ORM provides reward signal that drives the model toward search strategies that produce correct final answers.

The practical limit is that test-time compute scales logarithmically with accuracy at the frontier: going from 50% to 95% on MATH might require 40 samples; going from 95% to 99% might require 400. The marginal returns diminish because the remaining problems are structurally different from the ones in the training distribution.

References

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., and Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023 (arXiv preprint 2022). arXiv:2203.11171
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS 2023. arXiv:2305.10601
DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint. arXiv:2501.12948
OpenAI (2024). OpenAI o1 System Card. OpenAI Technical Report. https://openai.com/index/openai-o1-system-card/
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). Let's Verify Step by Step. ICLR 2024 (arXiv preprint 2023). arXiv:2305.20050
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. (2022). Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. ICLR 2023 (arXiv preprint 2022). arXiv:2205.10625
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762

Modern AI: Transformer Variants & Training Paradigms World Models