Prompting Reasoning Models

Reasoning models generate a hidden or summarised chain of reasoning before their answer, so prompting them shifts from guiding the reasoning to specifying the goal, the constraints and the output - and to choosing how much reasoning to pay for.

What Changed

Reasoning models are trained with reinforcement learning to think at length before answering, and their accuracy on hard problems rises with the amount of reasoning they do (test-time compute). By 2026 most frontier models reason by default, and the old distinction between "chat models" and "reasoning models" has become a setting: how much to reason.

What did not change: clear task definitions, the right context, explicit output contracts and evaluation matter as much as ever.

The Effort Dial

Each provider exposes a control for how much the model reasons. Higher settings improve accuracy on hard tasks and cost more output tokens and latency.

Older interfaces used a token budget instead of a level (Anthropic's budget_tokens, Gemini 2.5's thinking_budget); current models favour levels, because the model is better than a fixed number at judging how hard a given request is.

# OpenAI Responses API
resp = client.responses.create(model=MODEL, reasoning={"effort": "low"}, input=prompt)

# Anthropic Messages API (adaptive thinking is the default on current models)
resp = client.messages.create(model=MODEL, max_tokens=16000,
                              output_config={"effort": "low"},
                              messages=[{"role": "user", "content": prompt}])

Treat effort as a hyperparameter to evaluate, per task: run your eval set at two or three levels and plot accuracy against cost per item. Classification and extraction are often as accurate at the lowest level; planning, math, debugging and long agentic tasks benefit from more.

Practical notes:

• Reasoning tokens are billed as output tokens and count against the output limit. Set max_tokens high enough for thinking plus the answer, or you will get truncated outputs.
• Latency: time-to-first-visible-token includes the thinking. Stream, and show progress (reasoning summaries) in interactive products.
• Model-specific rules: some models reject certain combinations (for example, disabling thinking at the highest effort levels). Check the provider's model page.

Writing the Prompt

Before (written for a non-reasoning model)
  You are an expert analyst. Think step by step.
  Step 1: List every metric in the report.
  Step 2: For each metric compare to last quarter.
  Step 3: Decide which changes matter.
  Step 4: Write the summary.

After (for a reasoning model)
  Goal: a 5-bullet summary of what changed this quarter, for the CFO.
  What matters: changes over 5% in revenue, gross margin, churn or cash
  burn, and anything the report itself flags as a risk.
  Constraints: use only figures in the report; say "not reported" rather
  than estimating.
  Output: JSON {"bullets": [...], "risks": [...]}.
  <report>...</report>

Guidelines:

1. State the goal and why it matters - the model resolves ambiguity better when it knows the purpose.
2. Give success criteria and constraints, not a procedure.
3. Provide all the context the task needs (documents, definitions, prior decisions). Reasoning cannot recover missing facts - it can only hallucinate them more elaborately.
4. Specify the output contract and enforce it with structured outputs.
5. Keep examples for format and style, not for reasoning.
6. For agentic tasks, say when to stop and what "done" means, and whether to ask or proceed when information is missing.

Reasoning State in Multi-Turn and Tool Loops

When a reasoning model calls a tool, its reasoning so far is part of the state it needs to continue. Each API has a mechanism for carrying it across turns, and dropping it degrades quality or raises errors:

Most APIs return a summary of the reasoning or an encrypted form, not the raw tokens. Don't build features that depend on reading raw reasoning.

More Thinking Is Not Always Better

• Inverse scaling. Gema et al. (2025) built tasks on which accuracy falls as reasoning length grows - the model gets distracted by irrelevant details, overfits to familiar framings or drifts. Measure accuracy at several effort levels rather than assuming the highest is best.
• Overthinking simple tasks wastes tokens and latency; use low effort or a non-reasoning setting for routing, classification and extraction.
• Visible reasoning is not a faithful explanation. Chen et al. (2025) found reasoning models often fail to mention hints that demonstrably changed their answers. Use reasoning summaries for debugging and user feedback, not as an audit trail of why the model decided something - log inputs, tool calls and outputs instead.

Check Yourself

Exercises

Study Notes

Must-know:

• Reasoning models think before answering; prompts specify goal, context, constraints and output rather than steps
• Effort controls: OpenAI reasoning.effort, Anthropic adaptive thinking + effort, Gemini thinking_level; token budgets are the older style
• Reasoning tokens are billed as output and count toward max_tokens
• Carry reasoning state (reasoning items, thinking blocks, thought signatures) through tool loops
• More reasoning can hurt (inverse scaling); tune effort per task on an eval set
• Reasoning summaries are not faithful explanations

References

• OpenAI, Reasoning models guide
• Anthropic, Extended and adaptive thinking
• Google, Gemini thinking
• Gema et al., Inverse Scaling in Test-Time Compute (2025)
• Chen et al., Reasoning Models Don't Always Say What They Think (2025)
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)

Last reviewed: 2026-09