Prompt Fundamentals

A prompt is the complete input a language model receives for one call - instructions, examples, data, conversation history and tool results - and it is the only thing the model knows about your task.

A Prompt Is a Program

Each API call is stateless: the model sees only the tokens in this request. Chat "memory" is your application re-sending earlier turns. So the prompt carries everything - and the model's job is to produce the most likely continuation of it. A good prompt makes the output you want the natural continuation.

A useful mental model is to treat the prompt as a function:

The last two rows matter most. A prompt without an eval is untested code - you cannot tell whether an edit helped. The code lab makes this concrete: two prompts that "look" different in quality turn out to be statistically indistinguishable on 52 examples.

Anatomy of a Prompt

flowchart TD
    SYS["⚙️ System / developer message<br/>role, rules, output contract<br/>(stable across calls)"] --> CTX
    CTX["📚 Context<br/>background, retrieved documents,<br/>tool definitions, examples"] --> HIST
    HIST["💬 Conversation history<br/>earlier user / assistant / tool turns"] --> USR
    USR["❓ Current user turn<br/>the task and the input data,<br/>clearly delimited"] --> GEN
    GEN(["🤖 Model generates the assistant turn"])

    style SYS fill:#d8dfe8,stroke:#b0bac8
    style CTX fill:#e8e0d4,stroke:#c8b89a
    style HIST fill:#e8e2d9,stroke:#ccc4b8
    style USR fill:#dde4dc,stroke:#b0c4b0

Delimit input data so the model can tell it apart from your instructions - XML-style tags work well across providers:

prompt = """Summarise the report below in two sentences for an executive audience.

<report>
{report_text}
</report>"""

Build up, don't pile on. Start with the instruction alone, run it on a handful of real inputs, and add context, format rules or examples only to fix failures you actually observed. Long lists of speculative rules make prompts brittle: more rules to conflict, more rules to violate.

How the Model Sees Your Messages: Chat Templates

APIs take a list of messages, but the model consumes one token sequence. A chat template converts the messages into that sequence using special tokens the model was fine-tuned on. Each model family has its own:

Llama 3:  <|begin_of_text|><|start_header_id|>system<|end_header_id|>

          You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

          What is attention?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

ChatML (Qwen and many open fine-tunes):
          <|im_start|>system
          You are a helpful assistant.<|im_end|>
          <|im_start|>user
          What is attention?<|im_end|>
          <|im_start|>assistant

Others include Gemma's <start_of_turn> format, Mistral's [INST] format and OpenAI's "harmony" format for its open-weight gpt-oss models, which adds channels that separate reasoning from the final answer. With hosted APIs the provider applies the template for you. With open models you must - and getting it wrong is a silent failure: the model sees text that looks nothing like its training data and quality drops without any error.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is attention?"},
]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,  # append the assistant header so the model answers
    enable_thinking=False,       # Qwen3-specific switch; other templates ignore unknown kwargs
)

Always take the template from tokenizer.chat_template (or the serving engine's chat endpoint) rather than hand-writing it. The Fine-Tuning Lab shows the other side: training data must be rendered with the same template the model will be served with.

Message Roles and the Instruction Hierarchy

Models are trained to give higher-priority roles more authority - OpenAI describes this as an instruction hierarchy (system > developer > user > tool output). This is learned behaviour, not an architectural boundary: every role becomes tokens in the same sequence and the same attention layers read all of them. A crafted user message, or an instruction hidden in a retrieved web page, can still override your system prompt. That is the root of prompt injection, covered in Prompts in Production. Never put secrets in a system prompt, and never rely on it as a security control.

Assistant prefill - and why it is disappearing

Prefill means ending the message list with a partial assistant message (for example {) so the model continues from it. It was a common trick for forcing JSON or skipping preambles. On current hosted reasoning models it is being removed: Anthropic's models return an error for a prefilled final assistant turn from Opus 4.6 / Sonnet 4.6 onwards, and the replacement is native structured output (see Structured Outputs). With open models you still control the raw template, so prefill still works there.

Tokens: What the Model Actually Reads

The model reads subword tokens (typically byte-level BPE), not characters or words. Tokenization explains a family of surprising failures:

• Character-level tasks - counting letters, reversing strings, spelling - are hard because a word may be one token. Ask the model to spell the word out first, or do it in code.
• Numbers split into irregular chunks, so arithmetic without step-by-step working (or a code tool) is error-prone.
• Formatting sensitivity - whitespace, separators and casing change the token sequence. Sclar et al. (2024) measured accuracy swings of up to 76 points on one model from formatting changes alone - a strong argument for evaluating prompts on data rather than eyeballing them.
• Budgets are in tokens - context limits, prices and latency all count tokens, and different tokenizers count the same text differently. Measure with the provider's token-counting endpoint or the model's tokenizer; don't estimate from words.

import tiktoken                        # OpenAI tokenizers; use the model's own tokenizer for others
enc = tiktoken.get_encoding("o200k_base")
print([enc.decode([t]) for t in enc.encode("strawberry 1234567")])

Context windows of 200K to over 1M tokens are now common (see Model Landscape), but a long context is not a well-used context - Context Engineering covers why quality degrades as input grows.

Sampling Settings

The prompt decides what distribution the model predicts; sampling decides how a token is drawn from it.

Two things have changed with reasoning models:

1. Sampling controls are going away. Many reasoning models fix their sampling internally and reject or ignore temperature/top_p - Anthropic's newest models return an error for non-default sampling parameters, and OpenAI's reasoning models expose a reasoning-effort setting instead. The control you tune is now how much the model thinks (see Prompting Reasoning Models).
2. Temperature 0 was never fully deterministic. Floating-point operations are not associative, and a server batches your request with other users' requests, so kernels can reduce in a different order from one call to the next and argmax ties can flip. He et al. (2025) traced most inference non-determinism to this lack of batch invariance and showed that batch-invariant kernels make results reproducible, at some speed cost. For reproducibility, pin the model version, log the full request, and evaluate on sets rather than single outputs.

Check Yourself

Exercises

Study Notes

Must-know:

• A prompt is the model's entire view of the task; calls are stateless
• Four parts: instruction, context, delimited input, output contract - add parts to fix observed failures
• Chat templates turn messages into tokens; the wrong template fails silently
• Role priority (system > developer > user > tool) is trained, not enforced - the basis of prompt injection
• Prefill is being replaced by native structured outputs on hosted reasoning models
• Tokenization explains letter-counting, arithmetic and formatting-sensitivity failures
• Reasoning models replace temperature with reasoning-effort controls; temperature 0 is not deterministic on shared servers

References

• Brown et al., Language Models are Few-Shot Learners (2020) - prompting as conditioning a pretrained model
• Ouyang et al., Training language models to follow instructions with human feedback (2022) - why instruct models follow instructions
• Wallace et al., The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024)
• Sclar et al., Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (ICLR 2024)
• He et al., Defeating Nondeterminism in LLM Inference (Thinking Machines, 2025)
• Hugging Face, Chat templates
• OpenAI, harmony response format (2025)

Last reviewed: 2026-09