Concept

A Large Language Model is a neural network trained to predict the next token given a sequence of preceding tokens. That is the entire job description. Every capability - summarization, coding, reasoning, translation - emerges from doing this one thing at massive scale on diverse data.

This framing matters for interviews. An LLM is not:

• A knowledge base (it encodes statistical patterns, not facts with citations)
• A reasoning engine (it simulates reasoning by predicting tokens that look like reasoning)
• A retrieval system (it cannot look things up unless given tools or RAG)

The correct mental model: an LLM is a compressed probabilistic model of human text. It has processed a significant fraction of human-written text and learned what tokens tend to follow other tokens across contexts. When you prompt it, you are sampling from that distribution conditioned on your input.

Why "large"? The term is relative. In 2018, BERT-large (340M parameters) was "large." By 2024, the baseline is 7B–70B+ parameters. What scale provides:

• More parameters → more capacity to memorize patterns and relationships
• More training tokens → better generalization and knowledge coverage
• The relationship is governed by scaling laws (see Training and Pretraining)

Autoregressive generation: At inference time, the model generates one token at a time, appending each new token to the context and re-running the forward pass. This is why generation is slow - you cannot parallelize sequential token production.

Input:  "The capital of France is"
Step 1: model predicts " Paris" (highest probability) → appended
Step 2: model predicts "." → appended
Step 3: model predicts end-of-sequence token → stop
Output: "The capital of France is Paris."

Concept

LLMs operate on tokens - subword units produced by a tokenizer - not characters or words. Understanding tokens is critical for cost estimation, context window management, and debugging unexpected model behavior.

How BPE tokenization works:

1. Start with all individual characters as the vocabulary
2. Repeatedly merge the most frequent adjacent pair into a single new token
3. Continue until vocabulary reaches the target size (typically 32K–200K tokens)
4. The result: common words become single tokens; rare words split into subword pieces

Practical rules of thumb:

• English text: ~1 token ≈ 4 characters ≈ 0.75 words
• Code: more tokens per line (brackets, indentation, symbols are expensive)
• Non-Latin scripts (Chinese, Arabic): often 1–3 characters per token (less efficient)
• Numbers: tokenizers split them inconsistently - OpenAI's tokenizers group digits in runs of up to three, some others (e.g. Llama 2) split every digit - a source of arithmetic failures

Why tokenization causes reasoning failures:

• "9.11 > 9.9": OpenAI's tokenizers produce ["9", ".", "11"] and ["9", ".", "9"] - the model compares token sequences, not numbers, and "11" looks bigger than "9"
• "50,000" vs "50000" may tokenize differently, causing inconsistency
• Words split across tokens can confuse rhyming and spelling tasks

Token boundary quiz (common interview trap):

• Q: How many tokens is "ChatGPT is great"?
  A: It depends on the tokenizer. GPT-4's cl100k_base gives 5 tokens (["Chat", "G", "PT", " is", " great"]) - "ChatGPT" was rare in its training data; the newer o200k_base gives 4 (["Chat", "GPT", " is", " great"]). Always count with the model's own tokenizer.

Code

import tiktoken
from transformers import AutoTokenizer

# OpenAI tokenizers: cl100k_base (GPT-4), o200k_base (GPT-4o and later)
enc = tiktoken.get_encoding("o200k_base")
text = "The tokenization of language models is surprisingly tricky."
tokens = enc.encode(text)
print(f"Token count: {len(tokens)}")
print(f"Tokens: {[enc.decode([t]) for t in tokens]}")

# Show the arithmetic problem
number_text = "Is 9.11 greater than 9.9?"
print(f"\n'{number_text}'")
print(f"Tokens: {[enc.decode([t]) for t in enc.encode(number_text)]}")
# Output: ['Is', ' ', '9', '.', '11', ' greater', ' than', ' ', '9', '.', '9', '?'] - digits grouped, not numbers

# Compare across model families
llama_tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
sample = "Tokenization affects everything downstream."
print(f"\nLLaMA-3 tokens: {llama_tok.tokenize(sample)}")
print(f"LLaMA-3 count: {len(llama_tok.encode(sample))}")

Concept

The context window is the maximum number of tokens an LLM can attend to in one request - input and output combined. It is set by how the model was trained and configured, and enforced by the serving stack.

Why context windows are bounded:

• Compute: self-attention scores every pair of tokens, so prefill compute grows as O(n²) - a 128K-token prompt means 128K × 128K ≈ 16.4 billion query-key scores per head per layer
• Memory: the KV cache grows linearly with context (O(n) per sequence) and dominates GPU memory at long lengths; FlashAttention removes the O(n²) attention memory but not the O(n²) compute
• Quality: models only generalise reliably to lengths they were trained on; extending context needs position-encoding changes (RoPE scaling) plus long-context training

How context windows grew (for current numbers see the Model Landscape):

Era	Representative models	Context window
2020	GPT-3	2K
2023	GPT-4 (8K/32K), Llama 2 (4K), Claude 2 (100K)	4K-100K
2024	Llama 3.1 (128K), GPT-4o (128K), Gemini 1.5 Pro (1M)	128K-1M
2025-2026	Frontier closed models; Llama 4 Scout (10M advertised)	~1M standard at the frontier

The dirty secret: A large context window ≠ uniform retrieval quality. The "lost in the middle" problem (see Failure Modes) means models attend poorly to tokens buried in the middle of very long contexts. Putting the most important information at the beginning or end of the context is a practical mitigation.

Context is shared between input and output. A 128K window with a 100K-token prompt leaves only 28K tokens for the response.

Context window scaling approaches - how modern models extend context:

Approach	Description	Used by
Fixed Context	Choose max tokens at model design time (e.g., 2K, 8K)	Early GPTs, BERT
Sliding Window / Chunking	Split input into overlapping windows (e.g., 512 tokens with 128 overlap)	RAG / long-doc QA
Adaptive Context	Dynamically select relevant chunks based on task	Attention routing systems
Sparse / Local Attention	Each token attends to a subset (local window + global tokens) - reduces O(n²) cost	Longformer, BigBird, sliding-window layers (Mistral 7B v0.1, Gemma 2/3)
IO-aware Exact Attention	Same exact attention, computed in tiles so the n×n matrix is never materialized - memory O(n), compute still O(n²)	FlashAttention 1/2/3 (used almost everywhere)
Memory-efficient Transformers	Linear attention or recurrent state instead of full attention	Performer, RWKV, Mamba
Segment Recurrence	Carry hidden states from previous segments forward	Transformer-XL, RWKV
Retriever-Augmented (RAG)	Retrieve external chunks to keep active context short and focused	All RAG systems
Compressed Context	Summarize long text, feed a compact version to the model	Hybrid long-doc systems
RoPE Scaling	Rescale rotary position frequencies (Position Interpolation, NTK/base-frequency scaling, YaRN) plus a short long-context training stage	Code Llama, Llama 3.1, Qwen2.5/Qwen3 (YaRN)

Context window sizes in current models (2025/2026):

Closed-model providers do not publish how they reach long context, so only the open-weight rows below have a documented mechanism. See the Model Landscape page for current context lengths.

Model	Context Length	How it's achieved
GPT-3.5	4K / 16K	Dense attention
GPT-4.1	1M	Undisclosed
Claude (Sonnet 4.x)	200K (1M beta)	Undisclosed
Gemini 2.5 Pro	1M	Undisclosed
Llama 3.1	128K	RoPE scaling + staged long-context pretraining (Meta, 2024)
Llama 4 Scout	10M	Interleaved RoPE / no-position-encoding layers (iRoPE) + long-context training (Meta, 2025)
Mistral 7B	8K (v0.1) / 32K (v0.2+)	Sliding-window attention (v0.1); larger RoPE base (v0.2)
Longformer	4K–16K	Sparse local + global attention
RWKV	Infinite (streaming)	RNN-like with token recurrence
Mamba	Sub-quadratic	Attention-free state-space model

Practical guidelines - choosing context window strategy:

Use Case	Recommended Strategy
Small chatbot (FAQ, support)	2K–4K context is sufficient
Long documents (legal, medical, policy)	8K–32K + retrieval or summarization
Summarization of books or legal docs	RAG + chunking + re-ranking
Code generation	8K–16K (code needs larger context for multi-file awareness)
Training from scratch	Trade off context size vs GPU memory (O(n²) attention cost)
Local inference on laptop	Prefer 2K–4K with quantization

Concept

When the model predicts the next token, it produces a probability distribution over its entire vocabulary (e.g., 32K–200K tokens). Sampling parameters control how you draw from that distribution.

Temperature scales the logits before softmax:

P(token_i) = softmax(logits / T)[i]

Temperature	Effect	Use case
0.0	Greedy: always pick argmax	Deterministic tasks, factual QA
0.1–0.5	Sharp, conservative	Code generation, structured output
0.7–1.0	Balanced creativity	General chat, creative writing
1.5–2.0	Very random, often incoherent	Brainstorming, diversity sampling

Tricky Q: Is temperature=0 identical to greedy decoding?
Mathematically yes - as T→0, softmax concentrates all mass on the argmax, and implementations special-case temperature=0 as argmax (literal softmax(logits/0) would divide by zero). But it is not guaranteed to be deterministic in practice: batched GPU inference changes floating-point reduction order with batch composition, so near-tied logits can flip between runs (see Prompt Fundamentals).

Top-p (Nucleus Sampling): Consider only the smallest set of tokens whose cumulative probability ≥ p, then renormalize and sample from that nucleus.

Sort tokens by P descending: [0.4, 0.2, 0.15, 0.1, 0.05, ...]
top_p=0.9: include tokens until cumulative sum ≥ 0.9 → [0.4, 0.2, 0.15, 0.1, 0.05] (sum=0.9)
Sample from these 5 tokens (renormalized)

Top-p adapts: when the model is confident (peaked distribution), the nucleus is tiny; when uncertain, it's larger.

Top-k: Restrict to the k highest-probability tokens before sampling. Less adaptive than top-p.

Repetition penalty: lowers the score of tokens that have already appeared (the CTRL formulation, as implemented in Hugging Face):

if token appeared before:
    logit[token] = logit[token] / penalty   if logit > 0
                   logit[token] * penalty   if logit < 0

Values > 1.0 penalize repetition. Without it (or sampling), greedy decoding often collapses into loops. API providers expose related frequency_penalty / presence_penalty parameters that subtract from logits instead.

Parameter	Effect	Typical Range
temperature	Distribution sharpness	0.0–2.0
top_p	Nucleus size	0.7–1.0
top_k	Hard token cutoff	10–100
repetition_penalty	Loop prevention	1.0–1.3
max_new_tokens	Output length cap	task-specific

Concept

LLMs are categorized by their training architecture and objective. Three core types exist; decoder-only has been the dominant design for generative models since GPT-3.

1. Decoder-Only (Causal LM)

• Architecture: transformer with causal (left-to-right) attention masking
• Training objective: predict next token (Causal Language Modeling, CLM)
• Examples: GPT family, LLaMA, Gemma, Mistral, Phi, Falcon
• Best for: text generation, chat, reasoning, code, general-purpose tasks
• Dominant in production - has displaced other architectures for most tasks

2. Encoder-Only (Masked LM)

• Architecture: transformer with bidirectional attention (sees full context)
• Training objective: predict masked tokens (Masked Language Modeling, MLM)
• Examples: BERT, RoBERTa, DeBERTa, DistilBERT
• Best for: text classification, NER, semantic similarity, embeddings
• Not generative - cannot produce free-form text, only classify or embed

3. Encoder-Decoder (Seq2Seq)

• Architecture: encoder produces context representations; decoder attends to encoder output via cross-attention
• Training objective: span corruption (T5), denoising (BART)
• Examples: T5, BART, mT5, FLAN-T5, mBART
• Best for: translation, summarization, structured prediction (document → output)
• Still used for seq2seq tasks but decoder-only models have largely caught up via prompting

Why decoder-only won:

1. Unified training objective (CLM) - no need for masked prediction or denoising design choices
2. Generation is natural - the architecture is designed for it
3. Instruction fine-tuning (SFT + RLHF) works exceptionally well on top of CLM pretraining
4. Emergent in-context learning scales with model size

Mixture of Experts (MoE) - a fourth category: Not a separate architecture but a modification: instead of one dense FFN per layer, use N "expert" FFNs and a router that selects 1–2 experts per token. Result: large total parameters but only a fraction active per token (Mixtral-8x7B: 47B total, ~13B active).

Full architecture deep dive: see Transformer Architecture and Model Architecture Types.