Post-Training & Fine-Tuning - Q&A Review Bank

48 Q&A pairs. Q1-Q21 cover the methods, Q22-Q48 the hands-on fine-tuning workflow. Tags (methods questions): [Easy] = conceptual recall, [Medium] = design decisions and trade-offs, [Hard] = derivations, quantitative reasoning or system design.

---

Pipeline and SFT

Q1 [Easy] What are the typical stages of post-training for a chat or reasoning model?

Supervised fine-tuning on demonstrations, then preference tuning (DPO-family on chosen/rejected pairs) and/or online RL (PPO or GRPO with reward models or verifiers). Reasoning models add long-chain-of-thought RL with verifiable rewards, often wrapped in rejection-sampling SFT rounds.

---

Q2 [Medium] Why is the loss masked to the response tokens during SFT?

The model should learn to produce responses, not to predict the user's prompt or the system prompt. Masking prompt tokens (label -100) focuses the gradient on the assistant turns and avoids teaching the model to imitate user text.

---

Preference Optimization

Q3 [Medium] Write the Bradley-Terry reward-model loss and explain it.

L = -log σ(r(x, y_w) - r(x, y_l)). It maximizes the modelled probability that the chosen response beats the rejected one, which depends only on the difference of their scalar rewards.

---

Q4 [Hard] Sketch how DPO removes the reward model.

The KL-regularized RLHF objective has the closed-form optimum π*(y|x) ∝ π_ref(y|x)·exp(r(x,y)/β). Rearranging gives r(x,y) = β·log(π(y|x)/π_ref(y|x)) + const. Substituting that implicit reward into the Bradley-Terry loss cancels the constant and yields a loss purely in terms of the policy and reference log-probabilities: -log σ(β[log-ratio(chosen) - log-ratio(rejected)]).

---

Q5 [Medium] What does β control in DPO?

The strength of the implicit KL constraint: larger β keeps the policy closer to the reference model; smaller β allows larger moves (more gain, more risk of degradation or reward over-optimization).

---

Q6 [Medium] When would you choose KTO, SimPO or ORPO over DPO?

KTO when you only have unpaired good/bad labels (e.g. thumbs up/down). SimPO when you want to drop the reference model (memory) and counter length exploitation with length-normalized rewards and a margin. ORPO when you want to fold SFT and preference tuning into one reference-free stage.

---

Q7 [Medium] What are DPO's main limitations?

It is offline, so it cannot discover behaviour beyond its fixed pairs; it degrades when pairs come from a distribution far from the policy's; it can reduce the likelihood of both chosen and rejected responses; and it tends to lengthen outputs. Iterative/online DPO (regenerating pairs from the current policy) mitigates the first two.

---

Reinforcement Learning

Q8 [Medium] Which models does PPO-based RLHF keep in memory, and why?

The policy (trained), a frozen reference (KL anchor), a reward model (scores responses) and a value model (per-token value estimates for advantages). That is roughly four model copies - the main reason GRPO's removal of the value model matters.

---

Q9 [Medium] How does GRPO compute advantages?

Sample G responses per prompt, score them, and set each response's advantage to (reward - group mean) / group std, applied to all its tokens. The group acts as the baseline, so no value model is needed.

---

Q10 [Hard] A GRPO group of 4 gets rewards [1, 1, 0, 0]. What are the advantages, and what if all four were 1?

Mean 0.5, std 0.5 → +1 for the correct responses, -1 for the incorrect. If all are 1, the std is zero and every advantage is zero (implementations add ε), so the prompt contributes no gradient - which is why DAPO's dynamic sampling filters such prompts out.

---

Q11 [Medium] What is RLVR, and why did it matter?

RL with verifiable rewards: rewards come from programs - exact-match answers, unit tests, format checks - rather than learned reward models. It is hard to exploit and cheap, which made long reasoning-focused RL runs practical (the Tülu 3 paper named it; DeepSeek-R1 used it at scale).

---

Q12 [Hard] Name three changes DAPO made to GRPO and the problem each solves.

Clip-higher (a larger upper clip bound) prevents entropy collapse by letting low-probability good tokens grow; dynamic sampling drops prompts whose samples all share one reward, since they carry zero advantage; token-level loss averaging stops long responses being under-weighted; plus overlong reward shaping and removing the KL term.

---

Q13 [Medium] What is reward hacking? Give two examples and two defences.

The policy maximizes the reward as written rather than as intended - e.g. padding answers because the reward model likes length, or special-casing unit tests instead of fixing code. Defences: prefer robust verifiers and adversarial tests, and keep an independent held-out evaluation; also KL regularization, step caps and regular sample inspection.

---

Q14 [Medium] Outcome reward model vs process reward model?

An ORM scores the complete response; a PRM scores each reasoning step, giving denser credit assignment and catching right-answer-wrong-reasoning. PRMs need expensive step labels (or noisy automated ones) and can also be hacked; DeepSeek-R1 reported they did not pay off relative to simple outcome rewards in their setting.

---

Q15 [Hard] Why is RL training dominated by generation, and how do systems cope?

Each step needs many long sampled responses, and autoregressive decoding is slow relative to a training step. Frameworks embed a fast inference engine (vLLM/SGLang) in the loop, sync weights to it every step, run generation and training on separate or time-shared GPU pools, use asynchronous slightly off-policy updates, and cap or drop long-tail rollouts.

---

Reasoning Models

Q16 [Medium] Describe the DeepSeek-R1 training pipeline.

Cold-start SFT on thousands of curated long-CoT examples; reasoning-focused GRPO with accuracy, format and language-consistency rewards; rejection sampling to build ~600K reasoning + ~200K general SFT samples and a second SFT; then RL across all scenarios combining verifiable rewards and reward models for helpfulness and harmlessness.

---

Q17 [Medium] What happened in DeepSeek-R1-Zero, and why wasn't it shipped as the main model?

Pure GRPO from the base model with only accuracy and format rewards produced growing response lengths and emergent self-verification and backtracking, with strong reasoning scores - but outputs had poor readability and mixed languages, which the cold-start SFT stage in R1 fixed.

---

Q18 [Medium] Why distil reasoning into small models instead of running RL on them?

DeepSeek found SFT on a strong reasoning model's traces gave small models better reasoning than large-scale RL applied to the small models directly - RL is expensive and small models explore poorly, while distillation transfers behaviour the large model already discovered.

---

Q19 [Medium] List four ways to spend compute at test time.

Longer thinking (more reasoning tokens / higher effort), self-consistency (majority vote over samples), best-of-N with a verifier or reward model, and sequential self-revision; plus search over reasoning steps with a step-level scorer.

---

Q20 [Medium] What are the main failure modes of reasoning models in production?

Overthinking easy inputs (cost and latency), chains of thought that don't faithfully reflect the real reason for an answer, reward-hacking behaviours learned during RL, and smaller gains on open-ended tasks without verifiable rewards. Effort controls and per-route evaluation are the practical levers.

---

Preference Tuning in Practice

Q21 [Medium] Why did DPO become a common default over PPO-based RLHF, and where does online RL still win?

DPO reformulates preference optimization as a supervised loss directly on (chosen, rejected) pairs - no separate reward model training, no PPO. RLHF/PPO challenges: reward model is a proxy (can be gamed), PPO is unstable (KL constraint must be tuned carefully), requires two separate training stages. DPO: one training stage, same stability as SFT, competitive results on chat-preference benchmarks. Llama 3's post-training, for example, used SFT + rejection sampling + DPO rather than PPO. The limit: DPO is offline (it learns from a fixed set of pairs), so it cannot discover behaviours beyond that data. Reasoning models brought online RL back - GRPO-style RL with verifiable rewards (DeepSeek-R1, 2025) trains on the model's own fresh samples scored by a checker, which is how long chain-of-thought reasoning is elicited.

---

HuggingFace Ecosystem

Q22 Why does loading a mismatched tokenizer and model checkpoint not raise an error?

Tokenization always succeeds mechanically - it maps text to some valid vocabulary IDs regardless of which model those IDs were originally trained against. The model then computes on token IDs it was never trained to expect, producing degraded or incoherent output silently, with no exception raised anywhere in the pipeline.

---

Q23 What problem does apply_chat_template solve, and why does it matter for fine-tuning specifically?

It formats a list of role-tagged messages into the exact string format (special tokens, turn markers) a given instruction-tuned model expects, since this format varies by model family. For fine-tuning, if your training data isn't formatted with the same chat template the model will see at inference time, the model learns a distribution that doesn't match how it's actually prompted in production - a common and hard-to-diagnose cause of a fine-tune that "ignores instructions."

---

Q24 Why do many base (non-instruction-tuned) models ship without a pad_token, and what's the standard fix?

Base models are typically trained purely on next-token prediction over continuous text, with no need for a dedicated padding token during pretraining. The standard fix is tokenizer.pad_token = tokenizer.eos_token - safe because padded positions are excluded from the loss via the attention mask, so the model never trains on the pad token's presence.

---

Q25 You're about to fine-tune an open model you've never used before. What are the first three things you check on its model card before writing any code?

License (can you fine-tune and redistribute/commercially use the result), whether it's gated (requires accepting terms before from_pretrained will succeed), and its chat template/tokenizer compatibility with the training pipeline you're about to build. Skipping these is a common source of wasted setup time - discovering a license restriction or a gated-access block after writing a training script is avoidable.

---

Q26 Why does AutoModelForCausalLM.from_pretrained("some-model") work identically across totally different model architectures?

The Auto* classes read config.json in the model's repo to resolve the correct concrete architecture class (e.g. Qwen2ForCausalLM, LlamaForCausalLM) automatically. This is what lets a fine-tuning script stay architecture-agnostic - swapping the base model is usually a one-line change as long as the code only calls the Auto* interface.

---

LoRA & QLoRA Hands-On

Q27 What's the practical effect of increasing LoRA rank r from 8 to 64?

Trainable parameter count and adapter capacity increase roughly linearly with r, which can improve quality on more complex fine-tuning tasks - but it also increases VRAM/compute cost and raises overfitting risk on small datasets. Most tasks (style, format, narrow-domain behavior) are well served by r=8-16; broader behavior changes may justify r=32-64.

---

Q28 What is prepare_model_for_kbit_training() for, and what breaks if you skip it in a QLoRA setup?

It casts normalization layers to FP32 for numerical stability, enables input gradients on the embedding layer (required for gradient checkpointing to work through frozen quantized layers), and disables use_cache during training. Skipping it is a common cause of a QLoRA run that trains without errors but fails to actually converge - gradients don't flow correctly through the quantized base model.

---

Q29 Why can't you cleanly merge a QLoRA adapter directly back into its 4-bit quantized base model?

The base weights are stored in a lossy quantized format (NF4), so merging directly would compound quantization error into the merged result. The standard approach is to reload the base model in full precision (BF16/FP16), merge the adapter there, and optionally re-quantize the merged model afterward for serving.

---

Q30 When would you choose to keep a LoRA adapter unmerged rather than merging it into the base model?

When serving multiple fine-tuned variants from the same base model - multi-tenant deployments, A/B testing different fine-tunes, or per-customer customization. Keeping adapters separate (a few MB each) avoids duplicating the much larger base model per variant, at the cost of a small inference-time overhead from the extra low-rank matmul.

---

Q31 A teammate proposes target_modules=["q_proj"] only, to save training time. What's the tradeoff?

Fewer target modules means fewer trainable parameters and faster/cheaper training, but also less capacity to adapt behavior - the original LoRA paper adapted only q_proj and v_proj, but the QLoRA paper found that adapting all linear layers (attention and MLP projections) was needed to match full fine-tuning quality - which is why most current recipes use target_modules="all-linear". Adapting a single projection is a cheap first experiment, not a default.

---

Q32 Why does lora_alpha matter even though it doesn't change the number of trainable parameters?

lora_alpha scales the adapter's contribution to the output (output = base + (alpha/r) * B*A), effectively controlling how strongly the LoRA update influences generation relative to the frozen base weights - it's closer to a learning-rate-like knob on the adapter's influence than a capacity knob.

---

Q33 Explain the LoRA math. How many parameters does LoRA add for a linear layer of size 4096×4096 at rank r=16?

LoRA approximates the weight update as ΔW ≈ BA where B ∈ ℝ^{4096×16} and A ∈ ℝ^{16×4096}. Parameters: 16×4096 + 4096×16 = 2 × 4096 × 16 = 131,072. Full weight matrix: 4096 × 4096 = 16,777,216. Reduction: 128×. Initialization: B = zeros, A = random Gaussian → ΔW = BA = 0 at start, so training begins from exactly the base model's behavior.

---

Q34 What is QLoRA and what enables it to fine-tune 7B models on a single 24GB GPU?

QLoRA combines two innovations: (1) NF4 quantization of the base model weights (4-bit NormalFloat - bins at equal normal distribution probability intervals) → 4 GB for 7B weights instead of 14 GB. (2) BF16 LoRA adapters on top of the frozen NF4 base → ~0.3 GB adapters. Total VRAM: ~5–6 GB for weights + activations, fitting on an RTX 4090 (24 GB). Double quantization further reduces the quantization constant storage. The base model is frozen (no gradient through NF4 weights); only the BF16 adapters are updated.

---

Instruction Data & Training Runs

Q35 What does setting a label token to -100 actually do during training, mechanically?

-100 is PyTorch's CrossEntropyLoss default ignore_index. Any position in labels set to -100 is skipped entirely during loss computation - it contributes exactly zero to both the loss value and the resulting gradient, which is the concrete mechanism behind masking prompt tokens out of SFT loss.

---

Q36 Training loss decreases smoothly and ends near zero - is that always a good sign?

No. Loss dropping to near-zero very quickly is a common symptom of the learning rate being too high or the dataset being too small/repetitive, causing the model to memorize training examples rather than learn a generalizable pattern. Train loss alone can't distinguish this from healthy convergence - a held-out validation loss is needed to catch it.

---

Q37 What's the difference between how a hand-written PEFT training loop and a full fine-tuning loop are constructed?

Structurally they're the same five-step loop (forward → loss → zero_grad() → backward() → step()). The difference is what the optimizer tracks: in PEFT, the optimizer is constructed only over parameters with requires_grad=True (the small LoRA adapter), while the base model's parameters stay frozen and receive no gradient updates at all.

---

Q38 You see training loss oscillating without a clear downward trend - what are the two most likely fixes to try first?

Lower the learning rate, and/or increase the effective batch size via gradient_accumulation_steps. Both stabilize the gradient signal per optimizer step, which is the usual root cause of loss that bounces around instead of trending down.

---

Q39 Why is overfitting a bigger risk in instruction fine-tuning than in typical large-scale ML training?

Instruction fine-tuning datasets are often small - hundreds to low tens-of-thousands of examples - compared to pretraining-scale corpora. A model can begin memorizing a small dataset within a single epoch, so overfitting signatures (train loss down, val loss up) can appear much earlier than practitioners used to larger-scale training might expect.

---

Q40 Why is loss masking (labels = -100 on prompt tokens) necessary for instruction fine-tuning specifically, more so than for other supervised tasks?

Instructions vary widely in phrasing across examples with no single "correct" pattern to learn - training on them would waste gradient signal pushing the model toward memorizing instruction phrasing rather than optimizing output quality. Masking ensures every gradient update is driven purely by how well the model completes the task, not by how well it predicts the next word of an arbitrary instruction.

---

Q41 How do you handle task interference in multi-head fine-tuning?

Three main approaches: (1) LoRA per task: freeze the shared backbone, train separate LoRA adapters per task - the tasks share no trainable parameters, so they cannot interfere with each other's weights (each adapter can still degrade general behaviour when it is active). (2) Gradient surgery (PCGrad): project each task's gradient onto the perpendicular of conflicting gradients from other tasks before summing - reduces destructive interference. (3) Task-specific learning rates: use a small LR for the shared encoder (slow drift) and larger LR for task heads (fast adaptation). Also: balanced sampling (prevent one task dominating the batch) and auxiliary losses.

---

Benchmarking Base vs Tuned

Q42 Why must the eval set be split off before training starts, and never touched during training?

If the model sees eval examples during training (directly, or indirectly through hyperparameter tuning decisions), the eval score no longer measures generalization - it measures memorization of data the model has already seen, giving a falsely optimistic quality signal that won't hold up on real production traffic.

---

Q43 Why report both a task-specific metric (like ROUGE) and an LLM-as-judge score instead of just one?

Task metrics are objective and cheap but brittle - they penalize correct answers phrased differently from the reference. LLM-as-judge scores capture semantic/holistic quality (helpfulness, tone, correctness) more like a human reviewer would, but introduce their own model bias and variance. Reporting both triangulates a more trustworthy quality signal than either alone.

---

Q44 Why exclude the first model.generate() call from a latency benchmark?

The first generation call on a freshly loaded model pays a one-time cost for CUDA kernel compilation and memory allocator warm-up that subsequent calls don't repeat. Including it in the average understates the model's true steady-state throughput, sometimes significantly.

---

Q45 Give one concrete signal from a benchmark that should make you choose RAG over fine-tuning for a given problem.

If the fine-tuned model's errors are factual - wrong or outdated information - rather than stylistic or format errors, that's a sign the actual gap is missing knowledge, not missing behavior/style. Fine-tuning reliably changes behavior and output format but does not reliably or durably inject new factual knowledge the way retrieval-augmented generation does.

---

Q46 A fine-tune shows a large quality improvement on its own training-adjacent eval set, but a smaller improvement on a broader, independently sourced set of test prompts. Which number do you trust for a shipping decision?

The broader, independently sourced set - an eval set too similar to the training distribution will systematically overstate the model's real-world quality gain. This is exactly why the held-out eval set needs to represent production traffic diversity, not just be technically "unseen during training."

---

Q47 Given a benchmark showing the fine-tuned model matches base-model latency and VRAM but with a modest quality bump, and low request volume in production - what's the actual deployment recommendation, and why?

At low volume, the operational overhead of maintaining a fine-tuned model (re-training on data drift, versioning, redeployment) often outweighs a modest quality gain that a well-crafted prompt might capture just as well. The recommendation depends on whether few-shot prompting alone can close most of that quality gap - if so, skip the fine-tune; if the delta over prompting is large and consistent, the fine-tune is justified even at lower volume.

---

Q48 If a fine-tuned support-bot model gives more confident-sounding but occasionally factually wrong answers about current company policy, what does the benchmark methodology need to catch this, and what's the fix?

The LLM-as-judge and task-metric scores alone won't reliably catch factual drift unless the eval set specifically includes policy-sensitive questions with verifiable reference answers. The underlying fix isn't more fine-tuning data - it's pairing the model with RAG so current policy documents are retrieved and grounded at inference time, since fine-tuning does not durably encode facts that change over time.