Learn GenAI
Github
Contents
Map

Quiz · 08 · Training LLMs from Scratch

11 questions from 2 pages

These are the Check Yourself questions from each page of the module, collected in course order. Each heading links back to the page the questions test. All module quizzes →

Let's Build GPT with Andrej Karpathy, Visualized

Check yourself
0 / 6 answered
  1. xb has shape (4, 8). How many next-token training examples does the (xb, yb) pair hold?
  2. Before training, a 65-character model reports a loss of 4.17. What does that say about its predictions?
  3. What shape does self.token_embedding_table(idx) return when idx is (4, 8)?
  4. After 10,000 steps the bigram's loss settles near 2.5 and more training does not help. Why?
  5. In one head with head_size = 16, q and k are (4, 8, 16). What is the shape of wei = q @ k.transpose(-2, -1), and what does wei[b, i, j] mean?
  6. Why does every Transformer block write x = x + self.sa(self.ln1(x)) instead of x = self.sa(x)?

GPT From Scratch: Step by Step

Check yourself
0 / 5 answered
  1. You have a corpus of 2 million web documents. Where should the train/validation split happen?
  2. With block_size = 8, how many next-token predictions does one (x, y) window train on?
  3. Your first logged loss on a GPT-2-tokenized dataset is 10.9. What does that tell you?
  4. You raise grad_accum from 1 to 8 but forget to divide the loss by 8. What changes?
  5. Why does generate() crop the context to the last block_size tokens?