07 - PyTorch Fundamentals

What You Will Learn

• Tensor basics, requires_grad, and how the autograd computation graph is built and traversed
• Why gradients accumulate by default and when to call optimizer.zero_grad()
• Writing a custom Dataset (__len__ / __getitem__) and configuring a DataLoader for batching, shuffling, and parallel loading
• The canonical hand-written training loop: forward pass, loss, backward(), optimizer.step(), and the train/eval mode switch
• Checkpointing model and optimizer state, resuming training, and using mixed precision (autocast + GradScaler) to save VRAM and speed up training
• Moving tensors between CPU/GPU, reading CUDA OOM errors, and debugging shape mismatches
• Interview-ready answers on all PyTorch fundamentals topics with a dedicated Q&A bank

Chapter Map

#	File	Topic	Difficulty
1	Tensors & Autograd	Tensor basics, requires_grad, computation graph, .backward(), gradient accumulation	Beginner
2	Dataset & DataLoader	Custom Dataset, DataLoader batching/shuffling/num_workers, collation	Beginner
3	Training Loop From Scratch	Hand-written epoch/batch loop, model.train()/model.eval(), validation loop	Intermediate
4	Checkpointing & Mixed Precision	state_dict() save/load, resuming training, autocast, GradScaler	Intermediate
5	Debugging & GPU Memory	.to(device), CUDA OOM errors, shape-mismatch debugging, torch.cuda.memory_summary()	Intermediate
6	Q&A Review Bank	15 Q&A pairs across all topics	All levels

Recommended Learning Paths

Path A: Beginner - First PyTorch Model

1. Tensors & Autograd - understand tensors and how gradients flow
2. Dataset & DataLoader - learn to feed data into a model
3. Training Loop From Scratch - write your first end-to-end training loop
4. Raw PyTorch Classifier - apply everything in a runnable CNN

Path B: Interview Preparation (Accelerated)

1. Tensors & Autograd - gradient accumulation and .backward() questions are very common
2. Training Loop From Scratch - train/eval mode, why zero_grad() matters
3. Checkpointing & Mixed Precision - AMP questions come up in production-focused interviews
4. Q&A Review Bank - drill all 15 questions

Path C: Production/Debugging Focus (Advanced)

1. Debugging & GPU Memory - CUDA OOM triage and shape-mismatch fixes
2. Checkpointing & Mixed Precision - resuming long-running training jobs
3. Raw PyTorch Classifier - a complete, checkpointed, AMP-enabled training run

Resources

• Q&A Review Bank - 15 Q&A pairs in this module
• Cross-topic Interview Questions
• Raw PyTorch Classifier Code Lab - hands-on CNN trained end-to-end

Key Cross-References

• Pretraining objectives and how LLMs are trained at scale → Training & Pretraining
• LoRA/QLoRA fine-tuning built on the same autograd + optimizer mechanics covered here → Fine-Tuning
• VRAM estimation and mixed-precision tradeoffs at model scale → GPU & Hardware
• Hands-on fine-tuning that builds directly on these mechanics → Fine-Tuning Lab

Next Topic

This is a standalone Programming Languages reference module - see Knowledge Check to review across all modules, or revisit Fine-Tuning Lab to see these mechanics applied to LLM fine-tuning.