PyTorch enforces that operations between tensors require matching device and broadcast-compatible shape - both raise explicit RuntimeErrors with messages that name the mismatch directly, which makes them straightforward to triage once you know how to read them. GPU memory (VRAM) is consumed by four categories that grow with batch size and model size: parameters, gradients, optimizer state (e.g. Adam's two moment estimates per parameter), and activations retained for the backward pass - the last of these is usually the largest and most controllable via batch size, gradient checkpointing, and mixed precision.

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = MyModel().to(device)          # move model parameters once, outside the loop

for batch_x, batch_y in train_loader:
    batch_x = batch_x.to(device)       # move each batch, every iteration
    batch_y = batch_y.to(device)
    outputs = model(batch_x)            # now both are on the same device - safe

Common device-mismatch error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices,
cuda:0 and cpu!

Debugging checklist:

• Model moved to device exactly once, right after construction (or after loading a checkpoint)
• Every batch from the DataLoader moved to device inside the loop (data loaders yield CPU tensors by default)
• Any tensor created inside the forward pass (e.g. a mask, a constant) also created on device, or with .to(x.device) matching an existing input
• Loss targets moved to device, not just the inputs - easy to miss

Common causes and fixes:

# Debugging pattern: print shapes at every stage during development
def forward(self, x):
    print("input:", x.shape)
    x = self.conv1(x); print("after conv1:", x.shape)
    x = self.pool(x);   print("after pool:", x.shape)
    x = x.view(x.size(0), -1); print("after flatten:", x.shape)
    x = self.fc1(x);    print("after fc1:", x.shape)
    return x

RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 15.78 GiB total capacity;
13.24 GiB already allocated; 1.02 GiB free; 13.90 GiB reserved in total by PyTorch)

What consumes VRAM, roughly in order of controllability:

1. Activations (largest, most controllable) - every intermediate tensor kept for the backward pass; scales with batch size and sequence/image resolution
2. Optimizer state - Adam stores two extra tensors per parameter (first/second moment estimates), roughly 2x the parameter memory on top of the parameters themselves
3. Gradients - one tensor per parameter, same size as the parameters
4. Model parameters - fixed, doesn't scale with batch size

Mitigation, roughly in order to try first:

# 1. Reduce batch size (biggest, simplest lever)
train_loader = DataLoader(train_ds, batch_size=16)  # was 64

# 2. Use mixed precision (roughly halves activation memory)
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    outputs = model(batch_x)

# 3. Use gradient accumulation to keep effective batch size while lowering peak memory
#    (see 01-Tensors-and-Autograd.mdx for the accumulation pattern)

# 4. Gradient checkpointing - recompute activations during backward instead of storing them
from torch.utils.checkpoint import checkpoint
x = checkpoint(self.expensive_block, x)

# 5. Clear cached (but unused) memory PyTorch is holding onto
torch.cuda.empty_cache()

# 6. Inspect current allocation in detail
print(torch.cuda.memory_summary(device=device, abbreviated=True))

torch.cuda.memory_summary() prints a breakdown of allocated vs. reserved memory and the largest allocation blocks - useful for identifying whether memory is fragmented (many small blocks, empty_cache() may help) versus genuinely exhausted (need to actually reduce what's held, e.g. lower batch size).

Gradient checking basics: when debugging whether gradients are flowing correctly (e.g. after modifying a custom autograd.Function or freezing/unfreezing layers), inspect .grad directly after .backward():

for name, param in model.named_parameters():
    if param.grad is None:
        print(f"{name}: NO GRADIENT (check requires_grad or graph connectivity)")
    elif torch.isnan(param.grad).any():
        print(f"{name}: NaN gradient - check for exploding gradients or bad loss scaling")