A torch.Tensor wraps a contiguous memory buffer plus shape/stride/dtype/device metadata. When requires_grad=True, every operation on that tensor is recorded as a node in a dynamic, define-by-run computation graph (torch.autograd.Function objects, each holding a reference to its inputs via grad_fn). Calling .backward() traverses this graph in reverse topological order, applying the chain rule at each node to accumulate gradients into .grad on every leaf tensor that requires them.

Key attributes on every tensor: .shape (dimensions), .dtype (float32, float16, bfloat16, int64, ...), .device (cpu or cuda:0), and .requires_grad (whether autograd should track it). Tensors are created from Python lists/NumPy arrays, via factory functions (torch.zeros, torch.randn, torch.arange), or as the output of operations on existing tensors.

import torch

# Creation
x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
zeros = torch.zeros(3, 4)
randn = torch.randn(2, 3, requires_grad=True)

# Key attributes
print(x.shape, x.dtype, x.device)   # torch.Size([2, 2]) torch.float32 cpu

# Moving to GPU (no-op if no GPU is available)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
x = x.to(device)

# Reshaping - view() shares memory, reshape() may copy
x.view(4)          # flatten, must be contiguous
x.reshape(4)        # flatten, safe even if non-contiguous

# NumPy interop (shares memory on CPU!)
import numpy as np
arr = np.array([1.0, 2.0, 3.0])
t = torch.from_numpy(arr)

requires_grad=True marks a tensor as a leaf that needs gradients. Every subsequent op producing a new tensor from tracked tensors sets .grad_fn on the output, pointing back to the Function that created it (e.g. AddBackward0, MulBackward0). This forms a DAG: leaves (model parameters, inputs with requires_grad=True) at the "bottom," the scalar loss at the "top." Model parameters (nn.Parameter) have requires_grad=True by default; inputs typically don't.

w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)
x = torch.tensor(3.0)                      # no grad needed - it's data, not a parameter

y = w * x + b        # y.grad_fn -> AddBackward0
loss = (y - 10) ** 2  # loss.grad_fn -> PowBackward0

loss.backward()       # walks the graph backward from loss
print(w.grad)          # dLoss/dw
print(b.grad)          # dLoss/db

Use torch.no_grad() (or @torch.no_grad()) to disable graph-building entirely for inference/evaluation - it saves memory since PyTorch doesn't need to keep intermediate activations around for a backward pass that will never happen. tensor.detach() produces a new tensor sharing the same data but detached from the graph.

loss.backward() populates .grad on every leaf tensor with requires_grad=True, accumulating (summing) into whatever is already there. optimizer.step() then reads those .grad values and updates the parameters according to the optimizer's update rule (SGD, Adam, ...). Because gradients accumulate by default, you must call optimizer.zero_grad() (or model.zero_grad()) before each new .backward() call in the standard single-step training loop - otherwise gradients from the previous batch bleed into the current one.

import torch.nn as nn
import torch.optim as optim

model = nn.Linear(10, 1)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()

for batch_x, batch_y in dataloader:
    optimizer.zero_grad()          # 1. clear stale gradients
    pred = model(batch_x)          # 2. forward pass (builds the graph)
    loss = loss_fn(pred, batch_y)  # 3. compute loss
    loss.backward()                 # 4. backward pass (populates .grad)
    optimizer.step()                # 5. update parameters using .grad

Why gradient accumulation is sometimes deliberate: skipping zero_grad() for N batches before calling optimizer.step() is a common trick to simulate a larger effective batch size when GPU memory is limited - gradients from N small batches sum together as if they came from one big batch.

# Gradient accumulation over 4 mini-batches to simulate a 4x larger batch
accum_steps = 4
optimizer.zero_grad()
for i, (batch_x, batch_y) in enumerate(dataloader):
    pred = model(batch_x)
    loss = loss_fn(pred, batch_y) / accum_steps
    loss.backward()                 # accumulates - no zero_grad() here
    if (i + 1) % accum_steps == 0:
        optimizer.step()
        optimizer.zero_grad()