Distributed Training at Scale

GPU & Hardware introduced data, tensor and pipeline parallelism and ZeRO. Frontier runs combine all of them - plus context and expert parallelism - across tens of thousands of GPUs, and they have to survive hardware failures every few hours. This note is about how those pieces fit together, how low-precision (FP8) training works, and how you know whether a run is efficient.

The Parallelism Dimensions

flowchart TD
    subgraph CLUSTER["🏢 Cluster (thousands of GPUs)"]
        direction TB
        subgraph DPG["📋 Data-parallel / FSDP replicas - across the slowest links"]
            direction LR
            subgraph PPG["🚚 Pipeline stages - across nodes"]
                direction LR
                subgraph NODE["🖥️ One node / NVLink domain"]
                    TPG["✂️ Tensor + sequence parallel<br/>(fastest links)"]
                    CPG["🔄 Context parallel"]
                    EPG["🧩 Expert parallel"]
                end
            end
        end
    end

    style TPG fill:#d8dfe8,stroke:#b0bac8
    style CPG fill:#dde4dc,stroke:#b0c4b0
    style EPG fill:#e8e0d4,stroke:#c8b89a

The layout rule: put the most communication-heavy dimension on the fastest links. TP talks every layer, so it stays inside an NVLink domain (8 GPUs on HGX servers, 72 on a GB200 NVL72 rack). DP/FSDP talks once per step (and overlaps with compute), so it spans the data-center network.

A published example - Llama 3 405B (16K H100s): TP = 8 within each server, PP = 16 across servers, CP up to 16 for long-context stages, and data parallelism across the rest, with a modified FSDP. Meta reported 38-43% MFU in BF16.

The pipeline bubble

With p pipeline stages and m micro-batches per step, a simple 1F1B schedule leaves each GPU idle for roughly (p - 1) / (m + p - 1) of the step. More micro-batches shrink the bubble; interleaved schedules (several smaller stages per GPU) and DeepSeek's DualPipe (which overlaps forward and backward chunks, and their communication) shrink it further.

Context parallelism

Attention's memory grows with sequence length, and at 128K tokens even FlashAttention's activations don't fit on one GPU for a large model. Ring attention splits the sequence across GPUs: each holds a chunk of queries and passes key/value chunks around a ring, computing blockwise attention as they arrive, overlapping communication with compute.

Expert parallelism

An MoE layer sends each token to its top-k experts, which live on different GPUs. That means two all-to-all exchanges per MoE layer (dispatch and combine). The costs to manage: load imbalance (a hot expert stalls everyone), and all-to-all bandwidth across nodes. DeepSeek-V3 limited each token to experts on at most 4 nodes and wrote custom communication kernels to overlap it with compute.

FP8 Training

Concept

Hopper and Blackwell GPUs run FP8 matrix multiplies at roughly twice their BF16 throughput. FP8 has two formats:

FP8's range is tiny, so every tensor needs a scale factor. The granularity of that scale is the main design choice:

• Per-tensor scaling (NVIDIA Transformer Engine's original recipe): one scale per tensor, often computed from a history of recent maximum values ("delayed scaling"). Simple, but one outlier value wastes the range for the whole tensor.
• Fine-grained scaling (DeepSeek-V3): per 1×128 tile for activations and per 128×128 block for weights, with higher-precision accumulation. It tolerates outliers and was used to train a 671B MoE in FP8 end to end.
• Microscaling formats (MXFP8, and the 4-bit MXFP4/NVFP4 on Blackwell): the scale is part of the format - one shared scale per small block of values - so hardware applies it directly.

What stays in higher precision: master weights and optimizer state (FP32), and sensitive operations - embeddings, the output head, normalization, attention softmax, and often the MoE router.

DeepSeek-V3's full training took 2.788M H800 GPU-hours, which DeepSeek attributes in part to FP8 training and its communication/compute overlap.

Measuring Efficiency: MFU

MFU = achieved model FLOPs per second / peak hardware FLOPs per second
    = (6 · N · tokens_per_second) / (num_gpus · peak_flops_per_gpu)

Use the dense peak (vendor spec sheets often quote sparse numbers that are 2× higher) in the precision you actually train in. Rough bands for large dense models on H100-class GPUs:

Hardware FLOPs utilization (HFU) also counts recomputed activations, so HFU ≥ MFU when activation checkpointing is on.

Fault Tolerance

At 16K GPUs, failures are routine. During a 54-day period of Llama 3 405B pretraining, Meta logged 466 job interruptions, 419 of them unexpected - most traced to GPU or HBM faults, with others from network, host and software issues. The run still achieved over 90% effective training time.

What a large run needs:

• Frequent, fast checkpoints. Asynchronous, sharded saves (e.g. torch.distributed.checkpoint.async_save) so saving doesn't stall training; in-memory or local-SSD checkpoints between slower remote saves.
• Automated detection and restart. Health checks, NCCL timeouts and watchdogs to catch hung collectives; spare nodes to swap in; automatic resume from the last checkpoint.
• Deterministic data loading. The data loader must resume at exactly the right sample after a restart, or you silently repeat or skip data.
• Straggler and silent-corruption detection. One slow GPU slows every synchronous step; silent data corruption shows up as unexplained loss spikes, so compare per-rank statistics and replay suspicious steps.

Check Yourself

Exercises

Study Notes

Must-know:

• Five parallelism dimensions: DP/FSDP, TP (+SP), PP, CP (ring attention), EP (all-to-all)
• Map the chattiest dimension to the fastest links: TP inside an NVLink domain; DP/FSDP across the network
• Pipeline bubble ≈ (p - 1)/(m + p - 1); shrink with more micro-batches, interleaving, DualPipe
• FP8: E4M3 forward / E5M2 backward; scaling granularity (per-tensor, fine-grained, microscaling) is the key design choice; master weights and sensitive ops stay in higher precision
• MFU = 6N × tokens/s ÷ (GPUs × dense peak); 30-45% is typical for well-tuned large runs
• Failures are routine at scale; async sharded checkpoints, auto-restart and deterministic data loading are mandatory

References

• Shoeybi et al., Megatron-LM (2019); Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters (2021)
• Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models (2022) - sequence parallelism
• Rajbhandari et al., ZeRO (2019); Zhao et al., PyTorch FSDP (2023)
• Liu et al., Ring Attention with Blockwise Transformers (2023)
• Micikevicius et al., FP8 Formats for Deep Learning (2022); Rouhani et al., Microscaling Data Formats (2023)
• DeepSeek-AI, DeepSeek-V3 Technical Report (2024) - fine-grained FP8, DualPipe, expert parallelism
• Llama Team, The Llama 3 Herd of Models (2024) - 4D parallelism, MFU, failure statistics
• Liang et al., TorchTitan (2024)

Last reviewed: 2026-09