TL;DR: Organize the comparison by what each strategy communicates and how often: DP all-reduces gradients once per step (overlappable, scales to thousands), TP all-reduces activations inside every layer (huge, on the critical path, so it stays on NVLink at degree 2-8), PP sends only activations at stage boundaries (cheap, network-tolerant, but pays a pipeline bubble). Frontier runs compose all three as 3D parallelism in that communication order.
How to approach it
Do not recite definitions in isolation. Organize the comparison around two axes: what gets split, and what gets communicated (how much, how often, how latency-sensitive). The communication axis is what determines how the three map onto real hardware, and it is where interviewers separate people who have read the Megatron paper from people who have run it.
A strong answer
Data parallelism (DP) replicates the full model on every GPU; each rank processes different batches and all-reduces gradients once per step. Communication: one all-reduce of the full gradient set per step (~2 bytes/param in BF16), overlappable with the backward pass. It is the default whenever the model fits in one GPU's memory, scales to thousands of ranks, and its sharded variants (ZeRO/FSDP) extend it well past the fits-in-memory constraint.
Tensor parallelism (TP) splits individual layers, the weight matrices of each GEMM, across GPUs, Megatron-style (column-split then row-split, so one all-reduce per MLP block and per attention block). Communication: all-reduces of activations inside every layer, every microbatch, forward and backward. That is enormous volume on the critical path with nothing to overlap behind, which is why TP lives within a node on NVLink (~900 GB/s) and almost never crosses InfiniBand. Use it when a layer or the whole model will not fit, or to cut latency; typical degree is 2-8, matching the NVLink domain.
Pipeline parallelism (PP) splits the model by depth: contiguous groups of layers become stages on different nodes, microbatches flow through like an assembly line. Communication: just activations at stage boundaries, point-to-point, modest volume, latency-tolerant. The cost is not bandwidth, it is the bubble: stages idle while the pipeline fills and drains. Bubble fraction ≈ (stages − 1) / microbatches, so you need many microbatches in flight; 1F1B scheduling and interleaved stages shrink it further.
3D parallelism composes them in the order their communication demands dictate: TP innermost (within the 8-GPU NVLink island), PP across nodes (cheap point-to-point over the fabric), DP outermost (overlappable all-reduce, also across the fabric). A Llama-405B-style run looks like TP=8, PP=16, DP=whatever is left, for example 8×16×16 = 2,048 GPUs. The decision procedure I would give: model fits on one GPU, pure DP; does not fit, FSDP next (simplest), or TP up to 8 if FSDP's communication or memory profile loses; still does not fit or DP width is exhausted, add PP; then size microbatches to keep the bubble under ~10%.
The decision procedure earns more trust when you run it on real sizes, and the 16-bytes-per-parameter training-state rule makes it one division (arithmetic verified). Llama-70B on 64 H100-80GBs: 70B x 16 bytes = 1,120 GB of parameters, gradients, and optimizer states; sharded across 64 GPUs by FSDP that is 17.5 GB per GPU, leaving ample headroom for activations, so plain FSDP is comfortable and no exotic layout is justified. Same cluster, a 405B model: 6,480 GB / 64 = 101 GB per GPU of state alone, over the card's capacity before the first activation, so the answer is structurally different, more GPUs, or TP within nodes plus PP across them to bring per-GPU state under budget. Two divisions, two opposite architectures, and the point for the interview: "which parallelism?" is not a taste question, it is arithmetic you can do before touching a cluster, and doing it out loud is the difference between choosing a layout and reciting one.
Worth one sentence each: expert parallelism (MoE routing via all-to-all) and sequence/context parallelism for long sequences are the 4th and 5th dimensions in modern stacks.
Sorted by the communication axis that decides the layout:
| Strategy | What it splits | Comm volume | Comm frequency | Typical degree | Where it runs |
|---|---|---|---|---|---|
| Data (DP) | batches across model replicas | gradient all-reduce, ~2 bytes/param | once per step (overlappable) | thousands | across fabric |
| Tensor (TP) | weight matrices of each GEMM | activation all-reduces, huge | every layer, every microbatch, fwd + bwd | 2-8 | inside node on NVLink |
| Pipeline (PP) | layers by depth into stages | boundary activations, point-to-point | at stage boundaries | (matches node count) | across nodes |
What interviewers probe next
- "Why does TP stay on NVLink but DP can cross the network?" TP all-reduces activations per layer on the critical path; DP all-reduces gradients once per step and hides behind backward. Frequency times overlappability, not just volume.
- "What is the pipeline bubble for 8 stages and 32 microbatches?" (8−1)/32 ≈ 22% naive, which is why you would push microbatches up or interleave virtual stages.
- "Why not just FSDP everything?" At some scale per-GPU shards get thin, all-gather latency dominates, and activation memory still blows up on huge layers; hybrid sharding or TP+PP wins past roughly the few-hundred-GPU / 100B-param regime.
- "Which would you use for inference?" TP (and sometimes PP for giant models); DP is just replicas. Different problem: latency and KV memory, not gradient sync.
Common mistakes
- Defining all three correctly but having no answer for "when," which is the actual question. The fits-in-memory DP, layer-too-big TP, depth-across-nodes PP ladder is the minimum.
- Ignoring communication frequency: candidates compare bytes moved and conclude PP is "expensive" because moving activations sounds big. It is the cheapest of the three on the wire.
- Claiming PP saves compute. It saves memory; it adds bubble overhead and only helps throughput by enabling bigger models/batches.
- Never mentioning that these compose. Real frontier runs are 3D (or 4D with MoE); treating the three as exclusive alternatives dates the answer to 2019.
Key takeaways
- Sort the three by communication: DP once-per-step and overlappable, TP per-layer and unhideable, PP boundary-only and cheap.
- TP must stay inside the NVLink domain (degree 2-8); PP and DP are the dimensions that cross the fabric.
- 3D parallelism nests TP inside PP inside DP; the layout is forced by where each collective can afford to land.