Walk me through the GPU memory hierarchy, registers, shared memory, L2, HBM. What lives where and why?

TL;DR: Registers, shared memory, L2, and HBM trade size for speed across roughly 1 cycle to 600 cycles. Fast kernels exist because they stage data into the small fast levels and reuse it many times per HBM byte (tiled matmul, FlashAttention), since an H100's ~1,000 BF16 TFLOPs would starve on its ~3.35 TB/s of HBM if every operand came from global memory.

How to approach it

List the levels fast, then spend your time on the two that decide performance: shared memory and HBM. Attach approximate numbers. Interviewers at NVIDIA hear "registers are fast, global is slow" from everyone; the differentiator is knowing roughly how fast and what you would stage in each level for a real kernel like a tiled matmul.

A strong answer

From fastest to slowest, on a modern datacenter GPU (H100-class numbers, all approximate):

Plus two specialized spaces: constant memory (small, cached, broadcast-optimized, ideal when every thread reads the same value) and local memory, which is a trap. It is where register spills go, and despite the name it lives in HBM.

The reason the hierarchy matters: an H100 can do roughly 1,000 TFLOPs in BF16 but HBM only delivers ~3.35 TB/s. If every operand came from HBM you would be limited to a few hundred GFLOP-equivalents of useful math, orders of magnitude below peak. Fast kernels exist because data gets staged into shared memory and registers and reused many times per HBM byte.

Divide the two headline numbers and you get the whole optimization discipline in one figure: ~1,000 TFLOPs against ~3.35 TB/s means a kernel must perform roughly 295 floating-point operations per byte it pulls from HBM to keep the math units fed (computed from the specs, so quote it as "about 300"). Now score a BF16 vector add against that bar: two loads and one store per element is 6 bytes of traffic for 1 FLOP, an arithmetic intensity of 0.17. At 3.35 TB/s that caps out near 0.56 TFLOPs, about 0.06% of the chip's peak, and no amount of clever scheduling changes it, because the operation simply does not reuse anything. That is the roofline model in one worked line, and it explains kernel fusion in the same breath: fusing elementwise ops into the matmul that produced their input costs nearly nothing in compute and saves a full HBM round trip per element.

The canonical example is tiled matrix multiply: each block loads a tile of A and a tile of B from HBM into shared memory once, synchronizes, and then every thread reads those tiles dozens of times at scratchpad speed while accumulating in registers. Same data, roughly 100x cheaper access on every reuse. FlashAttention is the same idea applied to attention: it never materializes the full attention matrix in HBM, computing it tile-by-tile in shared memory, which is why it wins despite doing strictly more FLOPs (it recomputes in the backward pass).

The hierarchy also creates the failure modes interviewers probe. Use too many registers per thread and fewer warps fit per SM (occupancy drops, or values spill to "local" memory in HBM); shared memory has 32 banks and conflicting accesses serialize; uncoalesced global loads waste most of each memory transaction.

The four levels by speed and reach (H100-class, approximate):

What interviewers probe next

• "Your kernel got slower when you added a small per-thread array, why?" The compiler likely could not keep a dynamically indexed array in registers, so it spilled to local memory, meaning HBM round-trips on every access.
• "When would you use constant memory?" Values read uniformly by all threads (filter coefficients, scalar hyperparameters); the broadcast cache serves a warp in one shot.
• "Where does the KV cache live during LLM inference?" HBM, which is exactly why decode is memory-bandwidth-bound: weights plus KV get streamed from HBM for every generated token.
• "What changed with Hopper?" Bigger combined L1/shared (up to ~228 KB), TMA for async bulk copies into shared memory, and thread-block clusters allowing shared-memory access across blocks on neighboring SMs.

Common mistakes

• No numbers. Saying "shared memory is faster than global" without "tens of cycles vs hundreds, terabytes/s of aggregate scratchpad bandwidth" reads as memorized trivia.
• Believing local memory is fast because it sounds private. Spills are a classic silent performance killer; you find them with -Xptxas -v or Nsight showing local memory traffic.
• Describing shared memory as a transparent cache. It is programmer-managed; nothing lands there unless your code (or a library kernel) puts it there.
• Explaining the hierarchy but never connecting it to a real kernel. The tiled-matmul story is the minimum; mentioning FlashAttention's shared-memory tiling signals you connect architecture to the ML workloads these companies actually run.

Key takeaways

• Know the rough latencies (1 / tens / hundreds of cycles) and the ~3.35 TB/s HBM figure; the numbers are the answer.
• Performance comes from reuse per HBM byte: tiled matmul and FlashAttention both stage into shared memory.
• Local memory is HBM in disguise; spills are the silent killer found with -Xptxas -v or Nsight.