TL;DR: Compute the kernel's arithmetic intensity (FLOPs per byte of HBM traffic) and compare it to the hardware ridge point, ~295 FLOPs/byte on an H100 (989 dense BF16 TFLOPs / 3.35 TB/s). Below the ridge it is memory-bound, above it compute-bound, and if both DRAM and SM utilization are low it is the third state, latency-bound, where the roofline does not apply yet.
How to approach it
Give the analytical method (roofline: arithmetic intensity vs the hardware ratio) and the empirical method (profiler counters), and insist on using both. The model tells you what is possible, the profiler tells you what is happening. Then apply it to an ML-relevant kernel; LLM decode is the example interviewers at these companies care about.
A strong answer
Start with the machine balance. An H100 SXM does ~989 TFLOPs dense BF16 (label it "dense"; the datasheet often quotes the 2x sparse figure) and moves ~3.35 TB/s from HBM. The ridge point is the ratio: 989e12 / 3.35e12 ≈ 295 FLOPs per byte. Any kernel whose arithmetic intensity (FLOPs performed per byte of HBM traffic) is below ~295 cannot reach peak compute and is memory-bound by construction; above it, compute-bound. (For FP32 CUDA-core work the ridge is far lower, ~20; precision matters to the classification.)
Plotted on the roofline (log axes), every memory-bound kernel sits on the sloped bandwidth roof left of the ridge; only the large GEMM reaches the flat compute roof:
Compute arithmetic intensity for your kernel analytically. Examples worth having memorized:
- Elementwise op (GELU on BF16): ~1 FLOP per 4 bytes moved (read+write), intensity ~0.25. Hopelessly memory-bound; the only fix is fusion so the bytes are touched once for many ops.
- Large GEMM (M=N=K=8192): 2·8192³ FLOPs over 3·8192²·2 bytes, intensity ≈ 2,700. Deeply compute-bound, which is why training hits high MFU.
- LLM decode at batch 1: every generated token streams all weights (plus KV) for ~2 FLOPs per weight read of 1-2 bytes, intensity ~1-2. Memory-bound by two orders of magnitude; tokens/sec ≈ bandwidth / model bytes. Batching raises intensity roughly linearly with batch size, so you need batch in the hundreds before decode approaches the ridge, which is the entire economics of LLM serving in one sentence.
The decode line is worth finishing into an actual tokens-per-second prediction, because it turns the roofline into a forecasting tool. A 70B-parameter model in BF16 is 140 GB of weights; at batch 1 every generated token streams all of them, so the ceiling is bandwidth over bytes: 3.35 TB/s / 140 GB ≈ 24 tokens/sec on one H100, before any KV traffic or overhead (arithmetic verified). Measure a real deployment at 18-20 tokens/sec and you now know it is running at 80% of its physical ceiling and no kernel wizardry will find 2x, only smaller bytes will: FP8 weights halve the denominator and roughly double the ceiling, INT4 doubles it again on paper. Being able to predict a serving benchmark from two datasheet numbers, then explain the gap, is precisely the fluency this question is fishing for, and it is also the fastest way to sanity-check a vendor's throughput claim in a customer meeting.
Then verify empirically with Nsight Compute: the Speed-of-Light section reports SM throughput and DRAM throughput as % of peak. DRAM ~80-90% with SM low means memory-bound; the inverse means compute-bound; both low means neither. You are latency-bound (occupancy too low, divergence, sync stalls) and the roofline does not even apply yet, which trips up a lot of people. At training-job granularity, the analogous metric is MFU (model FLOPs utilization): achieved useful FLOPs over peak, where ~40-50% is a healthy large-scale run.
Remediation differs by diagnosis. Memory-bound: fuse kernels, tile through shared memory, fix coalescing, shrink data types (FP8/INT8 directly multiplies effective bandwidth). Compute-bound: use tensor cores properly, lower precision, or accept you are done. Latency-bound: occupancy and divergence work first.
What interviewers probe next
- "Why did FlashAttention win if it does more FLOPs?" Attention was memory-bound on HBM reads/writes of the N×N matrix; trading extra recompute (cheap) for eliminated HBM traffic (expensive) moves it up the roofline. Recomputation beats memorization whenever you are left of the ridge.
- "Quantizing weights to FP8 doubled my decode speed but INT4 did not give 4x, why?" At low batch you are bandwidth-bound on weight bytes, so 2x fewer bytes ≈ 2x; pushing further, dequant overhead, KV traffic, and kernel quality eat into the ideal ratio.
- "DRAM and SM utilization are both at 40%. Now what?" Latency-bound: check achieved occupancy, stall reasons, and whether you are launch-bound (many tiny kernels, fixed with CUDA graphs or fusion).
- "How does this change on a B200 or next-gen part?" Both peaks move; recompute the ridge. The method is the durable thing, not the numbers.
Common mistakes
- Jumping straight to "profile it" with no analytical model. You can classify most kernels on a napkin, and interviewers want the napkin.
- Quoting GPU utilization from
nvidia-smias evidence of compute-bound. That metric is "a kernel was resident," not "ALUs were busy"; a 100%-utilized GPU can be almost entirely stalled. - Forgetting the third state (latency-bound), then "optimizing bandwidth" on a kernel whose real problem is occupancy.
- Using FP32 peak FLOPs in the ridge calculation for tensor-core workloads, off by ~15x, which inverts the conclusion.
Key takeaways
- Ridge point ≈ 295 FLOPs/byte on an H100 (BF16); below it memory-bound, above it compute-bound.
- Decode at low batch sits at AI ~1-2, two orders below the ridge; batching is the only lever, which is the serving economics in one line.
- Both-utilizations-low is the latency-bound third state where bandwidth tuning is wasted; fix occupancy first.