Attention is bottlenecked by moving the N x N score matrix to and from HBM, not by FLOPs. FlashAttention tiles the computation in fast on-chip SRAM and fuses softmax with an online running statistic, so it computes the exact same output while never materializing the full matrix.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Attention is bottlenecked by moving the N x N score matrix to and from HBM, not by FLOPs. FlashAttention tiles the computation in fast on-chip SRAM and fuses softmax with an online running statistic, so it computes the exact same output while never materializing the full matrix.
The screen is whether the candidate knows attention is memory-bandwidth bound (HBM round-trips on the N x N matrix), not compute bound, and that FlashAttention is an IO-aware tiling that keeps blocks in SRAM and uses online softmax to avoid materializing the full matrix. The reserved follow-up: 'how does softmax work without seeing the whole row, and how does the backward pass avoid storing the matrix.' Saying 'it approximates attention' fails hard; the output is numerically identical (same math, fewer memory round-trips). Bonus for the memory-complexity drop from O(N^2) to O(N) and recomputation in the backward pass.
No comments yet — be the first to share your approach.
