IN THIS BREAKDOWN · 10 SECTIONS
You can read 18% of this breakdown.
The paper in one minute
NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs: values in E2M1 (sign plus 0, 0.5, 1, 1.5, 2, 3, 4 or 6), grouped in blocks of 16 that share an 8-bit E4M3 scale, with one FP32 scale per tensor. The authors give it two to three times FP8's arithmetic throughput and about 1.8 times less memory. Very large models quantize to it with little loss using post-training quantization (PTQ); small ones do not.
The standard repair is quantization-aware training (QAT): keep training the quantized model with its original loss and data, so the weights adapt to the quantization grid. That assumes you can replay the original training. Modern models go through SFT, several RL stages and model merging, and RL data is mostly prompts, not answers.
Quantization-aware distillation (QAD) replaces the task loss with KL divergence between the original BF16 model (teacher) and the quantized model (student) on the same inputs. Run it once, on a mix of SFT data or the model's own generations.
On Llama Nemotron Super V1, Nemotron Nano V2, Nemotron Nano V2 VL, Nemotron 3 Nano and AceReason Nemotron 1.1, QAD brought NVFP4 accuracy close to BF16. On the RL-heavy models, QAT on cold-start SFT data scored below doing nothing (PTQ): it pulled the model back toward its pre-RL behavior. QAD tolerated partial-domain data (code-only training recovered math), generated data, and even random tokens without breaking the model. It needed 0.3 to 6 billion tokens and conservative learning rates. NVFP4 checkpoints and code are released.
Research notes: the problem, approach and results
KL divergence from the BF16 model after QAD versus QAT on Llama Nemotron Super V1; both reached the same cross-entropy
AIME25 for AceReason Nemotron 7B in NVFP4 after QAT versus QAD; BF16 scores 63.5 and untrained PTQ 58.7
tokens of training needed to converge, against the original post-training budgets
NVFP4 4-bit inference roughly halves memory against FP8 on new NVIDIA GPUs, but post-training quantization costs small models noticeable accuracy. The usual fix, quantization-aware training, needs the original training pipeline, which for modern models means SFT, RL and model merging that nobody can easily replay.
Train the quantized model to match the full-precision model's next-token distribution with KL divergence (quantization-aware distillation), in one stage, on whatever data is at hand: SFT data, generations from the model itself, even random tokens. Keep the original model as the teacher.
Across five NVIDIA models, QAD recovered to within about 3 points of BF16 on every reported benchmark, while QAT on the same data fell below plain quantization on RL-trained models (AceReason AIME25: BF16 63.5, PTQ 58.7, QAT 46.1, QAD 62.0). It needed 0.3 to 6 billion tokens and worked with partial-domain or synthetic data.
Why it matters to an FDE
Customers serving open models on Blackwell-class GPUs will be offered NVFP4 for memory and throughput. The decision an FDE faces is what to do when 4-bit quantization costs accuracy on a small or mid-size model.
Don't fine-tune the quantized model on task data to fix it. That is QAT, and on RL-trained reasoning models it made things worse than plain PTQ: AceReason's AIME25 fell from 58.7 (PTQ) to 46.1. Training on labels re-teaches the model whatever its training data says, which for an RL-tuned model is an older, weaker version of itself. The validation loss looked fine; the behavior changed.
Distill from the original instead. QAD only needs the BF16 model and some inputs. The authors' data ablation is reassuring for customers without the original training data: the model's own generations from task prompts worked as well as the real SFT data.
Check behavior, not loss. Table 1 is the lesson in one row: QAT matched BF16 cross-entropy exactly while its KL from the original was 78 times QAD's. When validating any quantized model for a customer, compare its outputs to the original's, not just its loss or one benchmark.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
What QAD needs, where QAT broke, data and learning-rate findings, and three misreadings
Free account - 04How it works
The format, fake quantization in training, the QAD loss, data, layer choices and hyperparameters
Premium - 05The numbers that matter
SFT-heavy and RL-heavy results, data ablations, learning rates, loss choice and large-model PTQ
Premium - 06Try it yourself
Recheck the tables, build NVFP4 and compare formats, then watch QAT drift and QAD hold
Premium - 07Where it breaks
Remaining gaps, NVIDIA-only models and hardware, evaluation choices, missing costs and slips
Premium - 08Using it on a deployment
Deciding between PTQ and QAD for a customer's model, assembling data, and validating the 4-bit model
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium