Pretraining Large Language Models with NVFP4

This report demonstrates large-language-model pretraining using NVFP4 for many matrix-multiplication operands. The central result is a twelve-billion-parameter hybrid Mamba-Transformer trained on ten trillion tokens, with learning curves and several downstream scores close to an FP8 baseline.

NVFP4 combines four-bit values with shared scale factors. Its small blocks and two-level scaling improve the representation of local ranges, but the format alone is insufficient for the reported training run. The recipe also preserves sensitive operations in higher precision, transforms weight-gradient inputs, aligns weight quantization across forward and backward passes, and uses stochastic rounding for gradients.

The model's master weights, accumulated weight gradients and optimizer states remain in FP32. Other operations retain BF16 or FP32 paths. “Four-bit pretraining” therefore describes selected arithmetic and tensor representations, not a claim that the whole training process stores or computes everything in four bits.

This page reviews the March 2026 revision of the September 2025 paper. The authors explicitly focus on numerical methodology rather than complete runtime optimization. Transformer Engine's official NVFP4 documentation is the starting point for supported implementation details. Training quality, hardware arithmetic rates and end-to-end training time remain separate measurements.

An FDE may encounter four-bit claims while selecting hardware, planning model adaptation or comparing deployment costs. Those claims can refer to different things: low-bit inference weights, adapter training on a quantized base, or low-precision arithmetic during full pretraining. The memory and quality implications are not interchangeable.

This paper supplies a concrete example of why numerical details matter. A coarse quantizer can erase small updates, outliers can distort a block's scale, and different row-wise and column-wise representations can make the backward computation inconsistent with the forward weights.

Use quantization to separate representation choices from algorithmic behavior. Before accepting a cost claim, ask which tensors change precision, which states remain wide, what hardware supports the path and how the final model was evaluated. A lower operand bit count alone does not establish a proportional reduction in the full training budget.