IN THIS BREAKDOWN · 10 SECTIONS
You can read 23% of this breakdown.
The paper in one minute
Post-training quantization squeezes a trained model's weights into fewer bits, and it works well down to about 4 bits. Below that, rounding after the fact destroys too much: on the paper's 6.7B model, GPTQ at 2 bits scored a perplexity of 1,032 against 15.19 in FP16, and absmax at 1 bit produced numbers in the 10^23 range. BitNet takes the opposite route. If weights are going to be one bit, train them as one bit from the start, so the rest of the network learns around that constraint.
The building block is BitLinear, a drop-in replacement for a linear layer. It normalizes its input (a LayerNorm in the SubLN position) so the output keeps unit variance after quantization, quantizes activations to 8 bits by their absolute maximum, and binarizes the weights to +1 or −1 by the sign of each weight minus the matrix mean. The matrix multiply then needs only additions and subtractions, followed by one rescale. Training keeps a full-precision copy of every weight (the latent weights) that accumulates gradient updates, uses a straight-through estimator to pass gradients through the sign function, and uses a large learning rate, because small updates to the latent weights rarely flip any sign.
Trained on the same data as FP16 baselines from 125M to 30B parameters, BitNet's loss falls with model size following a power law like the FP16 curve, and the gap narrows as models grow. Fitting the law on models up to 6.7B predicted the 13B and 30B losses accurately. At 6.7B, BitNet's zero-shot average was within 2 points of FP16 and above every post-training method at 4 bits or fewer. An energy model puts its matrix-multiply energy 28 to 38 times below FP16. The successor, BitNet b1.58, adds a zero to the weight values.
Research notes: the problem, approach and results
zero-shot average of 1-bit BitNet and the FP16 Transformer at 6.7B (Table 3)
best post-training result at 2 bits (QuIP) on the same 6.7B comparison; 1-bit PTQ was near chance (Table 3)
estimated matrix-multiplication energy reduction against FP16 at 30B, 7nm, 512 tokens (Figure 1)
Post-training quantization works down to about 4 bits and then falls apart; at 2 or 1 bit, models quantized after training are close to useless. Large language models are expensive to serve mostly because of their matrix multiplications and memory traffic.
Train the model with 1-bit weights from the start. BitLinear binarizes weights with the sign function around their mean, quantizes activations to 8 bits with absmax, normalizes inputs (SubLN) to keep output variance stable, and trains with full-precision latent weights, a straight-through estimator and a large learning rate.
From 125M to 30B parameters, BitNet's loss follows a power law like FP16 Transformers', with the gap shrinking as size grows. At 6.7B, its four-task zero-shot average was 55.9 against 57.8 for FP16, while post-training methods scored 49.0 at 2 bits and near chance at 1 bit. Estimated matrix-multiply energy fell by 28x to 38x.
Why it matters to an FDE
BitNet is a research direction, not a tool you can apply to a customer's existing model, and that distinction is the main thing to carry into conversations.
It requires pretraining. BitNet's quality comes from training with 1-bit weights from scratch. You cannot convert a downloaded Llama to BitNet the way you can quantize it with GPTQ. Unless a 1-bit model of the size and quality the customer needs has been released, this paper changes nothing about today's deployment.
The savings are in arithmetic and memory traffic. With ±1 weights, matrix multiplication becomes addition; the paper's energy model says multiply-heavy FP16 matmuls cost tens of times more energy. Realizing that needs kernels or hardware built for it. On today's GPUs, the paper reports no measured speed, only estimated energy.
It sets expectations for "extreme quantization" claims. The comparison that matters is BitNet against post-training methods at the same bits: 55.9 against 49.0 at 2 bits. If a vendor offers a 1- or 2-bit version of a normal model, this paper is the evidence that post-training methods at those widths lose a lot.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
BitLinear's parts, the training recipe, the scaling result, the energy model and two misreadings
Free account - 04How it works
BitLinear step by step, why the normalization matters, the training tricks and group quantization
Premium - 05The numbers that matter
The 6.7B comparison table, scaling results, energy estimates and the training recipe
Premium - 06Try it yourself
Rebuild the energy table, train a 1-bit classifier with straight-through gradients, recheck Table 3
Premium - 07Where it breaks
Training cost, no released models in the paper, estimated energy only, narrow evaluation
Premium - 08Using it on a deployment
How to evaluate a 1-bit model offer and what to ask before planning around one
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium