GPT-4 Technical Report

The GPT-4 technical report describes a large multimodal model that takes images and text and produces text, trained to predict the next token and then fine-tuned with reinforcement learning from human feedback. It is unusual among model papers for what it leaves out. Citing competition and safety, it gives no architecture, model size, hardware, training compute, dataset construction or training method. What remains is an evaluation report, and three parts of it hold up as research.

Predictable scaling. OpenAI fitted a power law with an irreducible-loss term, L(C) = aC^b + c, to models trained with the same method but at most 10,000 times less compute, and predicted GPT-4's final loss on an internal code dataset shortly after the run began. It also predicted the pass rate on a subset of HumanEval coding problems from models with 1,000 times less compute.

Exams and benchmarks. With no exam-specific training, GPT-4 scored 298/400 on a simulated Uniform Bar Exam (about the 90th percentile; GPT-3.5 was about the 10th), 163 on the LSAT, and 86.4% on MMLU against GPT-3.5's 70.0%. Questions found in the training data were removed and the lower of the two scores reported.

What RLHF changed. Across 28 exam sections, the base and RLHF models scored 73.7% and 74.0%: post-training did not add capability. It did change calibration: the pre-trained model's confidence matched its accuracy closely (expected calibration error 0.007), while the post-trained model's did not (0.074). Safety training cut responses to disallowed requests by 82% relative to GPT-3.5.

This report is the template for closed-model documentation, and it teaches you how to read one.

Know what you are not told. With no size, data or method, you cannot reason from the architecture about cost, memory or failure modes. Your evaluation of a closed model has to be empirical, on the customer's task.

Do not use a chat model's confidence as a probability. The report shows post-training made GPT-4's stated confidence much less reliable. If a workflow routes low-confidence answers to a human, calibrate the threshold on labeled data rather than trusting the model's own numbers.

Benchmarks need contamination checks, and those checks are partial. OpenAI found BIG-bench had been mixed into training and dropped it, included part of GSM-8K's training set on purpose, and did not check its RLHF data against TruthfulQA. Treat public benchmark numbers as evidence, not proof, and build a held-out set from the customer's own data.

Predictable scaling is a planning tool. A small-scale fit that includes an irreducible-loss term can predict a large run. The same idea helps size fine-tuning runs before committing the budget.