Evaluating Large Language Models Trained on Code

Before this paper, generated code was usually graded by comparing it with a reference solution, token by token or with BLEU. A correct program that names its variables differently scores poorly; a broken program that looks like the reference scores well. The authors propose grading by execution: run the generated function against unit tests, the way a developer would accept a pull request.

To do that without training-set leakage they wrote HumanEval: 164 Python problems by hand, each with a function signature, a docstring, a reference body and an average of 7.7 unit tests. The problems test language comprehension, simple algorithms and arithmetic, some at the level of an easy interview question. They then fine-tuned GPT models of up to 12 billion parameters on 159 GB of Python collected from 54 million public GitHub repositories in May 2020, and called the result Codex. A descendant powered GitHub Copilot.

The metric is pass@k: draw k samples per problem and count the problem solved if any one passes. Computing it naively from exactly k samples is noisy, so the paper draws n = 200 samples and uses an unbiased estimator, 1 - C(n-c, k)/C(n, k) where c samples passed. With one sample, Codex-12B passed 28.8% of problems, GPT-J 11.4% and every GPT-3 model close to 0%. With 100 samples, at least one passed for 72.3%. A further fine-tune on about 50,000 standalone functions with tests, Codex-S, reached 37.7% with one sample and 77.5% with 100.

The paper also studies how to choose one sample without tests (mean log-probability works best), a docstring-writing model (Codex-D), the APPS contest benchmark, limitations on long specifications, and a broad hazard analysis.

Grade generated code by running it. When a customer wants a model that writes SQL, scripts or transformations, the evaluation that matters is whether the output executes and produces the right result on test inputs. The paper's Figure 8 shows that wrong programs often score higher BLEU than right ones, so a text-similarity score can rank the worse model first.

Pick the k that matches the product. An autocomplete shows one suggestion, so pass@1 is the number. An agent that can run tests and retry is closer to pass@k, because it can throw away samples that fail. The gap is large: 28.8% against 72.3% for the same model. Quoting pass@100 for a product that shows one answer overstates it by more than a factor of two.

Tune the temperature to k. The best temperature for pass@1 was 0.2 and for pass@100 was 0.8. If you sample many candidates and filter, low temperature wastes the budget on near-duplicates.

Run generated code in a sandbox. The authors executed untrusted code inside gVisor containers with eBPF firewall rules blocking network access. Any customer system that executes model-written code needs the same isolation before it needs a better model.

Know how wide your test set's error bar is. With 164 problems, a 28.8% score carries a 95% interval of about plus or minus 7 points. Two models a few points apart on HumanEval are not separated by it, and the same holds for a customer's own 150-item golden set.