The Prompt Report: A Systematic Survey of Prompt Engineering Techniques

Prompting papers multiplied after ChatGPT, often inventing new names for old ideas. Sander Schulhoff, Philip Resnik and a large group at the University of Maryland, Learn Prompting, OpenAI, Stanford, Microsoft and elsewhere set out to make a map. The Prompt Report is a systematic survey: they searched arXiv, Semantic Scholar and the ACL Anthology with 44 keywords, deduplicated to 4,247 records, and kept 1,565 papers on hard, prefix-style prompting after human and GPT-4 screening.

From those papers they define 33 vocabulary terms (prompt, prompt template, exemplar, answer extractor and so on) and organize 58 text prompting techniques into six families. Few-shot techniques choose and order examples. Zero-shot techniques change instructions without examples, such as role prompting or asking the model to re-read the question. Thought generation elicits reasoning, starting with chain of thought. Decomposition splits problems into sub-problems, as least-to-most or plan-and-solve do. Ensembling combines several outputs, as self-consistency does. Self-criticism asks the model to check and revise, as Self-Refine and Chain-of-Verification do. They add 40 techniques for other modalities and languages, and sections on agents, evaluation, prompt injection and jailbreaking, and alignment issues such as sensitivity and calibration.

Two small studies give the survey some data. On 2,800 MMLU questions with gpt-3.5-turbo, few-shot chain of thought scored best (69.2%) and zero-shot chain of thought worst (54.7%). And a case study follows one expert prompt engineer, also the lead author, through 47 documented steps and about 20 hours, tuning GPT-4 to detect a suicide-risk signal called entrapment in Reddit posts. The best manual prompt reached an F1 of 0.53 on the development set; an automated DSPy pipeline then did better on the test set, at 0.548.

The taxonomy is a checklist for when a prompt is not good enough: did you try better examples, a decomposition, an ensemble, a verification step? It also gives you and the customer the same words for those choices.

The case study is the more important part for field work, because it shows what prompt engineering actually looks like on a real task with a small labeled set.

Hand tuning is fragile. Removing an accidentally duplicated email from the prompt lowered F1 by 0.07; anonymizing the names in it lowered F1 by 0.08. Nobody could explain why. A prompt that depends on details like that will break when anything changes.

Small dev sets mislead. With about 111 scored posts, F1 moves several hundredths from sampling alone, and reruns at temperature 0 moved it by up to 0.04. Many of the 47 steps changed F1 by less than that.

Automate the search, then add judgment. DSPy's optimized prompt beat the expert's prompts on the test set. The authors' conclusion is that automation plus a human revising its output worked best.

Involve domain experts. The model's own idea of "entrapment" did not match the clinicians' definition, and the prompt only improved once that definition was in it.