IN THIS BREAKDOWN · 10 SECTIONS
You can read 23% of this breakdown.
The paper in one minute
People often add urgency or encouragement to prompts. This paper tests the habit. Drawing on three ideas from psychology (self-monitoring, social cognitive theory, and cognitive emotion regulation), the authors write 11 short emotional stimuli, EP01 to EP11. Examples: "This is very important to my career" (EP02), "You'd better be sure" (EP03), "Are you sure that's your final answer? It might be worth taking another look" (EP05), and "Take pride in your work and give it your best" (EP10). EP06 combines the first three. Each is appended to the end of an existing prompt; the combination is called EmotionPrompt.
They evaluate on 24 Instruction Induction tasks (simple tasks inferred from a few examples) and 21 BIG-Bench tasks (harder, scored so that 0 is random and 100 is a human expert), with Flan-T5-Large, Vicuna, Llama 2, BLOOM, ChatGPT and GPT-4. Baselines are the original human-written prompts, zero-shot chain of thought ("Let's think step by step"), and prompts generated by APE. Instruction Induction runs zero-shot and few-shot; BIG-Bench runs zero-shot only.
The abstract reports relative improvements of 8.00% on Instruction Induction and 115% on BIG-Bench. A human study with 106 participants rated GPT-4 answers to 30 questions with and without an emotional stimulus on performance, truthfulness and responsibility, and found a 10.9% average improvement. On TruthfulQA, using the best stimulus, ChatGPT, Vicuna and Flan-T5 improved on truthfulness and informativeness. An attention analysis on Flan-T5 suggests emotional words, especially positive ones, receive large input contributions, which the authors read as enriching the original prompt's representation.
Table 1 reports two versions of EmotionPrompt: the average over all 11 stimuli, and the best stimulus chosen separately for each task. The headline numbers come from the second.
Research notes: the problem, approach and results
the abstract's gains: means of best-of-11 relative gains in Table 1 (our recomputation)
average-of-11 relative gain, Instruction Induction zero-shot with human prompts (Table 1)
best-of-11 'gain' when no phrase does anything, in our simulation of 24 tasks
Prompt wording changes LLM results, and people already write prompts with urgency and encouragement. Nobody had tested systematically whether adding emotional or motivational phrases, drawn from psychology, improves model performance.
Write 11 short emotional stimuli based on self-monitoring, social cognitive theory and emotion regulation, such as 'This is very important to my career' and 'Are you sure?', and append each to task prompts. Test six models on 45 tasks, run a human rating study and TruthfulQA, and inspect input attention.
Reported relative gains of 8.00% on Instruction Induction and 115% on BIG-Bench, a 10.9% average improvement in human ratings of GPT-4 answers, and higher truthfulness on TruthfulQA. Averaged over all 11 phrases rather than the best one, the gains in Table 1 are much smaller and negative for some models.
Why it matters to an FDE
Never pick a prompt on the test set and report its score. Choosing the best of 11 phrases per task, using the same items you then score, produces a gain even when no phrase does anything; our simulation shows 6% to 13%. Customer prompt tuning falls into the same trap whenever one evaluation set is used both to choose and to report.
Split prompt selection from evaluation. Select prompt wording on a development set and report on a separate held-out set, or report the average over candidate wordings. The average-of-11 column in this paper is the honest estimate of what adding a random emotional phrase does.
Read relative gains next to their base. The 115% headline includes a 213% gain from a BIG-Bench score of 2.39 to 7.47 on a scale where 100 is a human expert: 5 points of absolute change.
Expect small effects from wording tricks. Averaged over phrases, gains on Instruction Induction were 0.2% to 4.3% relative, and three of six models scored lower with an average emotional phrase in the zero-shot setting. Effort spent on task instructions, examples and output structure usually pays more.
Know that phrasing can matter for some models. The paper is not evidence that wording never matters. Prompt sensitivity is real; the point is to measure it properly before adopting a rule.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The stimuli, what avg and max mean, the human study, TruthfulQA, attention, two misreadings
Free account - 04How it works
The stimuli, models and settings, how avg and max are computed, the human study and TruthfulQA
Premium - 05The numbers that matter
Table 1 averages per setting and model, the headline arithmetic, human study and TruthfulQA
Premium - 06Try it yourself
Rebuild the headline numbers, simulate picking the best of 11, and see gains on small bases
Premium - 07Where it breaks
Selection on the test set, gains on tiny bases, a small human study, older models and slips
Premium - 08Using it on a deployment
A prompt-selection protocol with a held-out set, averages and intervals, and what to test
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium