Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy

Does it matter whether you are polite to a language model? Earlier work by Yin and colleagues found that impolite prompts often hurt accuracy on models such as GPT-3.5 and Llama 2 70B, while overly polite wording did not reliably help. This short paper revisits the question with GPT-4o.

The authors used ChatGPT's Deep Research feature to write 50 multiple-choice questions of moderate to high difficulty in mathematics, science and history, each with four options, and to produce the answer key. Each question got five versions: very polite ("Would you be so kind as to solve the following question?"), polite ("Please answer the following question:"), neutral (no prefix), rude ("I doubt you can even solve this.") and very rude (prefixes such as "Hey gofer, figure this out."). That makes 250 prompts. Every prompt starts with an instruction to forget the session so far and answer with only the letter.

They sent each tone's 50 prompts to GPT-4o 10 times and recorded accuracy per run. Means rose steadily from very polite (80.8%) to polite (81.4%), neutral (82.2%), rude (82.8%) and very rude (84.8%). Paired t-tests over the 10 runs found 8 of 10 tone pairs significant at 5%, two with reported p-values of 0.0. The authors call the result surprising and preliminary, note that it differs from Yin et al., and suggest newer models may respond differently to tone. They list limits: one model, 50 questions, multiple choice only, and a concern that encouraging rudeness in interfaces could be harmful.

Translate percentages into items. On 50 questions, every 2 points is one question. The whole effect here is two questions per run. Before anyone changes a prompt standard because of a result like this, convert it into items and ask whether those two items are the same two every time.

Test across items, not across reruns of the same items. Running the same 50 prompts 10 times measures sampling noise on those prompts, which is small when a model answers consistently. A paired t-test across runs will call almost any fixed difference significant. Whether a tone helps in general is a question about questions, and needs a test over questions with enough of them.

Separate wording from tone. Each tone level used a few specific prefixes. If one prefix happens to tip one question, that is a wording effect on that question, not a property of rudeness. Vary the wording within each tone and use many questions before concluding about tone.

Size the test before running it. In our simulation, detecting a real 4-point effect with a question-level test takes about 250 questions; with 50 it succeeds about 1 time in 9.

Keep the guidance practical. For a customer's prompts, write clear, neutral instructions. Neither this paper nor the one it revisits supports insulting the model as a technique, and the authors themselves warn against it in user-facing systems.