Prompt Engineering

TL;DR: Prompt engineering is making a model reliable by being explicit: a system prompt that sets role and rules, clear instructions, a few examples when the task is fuzzy, and a stated output format. Reach for it first because it costs nothing to iterate on. Move to RAG when the model needs facts it does not have, and to fine-tuning when you need to change its default behavior, not its knowledge.

The pieces of a prompt

Most production prompts have a few distinct layers, and confusing them is a common source of flaky behavior.

• System prompt. Sets the model's role, tone, and hard rules that should hold across every turn. "You are a support agent for Acme. Answer only from the provided context. If the answer is not there, say you do not know." This is the right place for constraints, not the user turn.
• User prompt. The actual request or question for this turn.
• Instructions. What to do, stated as commands rather than wishes. "Return a JSON object with keys summary and risk_level" beats "it would be nice to get a summary."
• Few-shot examples. When the task is hard to describe but easy to demonstrate (a specific extraction format, a particular tone), show two or three input-output pairs. The model imitates the pattern. Zero-shot (instructions only) is fine for clear tasks; reach for examples when zero-shot output drifts from what you want.
• Output format constraints. Specify the exact shape you expect. Showing a literal template of the JSON or the structure you want is far more reliable than describing it in prose.

A worked example

Suppose you want to classify support tickets by urgency. A vague prompt ("how urgent is this ticket?") gives you a paragraph of prose you then have to parse. A structured one removes the ambiguity:

System: You triage support tickets. Output ONLY valid JSON.

Instructions: Classify the ticket's urgency as "low", "medium", or "high".
Base it on customer impact and time sensitivity only.

Format:
{"urgency": "low|medium|high", "reason": "<one sentence>"}

Example:
Ticket: "Login page returns 500 for all users."
{"urgency": "high", "reason": "Total outage affecting every user."}

Ticket: "Can you change the font on my invoice?"

Notice the moves: a system prompt that pins the role and the output discipline, an explicit label set so the model cannot invent a fourth category, a literal format line, and one example that demonstrates the reasoning you want. Each one removes a way the output could go wrong.

When prompting is enough, and when it is not

The spine below is what to do when it is nearly enough, which is the follow-up an interviewer holds back: the prompt works in testing and breaks on real traffic.

Good prompting solves more than people expect. If the task is "rewrite this in a friendlier tone" or "extract these five fields," a clear prompt with an example is usually the whole answer, and adding RAG or fine-tuning would be wasted effort.

Matching the problem to the lever keeps you on the cheap end:

It stops being enough on two fronts. When the model needs information it was never trained on or that changes over time, your company's pricing, this week's inventory, no prompt can supply facts the model does not have; that is RAG's job. When you need to change the model's ingrained behavior or output style at a level that examples cannot reach consistently, or you have many examples and need lower latency than stuffing them in the prompt allows, that points to fine-tuning. The order to try them is prompting, then RAG, then fine-tuning, because that is also the order from cheapest to most expensive to build and maintain.

Why interviewers probe this

It reveals whether you reach for the simplest tool first. The weak signal is a candidate who proposes fine-tuning for a problem a three-line system prompt would solve. The follow-up they hold back is "your prompt works in testing but breaks on 5% of real inputs, what do you do?" The strong answer is concrete: collect the failures, look for the pattern, make the instruction more explicit, add an example covering that case, and constrain the output format, rather than immediately blaming the model or jumping to training.

Common misconceptions

• "Prompting is just typing a question." Production prompting is interface design: role, instructions, examples, and format, each chosen to remove a failure mode.
• "More examples always help." Examples cost tokens and can bias the model toward their specifics. Two or three good ones usually beat ten mediocre ones.
• "Politeness or magic phrases improve accuracy." Explicitness and structure improve accuracy. Begging or threatening the model is folklore.
• "If the prompt fails, fine-tune." Most prompt failures are fixed by clearer instructions, a format spec, or one targeted example, all far cheaper than training.

Key takeaways

• Separate the layers: system prompt for role and rules, user prompt for the request, plus explicit instructions and a stated output format.
• Use few-shot examples when a task is easier to show than to describe; keep them few and representative.
• Constrain the output format with a literal template, especially when something downstream must parse the result.
• Try prompting first, RAG when facts are missing or change, fine-tuning when behavior or format must change; that order is also cheapest to most expensive.