DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

By 2023, serious LLM applications were pipelines: rewrite the question, retrieve, reason, check, answer. Each stage ran on a prompt someone had written and rewritten by hand until it worked. The authors call these "lengthy strings discovered via trial and error". Swap the model, add a stage or change the data, and the strings had to be rediscovered.

DSPy, from Stanford, proposes treating the pipeline like a program and the prompts like compiled output. You declare what each language model call should do with a signature, such as "context, question -> search_query". You compose those calls in ordinary Python modules, the way you would compose layers in a neural network. Then you give a compiler a small training set and a metric. The compiler runs your program on the training inputs, keeps the runs that score well, and uses their intermediate steps as few-shot demonstrations for every module. It can also search over which demonstrations to use, ensemble programs, or fine-tune a small model to imitate a large one.

In two case studies, math word problems and multi-hop question answering, programs of a few lines compiled in minutes to tens of minutes. They beat standard few-shot prompting by large margins and matched or beat demonstrations written by experts. Compiling also made a 13-billion-parameter open model competitive with GPT-3.5, and produced a fine-tuned 770-million-parameter T5 model that came close.

The broader idea outlived the framework's specific API: a prompt is a parameter to optimize against an evaluation, not a piece of craft to protect.

On a customer project, prompts are where maintenance goes to die. Five engineers edit one template over three months, nobody remembers why a sentence is there, and a model upgrade silently breaks it. This paper gives a disciplined alternative even if you never use the library.

Separate the what from the how. A signature says what a step takes and produces. The prompt text, examples and reasoning style are implementation details that can be regenerated. Keeping the two apart is what lets you switch models without rewriting everything by hand.

The metric becomes the specification. A compiler optimizes exactly what you measure. The paper shows it: on math problems, the simple "answer directly" program learned to reason inside its answer field, because the metric only extracted the final number. That was allowed, and it helped. In a business setting the same pressure can produce outputs that pass the metric and fail the user. Writing the metric carefully is now the main design task.

Hold out a test set. The compiler picks what scores best on a development set, which overfits it. GPT-3.5's best math program scored 80.3% on development examples and 72.9% on the untouched test set. If you optimize prompts against an eval, you need a second eval the optimizer never saw.