Small Language Models are the Future of Agentic AI

Many calls inside an agent perform a narrow operation: classify a request, extract fields or choose among a small set of tools. This position paper argues that such calls often do not need the broad capabilities of a large general-purpose language model. It proposes specializing smaller models for recurring work and retaining larger models where their additional capability is useful.

The argument combines operational considerations with examples from existing model research. It is not a controlled benchmark proving that a small model can replace every agent invocation. Its definition of small is practical: a model that fits on a common consumer device and serves a user's agent requests at useful latency. That definition depends on the task and deployment conditions.

The authors outline a conversion process: collect appropriate usage data, curate it, identify task clusters, select candidate models, adapt them and iterate. The useful engineering question is which call class can be replaced without damaging the full workflow.

The economic claim needs local evidence. A cheap first attempt followed by an expensive retry can cost more than going directly to the stronger model. Worse, an incorrect answer that the fallback policy accepts may never receive that second chance. Evaluate both the routing decision and the result it delivers.

A customer's agent bill often reflects repeated operations rather than a single difficult request. Trace the workflow and group calls by purpose, volume, input length and error cost. That can reveal a high-volume extraction step worth testing with a smaller model even when the overall agent still needs a strong planner.

The boundary matters. An apparently simple tool-selection step may encode policy knowledge or ambiguous user intent. Replacing it based on average benchmark performance can send the entire workflow down the wrong branch. Evaluate the call in its actual context and inspect downstream consequences.

Use model selection for enterprise to establish the requirements, then compare alternatives at the same quality floor. Include deterministic code as a candidate when the operation has a stable rule. A model-size decision should follow evidence about the task, rather than becoming an architectural preference applied everywhere.