S-LoRA: Serving Thousands of Concurrent LoRA Adapters

LoRA fine-tunes a model by learning two small matrices per weight, A and B, so the update to W is the product AB with a low rank r. A company that fine-tunes one base model for each customer or task ends up with hundreds or thousands of adapters. The LoRA paper's advice for serving was to merge AB into W, which removes the extra compute, and to add and subtract adapters to switch between them. With many adapters live at once, merging means either a full model copy per adapter or a GPU that serves one adapter at a time.

S-LoRA keeps W shared and computes x·A·B per request alongside the batched x·W. Since x·W dominates the cost and can be batched across all requests regardless of adapter, the small extra work pays for itself. Three pieces make it scale:

Host memory for adapters. All adapters live in CPU memory; only the ones used by the running batch are copied to the GPU, predicted from the waiting queue and prefetched during the current step.

Unified paging. Adapter weights (rank × hidden size) and KV cache (tokens × hidden size) share a dimension, so one memory pool with pages of one hidden-size vector stores both, extending vLLM's PagedAttention. Adapters of different ranks and sequences of different lengths come and go without fragmenting memory.

Kernels and tensor parallelism. Custom kernels (MBGMM for prefill in Triton, MBGMV for decode adapted from Punica) gather each request's adapter from non-contiguous pages and handle mixed ranks without padding. A tensor-parallel partition aligned with Megatron-LM's adds LoRA communication of 5(N-1)Br/N to the base model's 2(N-1)Bh/N, small because r is much less than h.

On an A100 80 GB with Llama-7B adapters, S-LoRA served 2,000 adapters at 95% of its 5-adapter throughput, about 4x vLLM running merged copies and about 30x Hugging Face PEFT.

Per-customer or per-task fine-tuning is a common ask: a model that writes in each client's style, follows each team's schema, or knows each product line. The cost question is not training, which LoRA makes cheap, but serving. If every adapter needs its own GPU or its own model copy, the economics fail at a few dozen customers.

Multi-adapter serving changes the pricing. With adapters batched on a shared base model, an extra customer adds an adapter in host memory (tens of MB), not a GPU. That makes a per-customer model a product option rather than a bespoke deployment.

Pick a serving stack that batches across adapters. "LoRA support" in a server can mean merging one adapter at a time. The question to ask is whether requests for different adapters run in the same batch, and how adapter memory shares the GPU with the KV cache.

Popularity skew helps. Throughput depends on how many distinct adapters are in each batch, not how many exist. A long tail of rarely used adapters costs host memory and occasional fetches, not GPU time.