The weights are the easy part of the memory budget

TL;DR: Memory is weights plus key-value cache plus overhead. The weights are fixed and easy; the cache grows with concurrency times context length and is what actually runs you out of memory. Compute all three before promising anything, because the failure arrives under load rather than at start-up.

Where you are. First lesson of the hardest course here. The customer has hardware, bought before you arrived, and a requirement that the model runs on it. Before any architecture discussion, you need to know whether that is possible, and this is arithmetic rather than opinion.

Why the obvious calculation is wrong

Somebody looks up a model with 70 billion parameters, notes that half precision is two bytes per parameter, and concludes it needs 140 gigabytes. Two accelerators with 80 gigabytes each, and there is headroom. The procurement request goes in.

Then it is deployed, serves a handful of test users beautifully, and falls over the first day the operations team actually uses it.

The weights are the part that does not change. What changes, continuously, is the key-value cache: the attention keys and values retained for every token in every in-flight request, so that generating token n+1 does not require recomputing the whole sequence. It scales with how many requests are in flight and how long each conversation is, both of which are properties of the workload rather than of the model, and neither of which appears in any model card.

The three terms

Weights. Parameters times bytes per parameter. A 70 billion parameter model is 140 GB at 16-bit, 70 GB at 8-bit, 35 GB at 4-bit. This is the number everybody computes.

Key-value cache. Per token, per layer, you retain a key and a value vector for each attention head that has its own key and value. So:

bytes per token = 2 (K and V)
                × layers
                × kv_heads          (grouped attention shares these; check the config)
                × head_dim
                × bytes per element (2 at 16-bit, and the cache is often kept at 16-bit
                                     even when the weights are quantised)

For a model with 80 layers, 8 key-value heads and a head dimension of 128, at 16-bit: 2 × 8 × 128 × 2 = 4,096 bytes per token per layer, times 80 layers, so roughly 320 KB per token.

That number is the one that matters. An 8,000-token conversation holds about 2.6 GB of cache. Sixteen of those in flight is 42 GB, which is more than the weights of the 4-bit model.

Overhead. The runtime itself, memory fragmentation, the activation working set during a forward pass. Reserve 10 to 15 per cent. Planning to the last gigabyte produces a system that runs until a slightly longer prompt arrives.

Do it in code, once

def fits(
    params_b: float,           # billions of parameters
    weight_bits: int,          # 16, 8, or 4
    layers: int,
    kv_heads: int,             # NOT total heads: grouped attention shares KV
    head_dim: int,
    card_gb: float,
    concurrency: int,
    context_tokens: int,
    kv_bits: int = 16,         # cache precision is usually independent of weights
    overhead: float = 0.12,
) -> dict:
    gb = 1024 ** 3
    weights = params_b * 1e9 * (weight_bits / 8)
    kv_per_token = 2 * layers * kv_heads * head_dim * (kv_bits / 8)
    kv_total = kv_per_token * context_tokens * concurrency
    subtotal = weights + kv_total
    total = subtotal * (1 + overhead)
    return {
        "weights_gb": round(weights / gb, 1),
        "kv_per_seq_gb": round(kv_per_token * context_tokens / gb, 2),
        "kv_total_gb": round(kv_total / gb, 1),
        "total_gb": round(total / gb, 1),
        "fits": total <= card_gb * gb,
        # The number the customer actually wants: how many users at once.
        "max_concurrency": int(
            (card_gb * gb / (1 + overhead) - weights) // (kv_per_token * context_tokens)
        ),
    }

max_concurrency is the output worth putting in front of a customer. "It fits" invites a follow-up; "it fits, and supports about sixteen simultaneous conversations at eight thousand tokens each, on this card" is an answer somebody can plan a rollout around.

The precision decision, stated honestly

What the precisions do to the numbers themselves, which is easier to feel than to read:

Two cautions that turn this table from a summary into a decision.

Quantisation loss is not uniform across tasks. It shows up first in the long tail: rare formats, unusual entity names, multi-step arithmetic. A general benchmark can look unchanged while the customer's hardest segment degrades noticeably, which is exactly the segment the exceptions team lives in. Run their evaluation set at each precision before choosing, not a public one.

The cache is usually not quantised with the weights. Teams quantise to 4-bit, expect a quarter of the memory, and find the cache unchanged because it is still 16-bit. Cache quantisation is a separate decision with its own quality cost, and on long-context workloads it is the more consequential one.

Two properties of the workload decide everything

Because cache scales with concurrency times context length, the two questions that actually determine the hardware are not about the model.

How many people use it at once, at peak? Not per day. Simultaneously, at the worst moment, which for an operations team is usually a predictable hour.

How long is a conversation? A single-turn extraction over a short document is a few hundred tokens. A multi-turn session over a long contract is tens of thousands, and the cache for one such session can exceed the memory of a small model entirely.

Get those two numbers in discovery. A customer who says "about forty users" is describing a licence count, not a concurrency, and the difference between forty users and forty simultaneous long conversations is the difference between one card and six.

What to do when it does not fit

In rough order of what to try first, because the cheapest options are also the least discussed.

Cap the context. A hard limit on input length, enforced and communicated, is the single most effective lever, because cache is linear in it. Most workflows have a much shorter genuine requirement than the maximum somebody asked for.

Bound concurrency and queue. Admitting eight requests and making the ninth wait produces predictable latency. Admitting all of them produces an out-of-memory error that takes down the in-flight ones too, which is strictly worse for everybody.

Quantise further, with evidence. Now the evaluation you ran per precision earns its keep.

Use a smaller model. Frequently the right answer and rarely the first suggestion. A smaller model that fits with room for concurrency often serves the workflow better than a larger one that thrashes, and the evaluation set will tell you whether the task actually needs the capacity.

Add hardware. Last, because it is slow, expensive and somebody else's budget cycle.

Do this before moving on

Take an open-weight model you might realistically deploy and find its real configuration: layer count, key-value head count, head dimension. Compute the cache cost per token, then per conversation at the context length your workflow needs, then the concurrency a single card supports at each precision. The number that surprises you will be the per-conversation cache, and it is the number nobody includes in a capacity estimate.

Go deeper

• GPU memory and VRAM is the underlying memory model in more depth than this arithmetic needs.
• KV cache explains why the cache exists and what it costs to give up.
• Quantization covers the formats and where their quality loss actually appears.
• Serving a 70B model on one 80GB card works this exact trade-off through end to end.
• The key-value cache memory maths is the arithmetic above with the mitigations enumerated properly.
• Paged attention is the allocation technique that changes the fragmentation term in this budget.

Key takeaways

• Memory is weights plus key-value cache plus overhead, and only the first is fixed.
• Cache scales with concurrency times context length, so the numbers that size the hardware are workload properties, not model properties; roughly 320 KB per token means one long conversation costs gigabytes.
• Quantisation loss lands on the hardest segment first, so evaluate at each precision on the customer's own set, and remember that quantising weights does not shrink the cache.
• When it does not fit: cap context, bound concurrency, quantise with evidence, use a smaller model, and only then buy hardware.