Budget weights, resident cache and runtime memory in consistent units

TL;DR: Estimate weight payload, cache for the admitted token population and runtime allocations separately, using bytes consistently. A model that loads is not yet a model that can serve the required workload; memory fit still needs per-device and peak-load verification.

Where you are. The customer has fixed hardware or a pending purchase. Before recommending a deployment, establish whether the intended architecture, context and concurrency can fit with operating margin.

The model, and the memory being budgeted

Counting parameters is useful but incomplete. A 70-billion-parameter payload uses 140 billion bytes at 16 bits, 70 billion at eight bits and 35 billion at four bits. Those are ideal packed weight sizes, and scales, zero points, higher-precision tensors, padding and runtime layout can all change the actual allocation.

Use GB for 10⁹ bytes and GiB for 2³⁰ bytes. A 35 GB payload is about 32.60 GiB, and a budget of 80 GiB holds about 85.90 GB, so it is not an 80 GB budget. Record the usable memory reported for the intended environment, including what other processes reserve.

Two devices' memory cannot simply be added as if it were one addressable pool. The runtime needs a supported partitioning strategy, enough capacity on every device and a suitable interconnect; some tensors or cache structures may be replicated, and multi-device deployment adds communication and workspace requirements of its own.

Cache size from the actual attention layout

For a standard full-attention decoder storing keys and values for every resident token, use:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per KV element
KV bytes for the batch = KV bytes per token × sum(resident tokens across sequences)

The first factor is keys plus values. Use the number of KV heads, not automatically the query-head count, because grouped-query attention shares keys and values among query heads; weight and cache precision are separate settings.

For the synthetic architecture used here, 80 layers, eight KV heads, head dimension 128 and two-byte cache elements give 327,680 bytes per token = 320 KiB. At 8,192 resident tokens, one sequence uses 2.5 GiB. The multiplication is worth doing once: 2 × 80 × 8 × 128 × 2 is 327,680 bytes per token, and 327,680 × 8,192 is 2,684,354,560 bytes, which is exactly 2.5 × 2³⁰. At 8,000 tokens it uses about 2.441 GiB instead, which is why you state the token count rather than relying on an ambiguous “8k.”

Resident tokens include the retained prompt and generated history, not only the new input, so admission planning may need to reserve space for future output or enforce a growing token budget. Multiplying every sequence by the same cap is a conservative full-reservation model when histories vary; allocating pages on demand changes occupancy but does not remove a capacity limit.

The inference chapter of How To Scale Your Model derives cache storage and its interaction with serving. Its calculations, like the worksheet below, rest on architectural assumptions; sliding-window attention, latent/compressed cache designs, hybrid layers, prefix sharing and offload need different accounting. Inspect the deployed configuration instead of applying this formula to every model name.

The worksheet, and its approximation

This standalone Python 3 example uses 80 GiB of usable memory, 70 billion parameters and an illustrative overhead of 12% of weight-plus-cache bytes. The percentage is a sensitivity assumption, not a measured reserve or a universal runtime rule, and it excludes quantization-format metadata from the weight payload.

import math

def memory_budget(params_b=70, weight_bits=4, layers=80, kv_heads=8,
                  head_dim=128, usable_gib=80, concurrency=16,
                  resident_tokens=8192, kv_bits=16, overhead=0.12):
    for name, value in {"layers": layers, "kv_heads": kv_heads,
                        "head_dim": head_dim, "resident_tokens": resident_tokens}.items():
        if type(value) is not int or value <= 0:
            raise ValueError(f"{name} must be a positive integer")
    if type(concurrency) is not int or concurrency < 0:
        raise ValueError("concurrency must be a nonnegative integer")
    for value in (params_b, usable_gib):
        if type(value) not in (int, float) or not math.isfinite(value) or value <= 0:
            raise ValueError("model size and usable memory must be positive finite numbers")
    if type(overhead) not in (int, float) or not math.isfinite(overhead) or overhead < 0:
        raise ValueError("overhead must be a nonnegative finite fraction")
    if weight_bits not in (4, 8, 16) or kv_bits not in (8, 16):
        raise ValueError("unsupported precision in this worksheet")
    gib = 2**30
    weights = params_b * 10**9 * weight_bits / 8
    kv_per_token = 2 * layers * kv_heads * head_dim * kv_bits / 8
    kv_per_sequence = kv_per_token * resident_tokens
    subtotal = weights + concurrency * kv_per_sequence
    total = subtotal * (1 + overhead)
    capacity = usable_gib * gib
    max_sequences = max(0, math.floor((capacity / (1 + overhead) - weights) / kv_per_sequence))
    return {"weights_gib": round(weights / gib, 2),
            "kv_per_sequence_gib": round(kv_per_sequence / gib, 3),
            "total_gib": round(total / gib, 2), "fits_estimate": total <= capacity,
            "max_sequences_by_memory": max_sequences}

for bits in (16, 8, 4):
    print(bits, memory_budget(weight_bits=bits))
print("four-bit, 15 sequences:", memory_budget(concurrency=15))

At the requested 16 sequences, none of the three precisions fits this 80 GiB estimate; the ideal weights occupy about 130.39, 65.19 and 32.60 GiB respectively. Cache is 40 GiB in all three cases because its precision and resident-token population did not change. The memory-only concurrency limits are zero, two and fifteen.

At four-bit weights and 15 sequences, the estimated total is about 78.51 GiB, and calling that “deployable” would go beyond the worksheet, because actual packed allocations, peak prefill workspace, cache bookkeeping and safety margin still need measurement. If measured usable memory is only 80 billion bytes, enter 80e9 / 2**30; the four-bit concurrency estimate falls to 13.

The interactive worksheet uses the same simplified model and explicit GiB units:

Its architecture presets are illustrative parameter sets, not complete configurations for every model of that size, and the slider's 8,192 tokens must include whatever retained input and output the capacity plan assumes. The memory limit is not a user-capacity or latency guarantee.

Overhead, measured instead of assumed as a percentage

Inspect memory after loading, during the longest supported prefill (the pass that reads the whole prompt before the first output token, which is where activation memory peaks), while cache grows to the admitted limit and during the supported concurrency mix. Include runtime workspaces, graph capture, allocator reservation, fragmentation, adapters and any co-resident service. Some allocations are fixed, some scale with tokens or batch shape, and some peak during transitions.

Do not subtract “allocated” from “reserved” without understanding the runtime's memory reporting, and keep units and the measurement boundary in the capacity note. In a sharded deployment, repeat the accounting per device; aggregate free memory can hide an overloaded rank.

Precision as a tested deployment choice

No precision guarantees a particular quality loss: a quantized model can match the reference on one test and fail a different segment, since the risk depends on the model, method, calibration and task. Run paired customer-relevant evaluations, including difficult and sensitive cases, before trading precision for concurrency.

The small rounding grid illustrates discretization without simulating a full quantization method or measuring task quality. Cache quantization is another independent choice, with its own supported formats, metadata, numerical behavior and tests. Changing weights to four bits does not imply cache changes to four bits.

An overload policy chosen from the workflow

Measure peak arrivals, admitted sequences and retained tokens instead of equating license count with concurrency. A conversation stored in a database does not necessarily occupy accelerator cache while inactive, and one long active request can consume much more cache than several short ones.

When the budget is insufficient, consider shorter retained context with evaluated retrieval, token-aware admission, bounded queues, an evaluated lower-precision or smaller model, and additional capacity. Choose among them on quality, latency, hardware and delivery constraints, bearing in mind that hardware can be the right early choice when a required workload cannot be reduced safely.

Queueing protects admission only when it is bounded and has a deadline, since an unbounded queue cannot give predictable latency. Define rejection or degradation behavior at the ceiling, including whether an output cap ends a response as incomplete. Capacity controls must not silently change the meaning of a user-facing answer.

The spine above is the worksheet as an order of operations, with the two places the arithmetic usually goes wrong marked: the unit you chose and the head count you multiplied by.

Do this before moving on

Run the worksheet and reproduce zero, two and fifteen as the memory-only concurrency limits. Then change resident tokens from 8,192 to 32,768 at four-bit weights; the per-sequence cache becomes 10 GiB and the limit falls to three. Change only KV precision to eight bits and explain why weight bytes stay fixed.

Compare 80 GiB with 80 GB and record the difference, and reject zero resident tokens, negative concurrency and non-finite overhead. Inspect the interactive controls and confirm that the four-bit 15-sequence estimate fits while 16 does not under the given assumptions.

For a real model, record its attention layout, actual weight format, cache dtype, per-device usable memory and input/output policy. Replace the percentage with measured allocations and a justified reserve, and submit a memory note that distinguishes worksheet estimates from verified serving capacity.

Go deeper

• GPU memory and VRAM develops the allocation and device-memory behavior behind the worksheet.
• KV cache explains retained attention state and the assumptions behind its cost.
• Quantization covers formats and evaluation beyond a scalar rounding illustration.
• Serving a 70B model on one 80GB card examines the constrained deployment decision with runtime and workload context.
• The key-value cache memory math extends the arithmetic to mitigation and architectural choices.
• Paged attention explains allocation efficiency without treating it as unlimited cache capacity.

Key takeaways

• Keep GB, GiB and exact token counts explicit throughout the estimate.
• Cache cost follows the actual attention architecture and resident-token population.
• Weight precision, cache precision and quality are separate decisions.
• A memory-only fit requires runtime-peak and per-device validation before rollout.
• Bound admission and queueing according to the workflow's latency and failure policy.