quantization

Adapter math, the VRAM arithmetic that makes 4-bit a 4x unlock, and the honest decision rule for on-prem deployments. The deep-internals question Mistral and Cohere loops use to find real practitioners.

Two compression levers with very different bills. The arithmetic that decides between them, when to stack both, and the eval discipline that keeps a 'cheaper' model from quietly tanking your hardest 10% of queries.

Bits buy bandwidth, bandwidth buys tokens, until quality quietly craters on code and math. A decision framework with the numbers, what GPTQ and AWQ actually do differently, and the eval discipline that keeps you out of trouble.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

A 7B VLM at 25 FPS on an embedded GPU sounds impossible until you stop processing every frame fully. The answer is token pruning, frame-to-frame KV reuse, and a three-stage pipeline that hides latency.