4-Bit Quantization Doesn't Mean 4x Less VRAM: What Eats the Rest

The maths seemed airtight. A 7B model in 4-bit is about 4GB of weights. You have a 12GB GPU. Three times the headroom. You start the server and it dies loading, or it loads and then falls over the first time two requests arrive at once:

Running out of memory loading 7B AWQ quantized models with 12GB vram
torch.OutOfMemoryError: CUDA out of memory.

Quantization shrinks the weights. It does not shrink everything else. The weights are only one of four things competing for your VRAM, and for a serving workload they are frequently not even the largest. Sizing a GPU off the quantized file size is the single most common way people end up with a model that "should fit" and does not.

Where the other 8GB goes

Think of GPU memory during inference as four buckets, filled in this order:

1. Quantized weights. The one number people quote. A 4-bit 7B model is roughly 4 to 5GB once you include the small unquantized pieces (embeddings, some norms, the scales and zero-points the quantization format keeps).
2. KV cache. Every token in every active sequence stores a key and value vector for every layer. This grows linearly with sequence length and with the number of concurrent sequences. At a long context and any real concurrency it can dwarf the weights. This is the bucket people forget entirely.
3. Activations. The transient tensors computed during each forward pass. Scales with batch size and hidden dimension.
4. Framework and CUDA overhead. The CUDA context itself, cuBLAS/cuDNN workspaces, memory fragmentation, and the serving runtime's own buffers. On a single GPU this is commonly around 1GB before you have loaded a single weight.

Quantization only attacks bucket one. Buckets two through four are computed in the compute dtype (typically fp16/bf16) regardless of how the weights are stored, so a 4-bit model and a 16-bit model with the same context and concurrency carry a similar KV cache and activation cost. That is why "4-bit" does not translate to "one quarter of the VRAM."

A worked sizing example

Take a 7B model, 4-bit AWQ, on a 12GB card, served with vLLM. Rough budget:

Bucket	Approx VRAM	Scales with
CUDA context + framework	~1GB	fixed per GPU
Quantized weights (4-bit 7B)	~4.5GB	model size and quant format
Activations	~0.5 to 1GB	batch size, hidden dim
KV cache	whatever is left	max_model_len x concurrent seqs

That leaves on the order of 5 to 6GB for the KV cache. A single 7B-class model in fp16 KV can spend well over 100MB per thousand tokens of context per sequence. Ask for a long max_model_len and several concurrent requests and you blow through 5GB fast, which is exactly the OOM people hit. vLLM pre-allocates the KV cache region up front based on --gpu-memory-utilization, so if the remaining budget cannot hold even one sequence at your requested length, it refuses to start rather than failing later.

Step 1: Cap the context length to what you actually use

KV cache is linear in max_model_len. If your prompts are 2k tokens, do not reserve for 32k. The single most effective lever:

vllm serve TheModel-AWQ \
  --quantization awq \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

Dropping max-model-len from 32768 to 4096 cuts the per-sequence KV cache reservation by 8x. According to the vLLM memory conservation docs, reducing max-model-len is the primary fix when the KV cache does not fit.

Step 2: Set gpu-memory-utilization deliberately

--gpu-memory-utilization tells vLLM what fraction of the card it may claim for weights plus KV cache. Raising it toward 0.95 gives the KV cache more room; setting it too high starves the CUDA context and activations and can itself OOM. On a shared or display-attached GPU, leave more headroom. Do not treat it as a magic dial; it only redistributes the same finite card.

Step 3: Bound concurrency

Total KV cache is per-sequence cost times the number of sequences in flight. If you need a longer context, you must accept fewer simultaneous requests, and vice versa. Cap it explicitly rather than discovering the ceiling in production:

vllm serve TheModel-AWQ --quantization awq \
  --max-model-len 8192 --max-num-seqs 8

Verification: test-load at your real target, not the file size

The only trustworthy check is to start the server at the exact max-model-len and concurrency you intend to run, then watch actual usage:

nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 2

Then fire concurrent requests at your intended --max-num-seqs and confirm memory settles below total with headroom. If it survives a clean load but OOMs under load, your KV cache budget is too tight for the concurrency you asked for. Weight file size told you none of this; the live load test tells you all of it.

What people get wrong

Sizing the GPU off the weight file. "4-bit 7B is 4GB, 12GB is plenty" ignores three of the four buckets. The weight file is a lower bound on VRAM, never an estimate of it. This is the same trap, one layer up, as assuming quantization gives proportional savings; it reduces the weight footprint and leaves the rest untouched.

Cranking --gpu-memory-utilization to 1.0 to "use the whole card." There is no such thing; the CUDA context and activations live outside vLLM's pool and you will OOM the moment real activations spike. Leave headroom.

Testing with one request and shipping. A single request barely touches the KV cache. The OOM appears at concurrency. Load-test at your real --max-num-seqs before you commit to a GPU tier.

When it is still broken

1. The server refuses to start even at a short context. Weights plus 1GB overhead may already be near your card. Drop to a smaller model, a more aggressive quant, or a bigger GPU; there is no context length that will save you if the weights alone leave no room for a single sequence.
2. It loads then OOMs minutes in. That is concurrency, or growing sequences reaching their length limit. Lower --max-num-seqs or --max-model-len.
3. You need long context and high concurrency on a small card. Those two multiply, and a 12GB card cannot serve both. Either shard across GPUs with tensor parallelism or accept the tradeoff. The economics of picking a tier are the whole subject of my real cost math for self-hosting open-weight models, which is worth reading before you buy or rent.
4. Fragmentation, not total, is the killer. If free memory looks sufficient but allocation still fails, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation before assuming you are truly out.