Self-Hosting Open-Weight Models: The VRAM and Cost Math Nobody Shows You

"We want to run our own model" is a sentence I hear weekly, usually followed by "for data privacy" and occasionally by "to save money". The privacy reason is often sound. The money reason almost never survives contact with a spreadsheet. This article is that spreadsheet: the actual formulas for VRAM, the throughput ceiling nobody mentions, and the break-even point against API pricing.

Formula one: how much VRAM the weights need

Weight memory is simply parameter count multiplied by bytes per parameter. The bytes depend on the numeric format:

Format	Bytes per parameter	7B model	70B model
FP16 / BF16	2.0	~14 GB	~140 GB
FP8 / INT8	1.0	~7 GB	~70 GB
Q5_K_M (GGUF)	~0.70	~4.9 GB	~49 GB
Q4_K_M (GGUF)	~0.55	~3.9 GB	~39 GB

The rule of thumb everyone quotes - "about half a gigabyte per billion parameters at 4-bit" - comes straight from that Q4_K_M row and it is close enough for planning. Q4_K_M is the default recommendation for good reason: roughly a 75 percent reduction against FP16 with quality loss that most applications cannot measure. Below 4-bit the degradation becomes visible, particularly on code and structured output.

Mixture-of-experts models change the shape of the problem, not the total. A 21B-parameter MoE that activates 3.6B per token still needs all 21B resident in memory. You get the speed of a small model and the memory bill of a large one.

Formula two: the KV cache, which is what actually kills you

Weights are the number people quote. KV cache is the number that makes deployments fall over at 3pm. Every token in every active sequence stores a key and a value vector at every layer:

def kv_cache_gb(layers, kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):
    # 2 = one key tensor plus one value tensor
    total_bytes = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem
    return total_bytes / (1024 ** 3)

def total_vram_gb(params_b, bytes_per_param, layers, kv_heads,
                  head_dim, seq_len, batch, framework_overhead_gb=1.5):
    weights = params_b * 1e9 * bytes_per_param / (1024 ** 3)
    kv = kv_cache_gb(layers, kv_heads, head_dim, seq_len, batch)
    return round(weights + kv + framework_overhead_gb, 1)

# A dense 8B model, 32 layers, 8 KV heads (grouped-query), head_dim 128
print(total_vram_gb(8, 0.55, 32, 8, 128, seq_len=8192,  batch=1))   # ~6.6 GB
print(total_vram_gb(8, 0.55, 32, 8, 128, seq_len=32768, batch=8))   # ~28.6 GB
print(total_vram_gb(8, 2.00, 32, 8, 128, seq_len=32768, batch=8))   # ~40.5 GB

Look at the second and third lines. Same model. The weights barely moved between 8K single-user and 32K eight-user, but the KV cache went from a rounding error to 12 GB. This is why a model that ran beautifully in your Ollama demo dies the moment five people use it with long documents. Grouped-query attention (few KV heads relative to query heads) is the reason modern models survive this at all; older multi-head architectures are several times worse.

Practical implications: quantise the KV cache to 8-bit if your engine supports it (halves that number), cap max_model_len at what you actually need rather than the model's advertised maximum, and set an explicit max_num_seqs so the scheduler queues rather than crashes.

Formula three: the throughput ceiling

Single-stream decoding is memory-bandwidth bound, not compute bound. Each generated token requires reading the whole active weight set from VRAM. So:

tokens per second ceiling is roughly memory bandwidth divided by bytes of active weights.

A card with about 1 TB/s of memory bandwidth running a 4-bit 8B model (roughly 4.5 GB active) tops out near 220 tokens per second for one request, and you will realistically see 60 to 70 percent of that. The same card running a 4-bit 70B model (roughly 39 GB) tops out near 25 tokens per second. No amount of prompt tuning changes this; it is arithmetic.

The escape hatch is batching. Continuous batching in vLLM or SGLang amortises the weight read across many concurrent sequences, so aggregate throughput can be ten to twenty times single-stream. That is the whole reason vLLM exists, and it is why your cost per token depends almost entirely on how busy your GPU is.

Choosing an engine

• Ollama - wraps llama.cpp, GGUF weights, one command to start, CPU offload when you run out of VRAM. Excellent for laptops, prototypes and single-user internal tools. Poor at concurrency.
• llama.cpp directly - same engine, more control, GBNF grammars for constrained output, runs acceptably on CPU-only boxes and Apple silicon. This is the option that works on a machine you already own.
• vLLM - paged attention, continuous batching, an OpenAI-compatible server, tensor parallelism across cards. This is what you deploy when more than one person is using it. Pair with AWQ or FP8 weights on CUDA hardware rather than GGUF.

# Single-user development box
ollama run qwen3:8b

# Production-shaped serving with an explicit memory budget
vllm serve Qwen/Qwen3-8B \
  --quantization awq \
  --max-model-len 16384 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --port 8000

# It speaks the OpenAI wire format, so most client libraries just work
curl http://localhost:8000/v1/chat/completions \
  -H "content-type: application/json" \
  -d '{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"hi"}]}'

Now the money

Cost per million output tokens, self-hosted, is:

def cost_per_mtok(hourly_cost_usd, tokens_per_second, utilisation=0.30):
    # utilisation = fraction of the wall-clock hour the GPU is actually generating
    effective_tps = tokens_per_second * utilisation
    tokens_per_hour = effective_tps * 3600
    return hourly_cost_usd / (tokens_per_hour / 1_000_000)

# Rented consumer card, small model, well batched, but idle most of the day
print(round(cost_per_mtok(0.45, 900, utilisation=0.10), 2))   # ~$1.39
print(round(cost_per_mtok(0.45, 900, utilisation=0.50), 2))   # ~$0.28
print(round(cost_per_mtok(0.45, 900, utilisation=0.95), 2))   # ~$0.15

That utilisation term is the entire argument. At 10 percent duty cycle a rented consumer GPU costs about the same per token as a small hosted model, with none of the quality and all of the operational burden. At 95 percent it is roughly ten times cheaper. GPUs bill by the wall clock; your users do not arrive uniformly.

Set that against hosted pricing. At the time of writing, Anthropic lists Claude Haiku 4.5 at 1 US dollar per million input tokens and 5 dollars per million output, and Claude Sonnet 5 at 3 and 15 dollars respectively - check the current pricing page before you build a business case on it, because these move. Rented GPU rates vary enormously by provider; consumer cards on marketplace providers have been available well under a dollar an hour, while H100-class capacity sits several times higher. Regional availability matters: if you need compute physically in Africa your options and prices are very different from us-east-1, and cross-border latency of 150 to 250 ms to European regions is a real product constraint for interactive features.

Owning the hardware

If you buy the card instead of renting it, the honest total cost of ownership has four lines:

• Capex. Landed cost, which for imported hardware means the invoice plus shipping plus import duty plus VAT plus clearing. Budget meaningfully above the sticker price you see online, and amortise over 24 to 36 months, not 60. These cards depreciate fast.
• Power. A 350 W card at full tilt is about 0.35 kWh per hour, plus cooling and host overhead. Multiply by your commercial tariff and your actual duty cycle. In Kenya, commercial power is expensive enough that a 24/7 idle GPU is a line item worth noticing.
• Availability. Grid interruptions, a UPS, and the fact that a single card is a single point of failure with no failover.
• Engineering time. Realistically the largest line. Somebody has to patch drivers, watch out-of-memory errors, upgrade the engine when a new model needs it, and be reachable at 22:00.

The decision that actually holds up

After doing this exercise repeatedly, the answer is almost always a split rather than a choice:

Self-host the small, high-volume, latency-sensitive models. Embeddings, rerankers, classifiers, PII scrubbers, intent routers. These are 100M to 8B parameter workloads that fit comfortably on one card, run constantly, and would cost real money per call at API prices. An open-weight multilingual embedding model such as BGE-M3 plus an open reranker will handle a serious RAG corpus on hardware you can buy once. This is where self-hosting genuinely wins, and it wins by a wide margin.

Use hosted APIs for the reasoning tier. The frontier gap on hard multi-step reasoning, long-context work and tool use is still real, the traffic is spiky, and you pay only for what you use. You also avoid owning the model-upgrade treadmill.

Self-host the reasoning tier only when a hard constraint forces it. Data residency written into a contract, a regulator that will not accept cross-border processing, an air-gapped deployment, or a genuinely steady high-volume workload where you can keep a GPU above 60 percent utilisation. Those are good reasons. "It feels cheaper" is not one until you have filled in the utilisation number.

A checklist before you commit

• Compute weights plus KV cache at your real maximum context and concurrency, not at batch 1.
• Add 1 to 2 GB of framework overhead, then leave 10 percent headroom.
• Check the licence. Apache 2.0 and MIT are clean for commercial use; several popular open-weight families carry additional restrictions you should read before shipping.
• Measure real tokens per second on your hardware with your prompt shape, not a benchmark from a blog post.
• Estimate duty cycle honestly from your existing traffic logs.
• Price the engineering hours. Then price them again.

Do the arithmetic first. It takes twenty minutes and it has saved several of my clients from buying a very expensive space heater.