The VRAM budget equation

02 — Context Sizing & Model Quantisation

How to choose quantisation and context size so the model, its KV cache, and your working memory all fit on the card.

The VRAM budget equation

Every model you run costs three things in VRAM:

VRAM ≈ weights(quant) + KV_cache(ctx × parallel) + overhead (~0.5 GB)

• Weights scale with model size and quantisation (a 4B model is ~2.5 GB at Q4_K, ~3.4 GB at Q6_K, ~4 GB at Q8_0).
• KV cache scales with context length, number of layers, and per-slot context (see the parallel pitfall below). Quantising the KV cache (cache-type-k / cache-type-v, typically q8_0 for K and q4_0 for V) roughly halves its cost at negligible quality loss — use it by default.
• Overhead — CUDA context, buffers, flash-attention workspace. Budget ~0.5 GB.

The trick is not to fill the card to 100%: the OS, the desktop compositor (if present), and peak batch work all need headroom. A good target is ~70–85% of VRAM at worst case, and on an 8 GB card, be conservative.

Weights quantisation

Rule of thumb for consumer cards:

Don’t over-quantise small models: Q3/Q2 on a 4B model is a bigger quality hit than Q4 on a 27B, because you’re cutting a model that is already small. If a model fits comfortably, pay for the extra quant.

Context sizing — the part people get wrong

Context is where most setups break, because the KV cache is what silently eats the VRAM you budgeted for “later”. Worked examples from the local setup:

24 GB card, gemma4-12b (Q8-ish weights ~12 GB): 256k context (ctx-size = 262144) with q8_0/q4_0 KV cache is viable because the card has ~11 GB of headroom after weights. This is the “main agent” profile — long documents, long tool conversations, compaction of other agents’ histories.

24 GB card, qwen3.8-27b (Q4_K_M weights ~16 GB): Only ~7 GB left for KV, so context drops to ~185k (ctx-size = 185000). Note the asymmetry: a 256k context on the smaller model beats an 185k context on the bigger model for pure throughput and cost — pick the model for the task, not the other way around.

8 GB card, qwen3.5-4b (Q6_K weights ~3.4 GB) plus a resident embedding model, bge-m3 (~1.75 GB):

3.4 (weights) + 1.75 (resident model) + ~0.5 (overhead) = 5.65 GB
remaining for KV: ~2.3 GB  →  8192 ctx × 2 slots at q8_0/q4_0

That leaves ~1.5 GB headroom at worst case — comfortable. The same card with a 7B Q4 model at 128k context is a 500 model failed to load (weights ~4.5 GB + KV ~5 GB > 8 GB). Context size is not a knob you set once and forget — it is a per-model budget line.

The working-memory rule of thumb: a model’s useful context is far smaller than its configured ctx-size. After the system prompt and tool schemas, a 4k context has maybe 2k tokens of actual working memory — too little for multi-step agent work, which is why an otherwise-fine 9B at 4k lost to a 7B at 16k as a worker model. For agent/sub-agent work, target at least 8k context and keep the system prompt lean.

Resident vs swap-in (load-on-startup, --models-max)

A models.ini preset lets one server hold several models. Two behaviours to choose per model:

• Resident (load-on-startup = true): the model is loaded at boot and never freed. Instant first request. You pay the VRAM forever.
• Swap-in (load-on-startup = false, usually with --models-max 1): nothing is loaded until requested; switching models evicts the current one (several seconds of load latency).

A pattern that works well on a small card: resident for the models that must be instant (a small embedding model, a small always-available chat worker), swap-in for the large models (the primary server keeps its GPU empty at boot and loads whichever of its two models is requested). If VRAM pressure appears, the fix ladder is: drop ctx-size, set load-on-startup = false on a profile, or re-enable idle sleep (sleep-idle-seconds).

Pitfalls learned the hard way

These were all verified failures on the local setup — expect the same ones:

• parallel silently divides your context. This llama.cpp build splits ctx-size across parallel slots unless kv-unified = true. With kv-unified = false (an explicit setting here), a “4096 ctx, 4 slots” profile actually gave each slot 1024 — and the client failed with request (1404 tokens) exceeds the available context size (1024). Set kv-unified = true when you want every slot to get the full context, and verify live after any preset change:

  curl -s 'http://127.0.0.1:PORT/slots?model=NAME' | python3 -m json.tool | grep n_ctx
  

• Embedding inputs are capped by the physical batch. ubatch-size is the maximum single embedding input in tokens; the 512 default rejected an ~860-token chunk with input (860 tokens) is too large to process. increase the physical batch size. Raise ubatch-size on embedding profiles if your memory chunks get longer.

• Reasoning models burn their token budget thinking. A Qwen-3.5-class 4B is a reasoning model by default: with thinking enabled it spent its entire max_tokens budget on reasoning_content and returned an empty answer (finish_reason = length). Disable globally with chat-template-kwargs = {"enable_thinking": false} in the [*] section and re-enable per-request where you actually want it (open-ended debugging, e.g.).

• Don’t set chat-template to a short name. That preset key takes a full Jinja template string; chat-template = chatml is invalid. Leave it unset and the template is taken from the GGUF metadata. If a model genuinely needs a specific template (e.g. a fine-tuned chat template), set jinja = true and chat-template-file = /path/to/template.jinja — that’s what qwen3.8-27b does with the froggeric template.

• Preset section names with digit-size tokens get mangled. INI section [qwen2.5:0.5B] registered as model ID qwen2.5:5B; [qwen2.5:7B-instruct] as qwen2.5:INSTRUCT. Name sections plainly (e.g. qwen3.5-4b) and check /v1/models after deploying.

A preset in practice

The two-tier split shows up directly in the presets — minimal, and every line earns its place:

# models-primary.ini — 24 GB card, swap-in (GPU empty at boot)
[*]                      # globals: inherited by every profile
sleep-idle-seconds = 3600   # free the GPU when idle
load-on-startup = false     # swap-in by default
flash-attn = true
batch-size = 1024
ubatch-size = 256
threads = 6
gpu-layers = 99             # all layers on GPU

[gemma4-12b]               # main agent: long documents, compaction
model = /usr/local/llama/models/Gemma4-12B-Q4_K_M.gguf
ctx-size = 262144
context-shift = true        # shift old tokens out when the context overflows
keep = 1024                 # ...but always keep this prefix (system prompt etc.)
cache-type-k = q8_0
cache-type-v = q4_0

[qwen3.8-27b]              # bigger brain, shorter context
model = /usr/local/llama/models/Qwen3.8-27B-UD-Q4_K_M.gguf
ctx-size = 185000
context-shift = true
keep = 1024
cache-type-k = q8_0
cache-type-v = q4_0
jinja = true
chat-template-file = /usr/local/llama/templates/froggeric_chat_template.jinja

# models-secondary.ini — 8 GB card, both models resident
[*]
flash-attn = true
threads = 6
gpu-layers = 99
chat-template-kwargs = {"enable_thinking": false}   # 4B is a reasoning model (see pitfalls)

[bge-m3]                   # resident embedding model
model = /usr/local/llama/models/BGE-M3.gguf
load-on-startup = true
pooling = cls               # BGE-M3 pools on the CLS token — required
ctx-size = 2048
cache-type-k = f16          # embeddings don't need a quantised KV cache
cache-type-v = f16
batch-size = 1024
ubatch-size = 1024          # physical batch = max single embedding input

[qwen3.5-4b]               # resident chat/worker model
model = /usr/local/llama/models/Qwen3.5-4B-Q6_K.gguf
load-on-startup = true
parallel = 2
ctx-size = 8192
kv-unified = true           # each slot gets the FULL ctx (see pitfalls)
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 512

Note that cache-type-k/cache-type-v are set per profile, not globally: the chat models get quantised q8_0/q4_0 caches, but the embedding model keeps f16 — a 2048-token KV cache is tiny anyway, and there’s no point quantising a model that never generates tokens.

Related

• Primary vs Secondary — which model plays which role, and why
• Sub-agents — context sizing for agent work specifically

Source Disclaimer

• AI Generated
• Human Generated
• AI Edited
• Human Edited