---
title: "02 — Context Sizing & Model Quantisation"
description: "## The VRAM budget equation"
section: ai-docs
raw: "02-context-and-quantisation.md"
source: ai-generated
tags: ai, llama.cpp, docker
last-updated: 2026-09-15
---

# 02 — Context Sizing & Model Quantisation

> How to choose quantisation and context size so the model, its KV cache, and your working memory all fit on the card.

## The VRAM budget equation

Every model you run costs three things in VRAM:

```text
VRAM ≈ weights(quant) + KV_cache(ctx × parallel) + overhead (~0.5 GB)
```

- **Weights** scale with model size and quantisation (a 4B model is ~2.5 GB at Q4_K, ~3.4 GB at Q6_K, ~4 GB at Q8_0).
- **KV cache** scales with context length, number of layers, and *per-slot* context (see the parallel pitfall below). Quantising the KV cache (`cache-type-k` / `cache-type-v`, typically `q8_0` for K and `q4_0` for V) roughly halves its cost at negligible quality loss — use it by default.
- **Overhead** — CUDA context, buffers, flash-attention workspace. Budget ~0.5 GB.

The trick is not to fill the card to 100%: the OS, the desktop compositor (if present), and peak batch work all need headroom. A good target is **~70–85% of VRAM at worst case**, and on an 8 GB card, be conservative.

## Weights quantisation

Rule of thumb for consumer cards:

| Card | Sweet spot | Notes |
| --- | --- | --- |
| 8 GB | 3–7B at Q5/Q6, 12–14B at Q4 | Q6_K is the sweet spot for ≤7B models — near-Q8 quality at ~30% more size than Q4 |
| 12–16 GB | 12–14B at Q6/Q8, 24B at Q4 | |
| 24 GB | 27B at Q4_K_M, 12B at Q8 | UD (uncensored/modified) Q4_K_M builds are the workhorse for 20–30B models |

Don't over-quantise small models: Q3/Q2 on a 4B model is a bigger quality hit than Q4 on a 27B, because you're cutting a model that is already small. If a model fits comfortably, pay for the extra quant.

## Context sizing — the part people get wrong

Context is where most setups break, because the KV cache is what silently eats the VRAM you budgeted for "later". Worked examples from the local setup:

**24 GB card, `gemma4-12b` (Q8-ish weights ~12 GB):**
256k context (`ctx-size = 262144`) with `q8_0`/`q4_0` KV cache is viable because the card has ~11 GB of headroom after weights. This is the "main agent" profile — long documents, long tool conversations, compaction of other agents' histories.

**24 GB card, `qwen3.8-27b` (Q4_K_M weights ~16 GB):**
Only ~7 GB left for KV, so context drops to ~185k (`ctx-size = 185000`). Note the asymmetry: a 256k context on the *smaller* model beats an 185k context on the *bigger* model for pure throughput and cost — pick the model for the task, not the other way around.

**8 GB card, `qwen3.5-4b` (Q6_K weights ~3.4 GB) plus a resident embedding model, `bge-m3` (~1.75 GB):**

```text
3.4 (weights) + 1.75 (resident model) + ~0.5 (overhead) = 5.65 GB
remaining for KV: ~2.3 GB  →  8192 ctx × 2 slots at q8_0/q4_0
```

That leaves ~1.5 GB headroom at worst case — comfortable. The same card with a 7B Q4 model at 128k context is a `500 model failed to load` (weights ~4.5 GB + KV ~5 GB > 8 GB). **Context size is not a knob you set once and forget — it is a per-model budget line.**

**The working-memory rule of thumb:** a model's *useful* context is far smaller than its configured `ctx-size`. After the system prompt and tool schemas, a 4k context has maybe 2k tokens of actual working memory — too little for multi-step agent work, which is why an otherwise-fine 9B at 4k lost to a 7B at 16k as a worker model. For agent/sub-agent work, target at least 8k context and keep the system prompt lean.

## Resident vs swap-in (`load-on-startup`, `--models-max`)

A `models.ini` preset lets one server hold several models. Two behaviours to choose per model:

- **Resident** (`load-on-startup = true`): the model is loaded at boot and never freed. Instant first request. You pay the VRAM forever.
- **Swap-in** (`load-on-startup = false`, usually with `--models-max 1`): nothing is loaded until requested; switching models evicts the current one (several seconds of load latency).

A pattern that works well on a small card: **resident for the models that must be instant** (a small embedding model, a small always-available chat worker), **swap-in for the large models** (the primary server keeps its GPU empty at boot and loads whichever of its two models is requested). If VRAM pressure appears, the fix ladder is: drop `ctx-size`, set `load-on-startup = false` on a profile, or re-enable idle sleep (`sleep-idle-seconds`).

## Pitfalls learned the hard way

These were all verified failures on the local setup — expect the same ones:

- **`parallel` silently divides your context.** This llama.cpp build splits `ctx-size` across `parallel` slots unless `kv-unified = true`. With `kv-unified = false` (an explicit setting here), a "4096 ctx, 4 slots" profile actually gave each slot 1024 — and the client failed with `request (1404 tokens) exceeds the available context size (1024)`. Set `kv-unified = true` when you want every slot to get the *full* context, and verify live after any preset change:

  ```bash
  curl -s 'http://127.0.0.1:PORT/slots?model=NAME' | python3 -m json.tool | grep n_ctx
  ```

- **Embedding inputs are capped by the physical batch.** `ubatch-size` is the maximum single embedding input in tokens; the 512 default rejected an ~860-token chunk with `input (860 tokens) is too large to process. increase the physical batch size`. Raise `ubatch-size` on embedding profiles if your memory chunks get longer.
- **Reasoning models burn their token budget thinking.** A Qwen-3.5-class 4B is a reasoning model by default: with thinking enabled it spent its entire `max_tokens` budget on `reasoning_content` and returned an *empty* answer (`finish_reason = length`). Disable globally with `chat-template-kwargs = {"enable_thinking": false}` in the `[*]` section and re-enable per-request where you actually want it (open-ended debugging, e.g.).
- **Don't set `chat-template` to a short name.** That preset key takes a full Jinja template string; `chat-template = chatml` is invalid. Leave it unset and the template is taken from the GGUF metadata. If a model genuinely needs a specific template (e.g. a fine-tuned chat template), set `jinja = true` and `chat-template-file = /path/to/template.jinja` — that's what `qwen3.8-27b` does with the froggeric template.
- **Preset section names with digit-size tokens get mangled.** INI section `[qwen2.5:0.5B]` registered as model ID `qwen2.5:5B`; `[qwen2.5:7B-instruct]` as `qwen2.5:INSTRUCT`. Name sections plainly (e.g. `qwen3.5-4b`) and check `/v1/models` after deploying.

## A preset in practice

The two-tier split shows up directly in the presets — minimal, and every line earns its place:

```ini
# models-primary.ini — 24 GB card, swap-in (GPU empty at boot)
[*]                      # globals: inherited by every profile
sleep-idle-seconds = 3600   # free the GPU when idle
load-on-startup = false     # swap-in by default
flash-attn = true
batch-size = 1024
ubatch-size = 256
threads = 6
gpu-layers = 99             # all layers on GPU

[gemma4-12b]               # main agent: long documents, compaction
model = /usr/local/llama/models/Gemma4-12B-Q4_K_M.gguf
ctx-size = 262144
context-shift = true        # shift old tokens out when the context overflows
keep = 1024                 # ...but always keep this prefix (system prompt etc.)
cache-type-k = q8_0
cache-type-v = q4_0

[qwen3.8-27b]              # bigger brain, shorter context
model = /usr/local/llama/models/Qwen3.8-27B-UD-Q4_K_M.gguf
ctx-size = 185000
context-shift = true
keep = 1024
cache-type-k = q8_0
cache-type-v = q4_0
jinja = true
chat-template-file = /usr/local/llama/templates/froggeric_chat_template.jinja

# models-secondary.ini — 8 GB card, both models resident
[*]
flash-attn = true
threads = 6
gpu-layers = 99
chat-template-kwargs = {"enable_thinking": false}   # 4B is a reasoning model (see pitfalls)

[bge-m3]                   # resident embedding model
model = /usr/local/llama/models/BGE-M3.gguf
load-on-startup = true
pooling = cls               # BGE-M3 pools on the CLS token — required
ctx-size = 2048
cache-type-k = f16          # embeddings don't need a quantised KV cache
cache-type-v = f16
batch-size = 1024
ubatch-size = 1024          # physical batch = max single embedding input

[qwen3.5-4b]               # resident chat/worker model
model = /usr/local/llama/models/Qwen3.5-4B-Q6_K.gguf
load-on-startup = true
parallel = 2
ctx-size = 8192
kv-unified = true           # each slot gets the FULL ctx (see pitfalls)
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 512
```

Note that `cache-type-k`/`cache-type-v` are set **per profile, not globally**: the chat models get quantised `q8_0`/`q4_0` caches, but the embedding model keeps `f16` — a 2048-token KV cache is tiny anyway, and there's no point quantising a model that never generates tokens.

## Related

- [Primary vs Secondary](/ai-docs/03-primary-vs-secondary/) — which model plays which role, and why
- [Sub-agents](/ai-docs/04-sub-agents/) — context sizing for agent work specifically

---

## Source Disclaimer

- [x] AI Generated
- [ ] Human Generated
- [x] AI Edited
- [x] Human Edited