Why two servers, not one

03 — Primary vs Secondary llama-server

The two-server pattern: a big “brain” card and a small “worker” card, and how to decide what lives on each.

Why two servers, not one

The temptation is one llama-server that sees both GPUs and holds everything. The two-server split is better for three reasons:

1. Hardware isolation. CUDA_VISIBLE_DEVICES pins each server to one card, so a context-heavy request on the big card can never evict or starve the small one.
2. Independent roles. The two cards do genuinely different jobs, with different models, context sizes, and loading behaviour — see below.
3. Independent lifecycle. Each is its own systemd unit. You can restart, upgrade, or stop one without touching the other — and on a dual-use box, stop both to hand the GPUs to a desktop.

The division of labour

The split is by job, not by model count. The primary is where “big brain” work belongs — long documents, long tool conversations, spec-writing. The secondary handles high-volume, cheap, latency-sensitive work (short sub-agent turns) without ever touching the primary’s context memory, and the embedding model has its own home on the CPU helper (below).

The primary server (swap-in)

ExecStart=/usr/local/bin/llama-server \
    --models-preset /etc/llama/models-primary.ini \
    --models-max 1 \
    --no-mmproj \
    --metrics \
    --host :: \
    --port 8082

• --models-max 1 — only one model resident in GPU memory at a time. With load-on-startup = false in the preset, the GPU is empty at boot and the requested model is loaded on first use. This is deliberate: on a 24 GB card you can’t hold a 12B and a 27B and their big KV caches simultaneously, so the server swaps between them. The cost is a few seconds of load latency on a model switch.
• --no-mmproj — text-only serving; don’t load the multimodal projection file you’re not using.
• --metrics — Prometheus endpoint at /metrics for monitoring.

The secondary server (resident + embeddings)

ExecStart=/usr/local/bin/llama-server \
    --models-preset /etc/llama/models-secondary.ini \
    --no-mmproj \
    --embeddings \
    --metrics \
    --host :: \
    --port 8083

• --embeddings is a command-line flag, not a preset key. The /v1/embeddings route is only exposed if the server is started with it — pooling set in the preset alone is not enough; without the flag the endpoint returns 501 for every model. This is a classic “it works in one place but not the other” trap: the embedding model (bge-m3) must be on a server that was started with --embeddings.
• No --models-max — the small card’s resident model loads eagerly (load-on-startup = true), so the first chat request never pays a load penalty.
• Restart=always — a worker you always want up.

The embedding endpoint

A dedicated small embedding model (bge-m3, ~567M params, 1024-dim vectors, CLS pooling) — now hosted on a CPU-only helper server in the current setup (below) — is the clean way to serve memory/RAG. Two things must both be true:

1. The server was started with --embeddings.
2. The model profile has pooling = cls (BGE-M3 pools on the CLS token).

The memory service then calls http://host.docker.internal:8084/v1/embeddings with model: "bge-m3". Because it’s a small, resident, fast model, embeddings are essentially free and never compete with the primary’s big context.

The embedding escape hatch: a dedicated CPU-only server

When the small card gets crowded, move the resident embedding model to a third CPU-only llama-server (own systemd unit, own preset file, gpu-layers = 0, port 8084). Benefits: frees ~1.75 GB GPU VRAM, embeddings get an independent lifecycle (restart/dual-use desktop switching never touches them), and they no longer compete with the chat worker. Note the --embeddings flag remains on secondary as a cheap option for boxes without a third server.

Choosing the split for your own box

The general principle: put the model that must be fast and cheap on the small card; put the model that must be smart and deep on the big card. Concretely:

• If you have two cards, one small card is ideal for a small always-resident chat worker (sub-agent, quick utility, the thing Open WebUI hits by default), with the embedding model on it while there’s spare VRAM — and on a dedicated CPU-only helper once the card gets tight (the pattern the current reference deployment settled on).
• The big card is for the primary agent with the largest context you can afford — and it’s fine to keep it swap-in so it idles empty.
• If you only have one card, collapse to a single server and use --models-max / load-on-startup to decide what’s resident; you lose hardware isolation but keep everything else.

Related

• Context & Quantisation — the VRAM math behind the choices above
• Sub-agents — the spec→code pattern that motivates the worker card
• 01 — Architecture

Source Disclaimer

• AI Generated
• Human Generated
• AI Edited
• Human Edited