---
title: "03 — Primary vs Secondary `llama-server`"
description: "## Why two servers, not one"
section: ai-docs
raw: "03-primary-vs-secondary.md"
source: ai-generated
tags: ai, llama.cpp, docker
last-updated: 2026-09-15
---

# 03 — Primary vs Secondary `llama-server`

> The two-server pattern: a big "brain" card and a small "worker" card, and how to decide what lives on each.

## Why two servers, not one

The temptation is one `llama-server` that sees both GPUs and holds everything. The two-server split is better for three reasons:

1. **Hardware isolation.** `CUDA_VISIBLE_DEVICES` pins each server to one card, so a context-heavy request on the big card can never evict or starve the small one.
2. **Independent roles.** The two cards do genuinely different jobs, with different models, context sizes, and loading behaviour — see below.
3. **Independent lifecycle.** Each is its own systemd unit. You can restart, upgrade, or stop one without touching the other — and on a dual-use box, stop both to hand the GPUs to a desktop.

## The division of labour

| | **Primary** (big card) | **Secondary** (small card) |
| --- | --- | --- |
| Card | RTX 3090 (24 GB), `CUDA_VISIBLE_DEVICES=0` | RTX 3070 (8 GB), `CUDA_VISIBLE_DEVICES=1` |
| Port | 8082 | 8083 |
| Role | **Main agent** — the model the user primarily chats with | **Worker** — sub-agent chat + the **embedding** model |
| Models | e.g. a 12B and a 27B (big weights, big context) | a small chat model + `bge-m3` (embeddings) |
| Context | ~185k–256k | 8k (chat) / 2k (embeddings) |
| Loading | **Swap-in** (`--models-max 1`, `load-on-startup = false`) | **Resident** (`load-on-startup = true`) |
| Restart | `on-failure` | `always` |

The split is **by job, not by model count.** The primary is where "big brain" work belongs — long documents, long tool conversations, spec-writing. The secondary handles high-volume, cheap, latency-sensitive work (short sub-agent turns, embeddings) without ever touching the primary's context memory.

## The primary server (swap-in)

```ini
ExecStart=/usr/local/bin/llama-server \
    --models-preset /etc/llama/models-primary.ini \
    --models-max 1 \
    --no-mmproj \
    --metrics \
    --host :: \
    --port 8082
```

- **`--models-max 1`** — only one model resident in GPU memory at a time. With `load-on-startup = false` in the preset, the GPU is **empty at boot** and the requested model is loaded on first use. This is deliberate: on a 24 GB card you can't hold a 12B *and* a 27B *and* their big KV caches simultaneously, so the server swaps between them. The cost is a few seconds of load latency on a model switch.
- **`--no-mmproj`** — text-only serving; don't load the multimodal projection file you're not using.
- **`--metrics`** — Prometheus endpoint at `/metrics` for monitoring.

## The secondary server (resident + embeddings)

```ini
ExecStart=/usr/local/bin/llama-server \
    --models-preset /etc/llama/models-secondary.ini \
    --no-mmproj \
    --embeddings \
    --metrics \
    --host :: \
    --port 8083
```

- **`--embeddings` is a command-line flag, not a preset key.** The `/v1/embeddings` route is only exposed if the server is started with it — pooling set in the preset alone is *not* enough; without the flag the endpoint returns **501** for every model. This is a classic "it works in one place but not the other" trap: the embedding model (`bge-m3`) must be on a server that was started with `--embeddings`.
- **No `--models-max`** — the small card can hold both active models at once, so they both load eagerly (`load-on-startup = true`). Embedding requests and the first chat request never pay a load penalty.
- **`Restart=always`** — a worker you always want up.

### The embedding endpoint

A dedicated small embedding model (`bge-m3`, ~567M params, 1024-dim vectors, CLS pooling) on the worker card is the clean way to serve memory/RAG. Two things must both be true:

1. The server was started with `--embeddings`.
2. The model profile has `pooling = cls` (BGE-M3 pools on the CLS token).

The memory service then calls `http://host.docker.internal:8083/v1/embeddings` with `model: "bge-m3"`. Because it's a small, resident, fast model, embeddings are essentially free and never compete with the primary's big context.

## Choosing the split for your own box

The general principle: **put the model that must be fast and cheap on the small card; put the model that must be smart and deep on the big card.** Concretely:

- If you have two cards, one small card is ideal for an **embedding model + a small always-resident chat worker** (sub-agent, quick utility, the thing Open WebUI hits by default).
- The big card is for the **primary agent** with the largest context you can afford — and it's fine to keep it swap-in so it idles empty.
- If you only have one card, collapse to a single server and use `--models-max` / `load-on-startup` to decide what's resident; you lose hardware isolation but keep everything else.

## Related

- [Context & Quantisation](/ai-docs/02-context-and-quantisation/) — the VRAM math behind the choices above
- [Sub-agents](/ai-docs/04-sub-agents/) — the spec→code pattern that motivates the worker card
- [01 — Architecture](/ai-docs/01-architecture/)

---

## Source Disclaimer

- [x] AI Generated
- [ ] Human Generated
- [x] AI Edited
- [ ] Human Edited