The two-tier pattern

01 — Architecture

The overall shape of a self-hosted local AI stack: two tiers, GPU isolation, and how the pieces find each other.

The two-tier pattern

A local AI machine splits cleanly into two tiers, and the split is worth keeping no matter how the details change:

• Inference tier — one or more llama.cpp servers running natively on the host (plain systemd services), each pinned to a specific GPU via CUDA_VISIBLE_DEVICES. They expose an OpenAI-compatible HTTP API.
• Application tier — Docker containers (chat UI, terminal, plugin orchestration) on a shared user-defined Docker network.

Why native for inference rather than Docker? A GPU workload wants the full device with no container indirection, and the model files live on the host filesystem. Keeping inference out of containers means the web tooling can be rebuilt, upgraded and restarted independently — a broken Open WebUI image never takes the models down with it.

The local machine follows this exactly:

One server per GPU

CUDA_VISIBLE_DEVICES is what gives you hardware isolation between servers: each llama-server only sees its own card, so a context-heavy request on the big card can never evict or starve the small one, and vice versa. This is the whole reason to run two servers on one box rather than one server on two GPUs — it is as good as two machines, except you can share models between them when convenient.

Each server is an ordinary, individually startable systemd unit (Restart=on-failure / Restart=always, RestartSec=5, logs appended to /var/log/llama_*.log). Ordinary units matter if the machine is dual-use: anything that needs the GPUs (a desktop, a render job) can simply systemctl stop the server, and the default state is whichever one you start at boot. This box additionally auto-switches between headless inference and an interactive desktop (or gaming machine) when a USB monitor switcher is plugged in or out — see Dual-Use: Inference ⇄ Gaming/Desktop.

How the tiers talk to each other

Docker containers cannot see host services by name, so the standard trick is used: every container declares

extra_hosts:
  - "host.docker.internal:host-gateway"

which maps the hostname host.docker.internal to the Docker host. Containers then reach the native servers with http://host.docker.internal:8082/v1 and :8083/v1. Two practical notes:

• The shared network itself (ai-shared-net) exists mainly so the application containers can address each other; it is not how they reach the inference tier.
• The inference base URLs are not in any config file here — they are entered through the Open WebUI web UI and stored in its data volume. That means they survive image upgrades but not a wiped data volume, so keep the data volume backed up.

Where data lives

A pattern that works well: application data is persisted to host paths under a single root (here /var/local/docker-files/<service>/), while model files live in one shared directory on the host (/usr/local/llama/models/ for the .gguf files) with a preset file per server (/etc/llama/models-*.ini) deciding which models each server loads. The deployment was originally built around Ollama’s per-server blob stores (.ollama/ for the primary, .ollama-secondary/ for the secondary) but has since migrated to a pure llama.cpp install — one directory is cleaner, and “which GPU does this model belong to” stays answerable from the preset it appears in. Databases that don’t need to scale (Postgres for Open WebUI’s data) can run on the host and be reached over a mounted Unix socket instead of being containerised — simpler lifecycle, lower latency, one fewer container.

Typical request flow

User ──HTTP──► Open WebUI (:8081)
                 │  base URL configured in the web UI:
                 │  http://host.docker.internal:8082/v1  (or :8083)
                 ▼
             llama-server (native, on the host)
                 │  runs inference on its own GPU
                 ▼
             token stream ──► Open WebUI ──► User

Tool-assisted chat adds a loop: the application tier calls a tool (e.g. Open Terminal’s API, an MCP server), the tool does something, and its output goes back into the model’s context for the next turn. Sub-agents are just a special case of that loop where the “tool” is another model endpoint — see Sub-agents.

Sizing the machine

The useful mental model when designing your own setup is VRAM budget per card, not raw model count:

VRAM_needed ≈ weights(quant) + KV_cache(ctx × layers × quant) + ~0.5 GB overhead

Everything else — which models to run, how much context, whether a model is resident or swapped — falls out of that equation. The local setup works out the numbers for an 8 GB and a 24 GB card; see Context & Quantisation.

Related

• Context & Quantisation
• Primary vs Secondary
• Open Terminal & Tooling
• Dual-Use: Inference ⇄ Gaming/Desktop

Source Disclaimer

• AI Generated
• Human Generated
• AI Edited
• Human Edited