How to build from scratch

08 — Build Instructions

How to build the system from scratch on a fresh Ubuntu install: native llama.cpp inference servers, a Docker application tier, and the initial Web UI wiring.

How to build from scratch

The build order, and what each piece produces:

1. Ubuntu                           → a working base OS
2. PostgreSQL                       → host database for Open WebUI
3. Docker + folder layout           → /etc/docker/compose, /var/local/docker-files
4. NVIDIA drivers                   → nvidia-smi shows both GPUs
5. llama.cpp                        → /usr/local/bin/llama-server
6. Model + config folders           → /usr/local/llama/models, /etc/llama
7. systemd services                 → llama-primary (:8082), llama-secondary (:8083)
8. Docker containers                → Open WebUI (:8081), Open Terminal (:8080), Pipelines (:9099)
9. Web UI configuration             → connections, integrations, sub-agent, skills
10. (Bonus) Authentik OAuth         → login via https://auth.<TLD>

Steps 1–5 are standard installs; where upstream documentation exists, follow it. This document covers the parts specific to this stack: the folder layout, the systemd units, the model preset files, the compose files, and the Web UI wiring.

Security note: all API keys and secrets are redacted (<REDACTED>) throughout. Generate your own.

1 — Install Ubuntu

Install Ubuntu Server (minimal) and SSH access per the official Ubuntu installation guide. No stack-specific configuration is needed at this stage.

2 — PostgreSQL

PostgreSQL is not required, but recommended for storing application and user data.

2.1 — Install PostgreSQL

Install per the official PostgreSQL installation guide:

sudo apt install postgresql

2.2 — Enable Password Authentication

Configure /etc/postgresql/<version>/main/pg_hba.conf for local authentication — ensure this line is present (uncomment or add):

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     scram-sha-256
# IPv4 local connections:
host    all             all             127.0.0.1/32            scram-sha-256
# IPv6 local connections:
host    all             all             ::1/128                 scram-sha-256

sudo systemctl reload postgresql

2.3 — Create the openwebui user and database

On Debian and Ubuntu, the quickest way to admin PostgreSQL is via the postgres system account:

sudo -u postgres psql

Once connected, create the user with:

CREATE USER openwebui WITH NOCREATEDB PASSWORD '<Insert-Password>';

With the user created, you can now create a database owned by that user:

CREATE DATABASE openwebui WITH OWNER openwebui;

You will need these credentials later when configuring the OpenWebUI docker container.

3 — Install Docker and create the folder layout

Install Docker Engine and the Compose plugin per the official Docker install guide.

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

Create the two roots used throughout the build:

/etc/docker/compose — one subdirectory per deployment, each holding its own docker-compose.yaml (+ .env).
/var/local/docker-files — persisted application data, one subdirectory per container.

/var/local follows the FHS convention for variable data belonging to applications installed under /usr/local¹.

sudo mkdir -p /etc/docker/compose/open-webui
sudo mkdir -p /var/local/docker-files/{open-webui,open-terminal,open-webui-pipelines}

4 — Install NVIDIA drivers and llama.cpp

While Ubuntu ships packaged NVIDIA drivers (apt install nvidia-driver installs the currently supported version), upstream drivers will provide new features.

4.1 — Install NVIDIA Upstream Drivers

Install the proprietary NVIDIA driver and toolkit per the NVIDIA Driver Installation Guide — Ubuntu.

Choose the Network Repository Enablement (amd64) instructions. The <distribution> is ubuntu2604.

Verify with nvidia-smi -L — you should see one line per GPU (GPU 0 = the big card, GPU 1 = the small card).

5 — Install llama.cpp

llama.cpp can be installed as pre-built packages, or compiled from source. If you intend to use asymmetric quantisation of the KV cache, currently you will need to build from source.

5.1 — Upstream llama.cpp packages

Install llama.cpp from the official release builds — download the CUDA-enabled llama-*.tar.gz asset and place the binaries in /usr/local/bin:

sudo tar -xzf llama-*.tar.gz -C /usr/local/bin

⚠️ The release asset names and folder layout change between releases; check the current release page.

5.2 — Build from source

Install the prerequisites:

sudo apt update
sudo apt install -y git build-essential cmake libcurl4-openssl-dev

Clone the repository:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

Build:

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_NATIVE=OFF
cmake --build build --config Release -j$(nproc)

Install:

sudo cmake --install build --prefix /usr/local

5.3 — Verify your installation

Verify:

/usr/local/bin/llama-server --version

6 — Create model and configuration folders

sudo mkdir -p /usr/local/llama/models /usr/local/llama/templates /etc/llama

/usr/local/llama/models — home for GGUF model files.
/usr/local/llama/templates — home for custom chat templates (.jinja) referenced by the presets.
/etc/llama — home for the model preset files (.ini) that the systemd units point at.

Download models from Ollama or Hugging Face and place the .gguf files under /usr/local/llama/models. The current deployment is a pure llama.cpp install (no Ollama): every preset model = line points at a named .gguf file in that directory, and custom chat templates live in /usr/local/llama/templates/.

7 — Create the systemd services for the inference servers

Two units, one per GPU — see Primary vs Secondary for the role split. The model presets live in /etc/llama/; sizing guidance is in Context & Quantisation.

/etc/llama/models-primary.ini (big card — the “brain”; two models, large context):

[version]
version = 1

[*]
# Global fallbacks for all models
sleep-idle-seconds = 3600
load-on-startup = false
flash-attn = true
batch-size = 1024
ubatch-size = 256
threads = 6
threads-batch = 6
gpu-layers = 99

[gemma4-12b]
model = /usr/local/llama/models/Gemma4-12B-Q4_K_M.gguf
ctx-size = 262144
context-shift = true
keep = 1024
cache-type-k = q8_0
cache-type-v = q4_0

[qwen3.8-27b]
model = /usr/local/llama/models/Qwen3.8-27B-UD-Q4_K_M.gguf
ctx-size = 185000
context-shift = true
keep = 1024
cache-type-k = q8_0
cache-type-v = q4_0
jinja = true
chat-template-file = /usr/local/llama/templates/froggeric_chat_template.jinja

/etc/llama/models-secondary.ini (small card — resident embedding model + sub-agent worker):

[version]
version = 1

[*]
flash-attn = true
threads = 6
threads-batch = 6
gpu-layers = 99
chat-template-kwargs = {"enable_thinking": false}

[bge-m3]
model = /usr/local/llama/models/BGE-M3.gguf
load-on-startup = true
pooling = cls
ctx-size = 2048
cache-type-k = f16
cache-type-v = f16
batch-size = 1024
ubatch-size = 1024

[qwen3.5-4b]
model = /usr/local/llama/models/Qwen3.5-4B-Q6_K.gguf
load-on-startup = true
parallel = 2
ctx-size = 8192
kv-unified = true
cache-type-k = q8_0
cache-type-v = q4_0
batch-size = 2048
ubatch-size = 512

/etc/systemd/system/llama-primary.service:

[Unit]
Description=Llama.cpp Primary Server

[Service]
Type=simple
User=ollama
Group=ollama
WorkingDirectory=/usr/local/bin
Environment=CUDA_VISIBLE_DEVICES=0
ExecStart=/usr/local/bin/llama-server \
    --models-preset /etc/llama/models-primary.ini \
    --models-max 1 \
    --no-mmproj \
    --metrics \
    --host :: \
    --port 8082
Restart=on-failure
RestartSec=5
StandardOutput=append:/var/log/llama_primary.log
StandardError=append:/var/log/llama_primary_error.log

[Install]
WantedBy=multi-user.target

/etc/systemd/system/llama-secondary.service — same shape, with these differences:

[Unit]
Description=Llama.cpp Secondary Server (preset: bge-m3 embeddings, qwen2.5:0.5B, qwen2.5:7B-instruct, qwen3.5:9B)
After=network.target

[Service]
User=ollama
Group=ollama
WorkingDirectory=/usr/local/bin
Environment=CUDA_VISIBLE_DEVICES=1
ExecStart=/usr/local/bin/llama-server \
    --models-preset /etc/llama/models-secondary.ini \
    --no-mmproj \
    --embeddings \
    --metrics \
    --host :: \
    --port 8083
Restart=always
RestartSec=5
StandardOutput=append:/var/log/llama_secondary.log
StandardError=append:/var/log/llama_secondary_error.log

Notes:

User=ollama — create a dedicated service account first (sudo useradd -r -s /usr/sbin/nologin ollama); the units run unprivileged but still need read access to the model files.
The --embeddings flag on the secondary enables the /v1/embeddings endpoint for the memory service.
The --models-preset file format is documented in the llama.cpp models preset documentation.

Enable and verify:

sudo systemctl daemon-reload
sudo systemctl enable --now llama-primary llama-secondary
curl http://localhost:8082/v1/models
curl http://localhost:8083/v1/models

8 — Deploy the Docker containers

Create the compose directories from step 3 and bring up the containers without OAuth configured (that is step 10). All application containers share the ai-shared-net network and map host.docker.internal to the Docker host so they can reach the native llama servers.

/etc/docker/compose/open-webui/docker-compose.yaml (three services; without OAuth the .env is minimal — just WEBUI_AUTH=true and a DATABASE_URL, see step 9 for the full version):

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "8081:8080"
    env_file:
    - .env
    volumes:
      - /var/local/docker-files/open-webui:/app/backend/data
      - /run/postgresql:/postgresql
    networks:
      - ai-network
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

  open-terminal:
    image: ghcr.io/open-webui/open-terminal
    container_name: open-terminal
    ports:
      - "8080:8080"
    volumes:
      - /var/local/docker-files/open-terminal:/home
    environment:
      - OPEN_TERMINAL_API_KEY=<REDACTED>
      - OPEN_TERMINAL_MULTI_USER=true
    networks:
      - ai-network
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

  pipelines:
    image: ghcr.io/open-webui/pipelines:main
    container_name: open-webui-pipelines
    ports:
      - "9099:9099"
    environment:
      - PIPELINES_API_KEY=<REDACTED>
    volumes:
      - /var/local/docker-files/open-webui-pipelines:/app/pipelines
    networks:
      - ai-network
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

networks:
  ai-network:
    name: ai-shared-net

Deploy:

docker network create ai-shared-net   # once; the first compose file also creates it
cd /etc/docker/compose/open-webui && docker compose up -d

9 — Initial Web UI configuration

Open WebUI at http://<host>:8081. On first start it creates an admin account — note it; you’ll be the first user.

9.1 — Configure model connections

Admin Panel → Connections → Add Connection (OpenAI API style). The base URLs are entered in the UI and stored in Open WebUI’s data volume, so they survive image upgrades but not a wiped volume — keep the volume backed up (see 01 — Architecture).

Name	Base URL	Model(s)	Role
`llama-primary`	`http://host.docker.internal:8082/v1`	brain models (12B / 27B)	main chat, sub-agent planning
`llama-secondary`	`http://host.docker.internal:8083/v1`	worker (4B), `bge-m3`	sub-agent worker, embeddings

⚠️ The exact UI labels for a raw OpenAI-compatible endpoint vs the “OpenAI API” connection type vary between Open WebUI releases; if a plain endpoint isn’t offered, use the “OpenAI API” connection type with the base URLs above and no API key.

9.2 — Configure Integrations

Add Local-Terminal as an integration (Admin Panel → Integrations → Add), pointing at the Open Terminal service from step 8:

Name	Base URL	Notes
`Local-Terminal`	`http://host.docker.internal:8080`	Open Terminal tool; API key from step 8

See Open Terminal & Tooling for the security notes that apply to this integration.

9.3 — Configure the sub-agent

Sub-agents are provided by the community Sub Agent tool — import the tool code from the Sub Agent Tool post into the Admin Panel (Tools → Add Tool → paste the code) and enable it. It runs tool-heavy tasks in an isolated context and returns only the final result; the model it delegates to is the worker on the llama-secondary connection (see Sub-agents for the pattern).

Requirement: the model must have native Function Calling enabled (Model settings → Advanced Params → Function Calling: native).

9.4 — Configure skills

Skills are files in the agent’s workspace, not Web UI settings — see Skills for the concept and 05 — Open Terminal & Tooling for the terminal they direct. In practice: create a known directory (persisted, e.g. under /var/local/docker-files/open-terminal/ so it survives container rebuilds), drop skill files in it, and reference that path in the system prompt or the sub-agent briefs.

⚠️ The skill storage location and how the deployed agent discovers skills is deployment-specific — confirm the path the agent actually reads.

10 — Bonus: configure OAuth for Authentik

Once the UI works with a local account, switch it to Authentik via OpenID Connect, per the Open WebUI OAuth2/OIDC documentation.

Create the application. In Authentik, create an Application (Provider: OpenID, Protocol: OIDC) for Open WebUI and note the client ID/secret.

Replace the minimal .env from step 8:

OAUTH_CLIENT_ID=<REDACTED>
OAUTH_CLIENT_SECRET=<REDACTED>
OAUTH_PROVIDER_NAME=Auth.<TLD>
OAUTH_AUTO_REDIRECT=true
OPENID_PROVIDER_URL=https://auth.<TLD>/application/o/open-web-ui/.well-known/openid-configuration
OPENID_REDIRECT_URI=https://ai.<TLD>/oauth/oidc/callback

# Core auth: OAuth only, no local sign-up
WEBUI_AUTH=true
ENABLE_LOGIN_FORM=false
ENABLE_SIGNUP=false
ENABLE_OAUTH_SIGNUP=true

# General
WEBUI_URL=https://ai.<TLD>
CORS_ALLOW_ORIGIN=https://ai.<TLD>;http://<lan-ip>:8081
DATABASE_URL=postgresql://openwebui:<REDACTED>@/openwebui?host=/postgresql

Restart: docker compose up -d.

⚠️ This assumes a reverse proxy (Caddy/Traefik/nginx) terminating TLS at ai.<TLD> and forwarding to :8081, with the container able to reach the Authentik server. The redirect URI, WEBUI_URL and CORS_ALLOW_ORIGIN must all match the proxy hostname.

Uncertainties

Areas above that are flagged with ⚠️, summarised:

llama.cpp release naming — asset names change between releases.
Open WebUI connection UI — exact connection type/labels depend on the deployed version.
Skill discovery path — deployment-specific.

01 — Architecture — what is being built and why
03 — Primary vs Secondary — the inference server split
05 — Open Terminal & Tooling — the terminal container and its security notes
06 — Skills — step 9.4

Source Disclaimer

AI Generated
Human Generated
AI Edited
Human Edited

FHS — Variable Hierarchy ↩︎

08 — Build Instructions

How to build from scratch

1 — Install Ubuntu

2 — PostgreSQL

2.1 — Install PostgreSQL

2.2 — Enable Password Authentication

2.3 — Create the openwebui user and database

3 — Install Docker and create the folder layout

4 — Install NVIDIA drivers and llama.cpp

4.1 — Install NVIDIA Upstream Drivers

5 — Install llama.cpp

5.1 — Upstream llama.cpp packages

5.2 — Build from source

5.3 — Verify your installation

6 — Create model and configuration folders

7 — Create the systemd services for the inference servers

8 — Deploy the Docker containers

9 — Initial Web UI configuration

9.1 — Configure model connections

9.2 — Configure Integrations

9.3 — Configure the sub-agent

9.4 — Configure skills

10 — Bonus: configure OAuth for Authentik

Uncertainties

Related

Source Disclaimer