--- title: "08 — Build Instructions" description: "## How to build from scratch" section: ai-docs raw: "08-build-instructions.md" source: ai-generated tags: ai, llama.cpp, docker last-updated: 2026-09-15 --- # 08 — Build Instructions > How to build the system from scratch on a fresh Ubuntu install: native `llama.cpp` inference servers, a Docker application tier, and the initial Web UI wiring. ## How to build from scratch The build order, and what each piece produces: ```text 1. Ubuntu → a working base OS 2. PostgreSQL → host database for Open WebUI 3. Docker + folder layout → /etc/docker/compose, /var/local/docker-files 4. NVIDIA drivers → nvidia-smi shows both GPUs 5. llama.cpp → /usr/local/bin/llama-server 6. Model + config folders → /usr/local/llama/models, /etc/llama 7. systemd services → llama-primary (:8082), llama-secondary (:8083) 8. Docker containers → Open WebUI (:8081), Open Terminal (:8080), Pipelines (:9099) 9. Web UI configuration → connections, integrations, sub-agent, skills 10. (Bonus) Authentik OAuth → login via https://auth. ``` Steps 1–5 are standard installs; where upstream documentation exists, follow it. This document covers the parts specific to this stack: the folder layout, the systemd units, the model preset files, the compose files, and the Web UI wiring. > **Security note:** all API keys and secrets are redacted (``) throughout. Generate your own. ## 1 — Install Ubuntu Install Ubuntu Server (minimal) and SSH access per the official [Ubuntu installation guide](https://ubuntu.com/tutorials/install-ubuntu-server). No stack-specific configuration is needed at this stage. ## 2 — PostgreSQL PostgreSQL is not required, but recommended for storing application and user data. ### 2.1 — Install PostgreSQL Install per the [official PostgreSQL installation guide](https://www.postgresql.org/download/linux/ubuntu/): ```bash sudo apt install postgresql ``` ### 2.2 — Enable Password Authentication Configure `/etc/postgresql//main/pg_hba.conf` for local authentication — ensure this line is present (uncomment or add): ```text # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all scram-sha-256 # IPv4 local connections: host all all 127.0.0.1/32 scram-sha-256 # IPv6 local connections: host all all ::1/128 scram-sha-256 ``` ```bash sudo systemctl reload postgresql ``` ### 2.3 — Create the openwebui user and database On Debian and Ubuntu, the quickest way to admin PostgreSQL is via the postgres system account: ```bash sudo -u postgres psql ``` Once connected, create the user with: ```PostgreSQL CREATE USER openwebui WITH NOCREATEDB PASSWORD ''; ``` With the user created, you can now create a database owned by that user: ```PostgreSQL CREATE DATABASE openwebui WITH OWNER openwebui; ``` You will need these credentials later when configuring the OpenWebUI docker container. ## 3 — Install Docker and create the folder layout Install Docker Engine and the Compose plugin per the official [Docker install guide](https://docs.docker.com/engine/install/ubuntu/). Register the NVIDIA runtime so containers can request GPUs (add to `/etc/docker/daemon.json`): ```json { "runtimes": { "nvidia": { "args": [], "path": "nvidia-container-runtime" } } } ``` Create the two roots used throughout the build: - `/etc/docker/compose` — one subdirectory per deployment, each holding its own `docker-compose.yaml` (+ `.env`). - `/var/local/docker-files` — persisted application data, one subdirectory per container. `/var/local` follows the FHS convention for variable data belonging to applications installed under `/usr/local`[^fhs]. [^fhs]: FHS — [Variable Hierarchy](https://specifications.freedesktop.org/fhs/latest/varHierarchy.html) ```bash sudo mkdir -p /etc/docker/compose/open-webui sudo mkdir -p /var/local/docker-files/{open-webui,open-terminal,open-webui-pipelines} ``` ## 4 — Install NVIDIA drivers and llama.cpp While Ubuntu ships packaged NVIDIA drivers (`apt install nvidia-driver` installs the currently supported version), upstream drivers will provide new features. ### 4.1 — Install NVIDIA Upstream Drivers Install the proprietary NVIDIA driver and toolkit per the [NVIDIA Driver Installation Guide — Ubuntu](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/latest/ubuntu.html). Choose the Network Repository Enablement (amd64) instructions. The `` is `ubuntu2604`. Verify with `nvidia-smi -L` — you should see one line per GPU (GPU 0 = the big card, GPU 1 = the small card). ## 5 — Install llama.cpp llama.cpp can be installed as pre-built packages, or compiled from source. If you intend to use asymmetric quantisation of the KV cache, currently you will need to build from source. ### 5.1 — Upstream llama.cpp packages Install `llama.cpp` from the [official release builds](https://github.com/ggml-org/llama.cpp/releases) — download the CUDA-enabled `llama-*.tar.gz` asset and place the binaries in `/usr/local/bin`: ```bash sudo tar -xzf llama-*.tar.gz -C /usr/local/bin ``` > ⚠️ The release asset names and folder layout change between releases; check the current release page. ### 5.2 — Build from source Install the prerequisites: ```bash sudo apt update sudo apt install -y git build-essential cmake libcurl4-openssl-dev ``` Clone the repository: ```bash git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp ``` Build: ```bash cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_NATIVE=OFF cmake --build build --config Release -j$(nproc) ``` Install: ```bash sudo cmake --install build --prefix /usr/local ``` ### 5.3 — Verify your installation Verify: ```bash /usr/local/bin/llama-server --version ``` ## 6 — Create model and configuration folders ```bash sudo mkdir -p /usr/local/llama/models /usr/local/llama/templates /etc/llama ``` - `/usr/local/llama/models` — home for GGUF model files. - `/usr/local/llama/templates` — home for custom chat templates (`.jinja`) referenced by the presets. - `/etc/llama` — home for the model preset files (`.ini`) that the systemd units point at. Download models from [Ollama](https://ollama.com/library) or [Hugging Face](https://huggingface.co/models?search=gguf) and place the `.gguf` files under `/usr/local/llama/models`. The current deployment is a pure llama.cpp install (no Ollama): every preset `model =` line points at a named `.gguf` file in that directory, and custom chat templates live in `/usr/local/llama/templates/`. ## 7 — Create the systemd services for the inference servers Two units, one per GPU — see [Primary vs Secondary](/ai-docs/03-primary-vs-secondary/) for the role split. The model presets live in `/etc/llama/`; sizing guidance is in [Context & Quantisation](/ai-docs/02-context-and-quantisation/). `/etc/llama/models-primary.ini` (big card — the "brain"; two models, large context): ```ini [version] version = 1 [*] # Global fallbacks for all models sleep-idle-seconds = 3600 load-on-startup = false flash-attn = true batch-size = 1024 ubatch-size = 256 threads = 6 threads-batch = 6 gpu-layers = 99 [gemma4-12b] model = /usr/local/llama/models/Gemma4-12B-Q4_K_M.gguf ctx-size = 262144 context-shift = true keep = 1024 cache-type-k = q8_0 cache-type-v = q4_0 [qwen3.8-27b] model = /usr/local/llama/models/Qwen3.8-27B-UD-Q4_K_M.gguf ctx-size = 185000 context-shift = true keep = 1024 cache-type-k = q8_0 cache-type-v = q4_0 jinja = true chat-template-file = /usr/local/llama/templates/froggeric_chat_template.jinja ``` `/etc/llama/models-secondary.ini` (small card — resident embedding model + sub-agent worker): ```ini [version] version = 1 [*] flash-attn = true threads = 6 threads-batch = 6 gpu-layers = 99 chat-template-kwargs = {"enable_thinking": false} [bge-m3] model = /usr/local/llama/models/BGE-M3.gguf load-on-startup = true pooling = cls ctx-size = 2048 cache-type-k = f16 cache-type-v = f16 batch-size = 1024 ubatch-size = 1024 [qwen3.5-4b] model = /usr/local/llama/models/Qwen3.5-4B-Q6_K.gguf load-on-startup = true parallel = 2 ctx-size = 8192 kv-unified = true cache-type-k = q8_0 cache-type-v = q4_0 batch-size = 2048 ubatch-size = 512 ``` `/etc/systemd/system/llama-primary.service`: ```ini [Unit] Description=Llama.cpp Primary Server [Service] Type=simple User=ollama Group=ollama WorkingDirectory=/usr/local/bin Environment=CUDA_VISIBLE_DEVICES=0 ExecStart=/usr/local/bin/llama-server \ --models-preset /etc/llama/models-primary.ini \ --models-max 1 \ --no-mmproj \ --metrics \ --host :: \ --port 8082 Restart=on-failure RestartSec=5 StandardOutput=append:/var/log/llama_primary.log StandardError=append:/var/log/llama_primary_error.log [Install] WantedBy=multi-user.target ``` `/etc/systemd/system/llama-secondary.service` — same shape, with these differences: ```ini [Unit] Description=Llama.cpp Secondary Server (preset: bge-m3 embeddings, qwen2.5:0.5B, qwen2.5:7B-instruct, qwen3.5:9B) After=network.target [Service] User=ollama Group=ollama WorkingDirectory=/usr/local/bin Environment=CUDA_VISIBLE_DEVICES=1 ExecStart=/usr/local/bin/llama-server \ --models-preset /etc/llama/models-secondary.ini \ --no-mmproj \ --embeddings \ --metrics \ --host :: \ --port 8083 Restart=always RestartSec=5 StandardOutput=append:/var/log/llama_secondary.log StandardError=append:/var/log/llama_secondary_error.log ``` Notes: - `User=ollama` — create a dedicated service account first (`sudo useradd -r -s /usr/sbin/nologin ollama`); the units run unprivileged but still need read access to the model files. - The `--embeddings` flag on the secondary enables the `/v1/embeddings` endpoint for the memory service. - The `--models-preset` file format is documented in the llama.cpp [models preset documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/model-preset.md). Enable and verify: ```bash sudo systemctl daemon-reload sudo systemctl enable --now llama-primary llama-secondary curl http://localhost:8082/v1/models curl http://localhost:8083/v1/models ``` ## 8 — Deploy the Docker containers Create the compose directories from step 3 and bring up the containers **without** OAuth configured (that is step 10). All application containers share the `ai-shared-net` network and map `host.docker.internal` to the Docker host so they can reach the native llama servers. `/etc/docker/compose/open-webui/docker-compose.yaml` (three services; without OAuth the `.env` is minimal — just `WEBUI_AUTH=true` and a `DATABASE_URL`, see step 9 for the full version): ```yaml services: open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui ports: - "8081:8080" env_file: - .env volumes: - /var/local/docker-files/open-webui:/app/backend/data - /run/postgresql:/postgresql networks: - ai-network extra_hosts: - "host.docker.internal:host-gateway" restart: unless-stopped open-terminal: image: ghcr.io/open-webui/open-terminal container_name: open-terminal ports: - "8080:8080" volumes: - /var/local/docker-files/open-terminal:/home environment: - OPEN_TERMINAL_API_KEY= - OPEN_TERMINAL_MULTI_USER=true networks: - ai-network extra_hosts: - "host.docker.internal:host-gateway" restart: unless-stopped pipelines: image: ghcr.io/open-webui/pipelines:main container_name: open-webui-pipelines ports: - "9099:9099" environment: - PIPELINES_API_KEY= volumes: - /var/local/docker-files/open-webui-pipelines:/app/pipelines networks: - ai-network extra_hosts: - "host.docker.internal:host-gateway" restart: unless-stopped networks: ai-network: name: ai-shared-net ``` Deploy: ```bash docker network create ai-shared-net # once; the first compose file also creates it cd /etc/docker/compose/open-webui && docker compose up -d ``` ## 9 — Initial Web UI configuration Open WebUI at `http://:8081`. On first start it creates an admin account — note it; you'll be the first user. ### 9.1 — Configure model connections Admin Panel → Connections → Add Connection (OpenAI API style). The base URLs are entered in the UI and stored in Open WebUI's data volume, so they survive image upgrades but not a wiped volume — keep the volume backed up (see [01 — Architecture](/ai-docs/01-architecture/)). | Name | Base URL | Model(s) | Role | | --- | --- | --- | --- | | `llama-primary` | `http://host.docker.internal:8082/v1` | brain models (12B / 27B) | main chat, sub-agent planning | | `llama-secondary` | `http://host.docker.internal:8083/v1` | worker (4B), `bge-m3` | sub-agent worker, embeddings | > ⚠️ The exact UI labels for a raw OpenAI-compatible endpoint vs the "OpenAI API" connection type vary between Open WebUI releases; if a plain endpoint isn't offered, use the "OpenAI API" connection type with the base URLs above and no API key. ### 9.2 — Configure Integrations Add `Local-Terminal` as an integration (Admin Panel → Integrations → Add), pointing at the Open Terminal service from step 8: | Name | Base URL | Notes | | --- | --- | --- | | `Local-Terminal` | `http://host.docker.internal:8080` | Open Terminal tool; API key from step 8 | See [Open Terminal & Tooling](/ai-docs/05-open-terminal-and-tooling/) for the security notes that apply to this integration. ### 9.3 — Configure the sub-agent Sub-agents are provided by the community **Sub Agent tool** — import the tool code from the [Sub Agent Tool post](https://openwebui.com/posts/sub_agent_7bfeb0b7) into the Admin Panel (Tools → Add Tool → paste the code) and enable it. It runs tool-heavy tasks in an isolated context and returns only the final result; the model it delegates to is the worker on the `llama-secondary` connection (see [Sub-agents](/ai-docs/04-sub-agents/) for the pattern). Requirement: the model must have native **Function Calling** enabled (Model settings → Advanced Params → Function Calling: native). ### 9.4 — Configure skills Skills are files in the agent's workspace, not Web UI settings — see [Skills](/ai-docs/06-skills/) for the concept and [05 — Open Terminal & Tooling](/ai-docs/05-open-terminal-and-tooling/) for the terminal they direct. In practice: create a known directory (persisted, e.g. under `/var/local/docker-files/open-terminal/` so it survives container rebuilds), drop skill files in it, and reference that path in the system prompt or the sub-agent briefs. > ⚠️ The skill storage location and how the deployed agent discovers skills is deployment-specific — confirm the path the agent actually reads. ## 10 — Bonus: configure OAuth for Authentik Once the UI works with a local account, switch it to [Authentik](https://docs.authentik.io/) via OpenID Connect, per the [Open WebUI OAuth2/OIDC documentation](https://docs.open-webui.com/docs/oauth). **Create the application.** In Authentik, create an **Application** (Provider: OpenID, Protocol: OIDC) for Open WebUI and note the client ID/secret. **Replace the minimal `.env`** from step 8: ```bash OAUTH_CLIENT_ID= OAUTH_CLIENT_SECRET= OAUTH_PROVIDER_NAME=Auth. OAUTH_AUTO_REDIRECT=true OPENID_PROVIDER_URL=https://auth./application/o/open-web-ui/.well-known/openid-configuration OPENID_REDIRECT_URI=https://ai./oauth/oidc/callback # Core auth: OAuth only, no local sign-up WEBUI_AUTH=true ENABLE_LOGIN_FORM=false ENABLE_SIGNUP=false ENABLE_OAUTH_SIGNUP=true # General WEBUI_URL=https://ai. CORS_ALLOW_ORIGIN=https://ai.;http://:8081 DATABASE_URL=postgresql://openwebui:@/openwebui?host=/postgresql ``` **Restart:** `docker compose up -d`. > ⚠️ This assumes a reverse proxy (Caddy/Traefik/nginx) terminating TLS at `ai.` and forwarding to `:8081`, with the container able to reach the Authentik server. The redirect URI, `WEBUI_URL` and `CORS_ALLOW_ORIGIN` must all match the proxy hostname. ## Uncertainties Areas above that are flagged with ⚠️, summarised: - **llama.cpp release naming** — asset names change between releases. - **Open WebUI connection UI** — exact connection type/labels depend on the deployed version. - **Skill discovery path** — deployment-specific. ## Related - [01 — Architecture](/ai-docs/01-architecture/) — what is being built and why - [03 — Primary vs Secondary](/ai-docs/03-primary-vs-secondary/) — the inference server split - [05 — Open Terminal & Tooling](/ai-docs/05-open-terminal-and-tooling/) — the terminal container and its security notes - [06 — Skills](/ai-docs/06-skills/) — step 9.4 --- ## Source Disclaimer - [x] AI Generated - [ ] Human Generated - [ ] AI Edited - [x] Human Edited