---
title: "AI Machine — Concepts & Tips"
description: "This directory documents a self-hosted AI inference and application stack: native `llama.cpp` inference servers on the GPUs, fronted by a Docker application tier (Open WebUI, Open Terminal, Pipelines)…"
section: ai-docs
raw: "README.md"
source: ai-generated
tags: ai, llama.cpp, docker
last-updated: 2026-09-15
---

# AI Machine — Concepts & Tips

> Concept-based documentation for building and improving a self-hosted local AI stack. Written for a third party setting up their own instance — the patterns, the trade-offs, and the pitfalls — not a line-by-line description of one specific deployment.

This directory documents a self-hosted AI inference and application stack: native `llama.cpp` inference servers on the GPUs, fronted by a Docker application tier (Open WebUI, Open Terminal, Pipelines). The source configuration files it was derived from live in the `Hex-etc` directory (`docker/compose/`, `systemd/system/`, `llama/models-*.ini`) and are intentionally **not** in this repo.

> **Security note:** all API keys and secrets are **redacted** (`<REDACTED>`)
> throughout. Do not re-introduce real credentials into the repository.

## What you'll learn here

The docs are organised around the decisions that actually shape a local AI setup, not around the files that describe it. Each page leads with the concept, then the tips/tricks and pitfalls learned from running the system.

## Document index

| Document | Concepts & tips covered |
| --- | --- |
| [01 — Architecture](/ai-docs/01-architecture/) | The two-tier pattern (native inference / Docker apps), one server per GPU, `host.docker.internal` wiring, where data lives, sizing the machine. |
| [02 — Context Sizing & Quantisation](/ai-docs/02-context-and-quantisation/) | The VRAM budget equation, weight quantisation choices, context as a budget line, resident vs swap-in loading, and the pitfalls (parallel slot division, embedding batch caps, thinking models, mangled model names). |
| [03 — Primary vs Secondary `llama-server`](/ai-docs/03-primary-vs-secondary/) | The big-brain / small-worker split, swap-in vs eager-resident design, the `--embeddings` flag, and how to choose the split for your own cards. |
| [04 — Sub-agent Use](/ai-docs/04-sub-agents/) | The spec→code pattern, context isolation, self-contained sub-task prompts, matching models to roles, parallel delegation. |
| [05 — Open Terminal & Tooling](/ai-docs/05-open-terminal-and-tooling/) | Giving the model a shell, the API-key/multi-user setup, persistence, and the security guardrails that matter. |
| [06 — Skills](/ai-docs/06-skills/) | Packaged on-demand knowledge, anatomy of a good skill, skills vs tools vs sub-agents vs MCP, and tips for building them. |
| [07 — Dual-Use: Inference ⇄ Gaming/Desktop](/ai-docs/07-dual-use-gaming/) | Auto-switching a headless AI box into a desktop/gaming machine (and back) with a udev rule + two scripts; ordering, logging, and DDC/CI tips. |
| [08 — Build Instructions](/ai-docs/08-build-instructions/) | How to build from scratch: Ubuntu → Docker → llama.cpp → systemd services → containers → Web UI wiring, with example configs and scripts. |

## Quick orientation

```text
                       ┌────────────────────────────────────────────┐
   Browser / clients   │  APPLICATION TIER (Docker, ai-shared-net)  │
   ──────────────────► │  Open WebUI (8081) · Open Terminal (8080)  │
                       │  Pipelines (9099)                          │
                       └───────────────┬────────────────────────────┘
                                       │ host.docker.internal (host-gateway)
                       ┌───────────────▼────────────────────────────┐
   GPU 0 (RTX 3090)    │  INFERENCE TIER (native, systemd)          │
   ──────────────────► │  llama-primary  :8082 — the big "brain"    │
                       │  llama-secondary:8083 — worker + embeddings│
                       └────────────────────────────────────────────┘

   Sub-agents: brain model ──delegates──► worker model (fresh context)
   Tools:      any tier ──API call──► Open Terminal ──► real shell
   Skills:     packaged procedure + reference, loaded on demand
```

## Conventions

- Plain markdown only; diagrams in ascii-art or mermaid.js; prose is not hard-wrapped.
- New documents start from [templates/doc.md](/raw/_shared/templates/doc.md) and end with the Source Disclaimer block.
- Config files are referenced by name, never quoted with secrets; anything credential-like is `<REDACTED>`.

---

## Source Disclaimer

- [x] AI Generated
- [ ] Human Generated
- [x] AI Edited
- [ ] Human Edited