docs.na.id.au

This directory documents a self-hosted AI inference and application stack: native `llama.cpp` inference servers on the GPUs, fronted by a Docker application tier (Open WebUI, Open Terminal, Pipelines)…

AI Machine — Concepts & Tips

Concept-based documentation for building and improving a self-hosted local AI stack. Written for a third party setting up their own instance — the patterns, the trade-offs, and the pitfalls — not a line-by-line description of one specific deployment.

This directory documents a self-hosted AI inference and application stack: native llama.cpp inference servers on the GPUs, fronted by a Docker application tier (Open WebUI, Open Terminal, Pipelines). The source configuration files it was derived from live in the Hex-etc directory (docker/compose/, systemd/system/, llama/models-*.ini) and are intentionally not in this repo.

Security note: all API keys and secrets are redacted (<REDACTED>) throughout. Do not re-introduce real credentials into the repository.

What you’ll learn here

The docs are organised around the decisions that actually shape a local AI setup, not around the files that describe it. Each page leads with the concept, then the tips/tricks and pitfalls learned from running the system.

Document index

DocumentConcepts & tips covered
01 — ArchitectureThe two-tier pattern (native inference / Docker apps), one server per GPU, host.docker.internal wiring, where data lives, sizing the machine.
02 — Context Sizing & QuantisationThe VRAM budget equation, weight quantisation choices, context as a budget line, resident vs swap-in loading, and the pitfalls (parallel slot division, embedding batch caps, thinking models, mangled model names).
03 — Primary vs Secondary llama-serverThe big-brain / small-worker split, swap-in vs eager-resident design, the --embeddings flag, and how to choose the split for your own cards.
04 — Sub-agent UseThe spec→code pattern, context isolation, self-contained sub-task prompts, matching models to roles, parallel delegation.
05 — Open Terminal & ToolingGiving the model a shell, the API-key/multi-user setup, persistence, and the security guardrails that matter.
06 — SkillsPackaged on-demand knowledge, anatomy of a good skill, skills vs tools vs sub-agents vs MCP, and tips for building them.
07 — Dual-Use: Inference ⇄ Gaming/DesktopAuto-switching a headless AI box into a desktop/gaming machine (and back) with a udev rule + two scripts; ordering, logging, and DDC/CI tips.
08 — Build InstructionsHow to build from scratch: Ubuntu → Docker → llama.cpp → systemd services → containers → Web UI wiring, with example configs and scripts.

Quick orientation

                       ┌────────────────────────────────────────────┐
   Browser / clients   │  APPLICATION TIER (Docker, ai-shared-net)  │
   ──────────────────► │  Open WebUI (8081) · Open Terminal (8080)  │
                       │  Pipelines (9099)                          │
                       └───────────────┬────────────────────────────┘
                                       │ host.docker.internal (host-gateway)
                       ┌───────────────▼────────────────────────────┐
   GPU 0 (RTX 3090)    │  INFERENCE TIER (native, systemd)          │
   ──────────────────► │  llama-primary  :8082 — the big "brain"    │
                       │  llama-secondary:8083 — worker + embeddings│
                       └────────────────────────────────────────────┘

   Sub-agents: brain model ──delegates──► worker model (fresh context)
   Tools:      any tier ──API call──► Open Terminal ──► real shell
   Skills:     packaged procedure + reference, loaded on demand

Conventions

Source Disclaimer