local-llm-serving
Run local LLMs in production — model/quant choice, serving stacks, VRAM budgets, fallback chains, evaluation.
SKILL.mdskills/local-llm-serving/SKILL.md
GitHubLocal LLM Serving
Local inference is infrastructure: match the model to the box, serve it safely, and never let it become a single point of failure.
When to use
- Running llama.cpp / Ollama / GGUF engines on laptop, box, or GPU hosts.
- Choosing models under VRAM/RAM budgets (16GB-class cards, CPU-only fallbacks).
- Building fallback chains that survive one engine going down.
Method
- Workload → budget first: tasks, context lengths, concurrency → compute + VRAM/RAM budget; remember the KV cache grows with context — leave headroom; quantize deliberately (Q4/Q5/Q8 trade-offs), not by default.
- Model selection: match model size to task (small for routing/summaries, larger for reasoning); check licenses for commercial use before adopting; pin exact versions + checksums (provenance).
- Serving stack: llama.cpp server / Ollama / vLLM-class as fits; run daemonless under systemd (or Windows service) with auto-restart; bind loopback — remote access only via authenticated tunnel.
- Performance: measure tokens/s + time-to-first-token at realistic context; tune offload layers, parallelism; don't overcommit VRAM ("it loads" ≠ "it serves").
- Operations: health endpoint, structured logs, model catalog pinned, disk space monitored for weights; upgrades = backup → swap → verify → rollback path.
- Fallback chains: local primary → local secondary → declared provider; swap models behind a stable API (OpenAI-compatible surface) so apps never change; test each failover deliberately.
- Evaluation: golden task suite per model change (
agent-evaluation); record quality + speed before promotion; no silent model swaps in production.
Quality gates
- VRAM/RAM + context budget documented for each served model.
- Versions/license pinned and recorded; loopback-only binding verified.
- Fallback chain tested (fail the primary on purpose).
- Golden-task evals recorded per model change.
Red lines
- No unauthenticated exposure of inference ports; no unvetted model licenses; no silent model swaps.
Related
agent-infrastructure-ops · agent-evaluation · non-docker-isolation.