#llama.cpp

llama.cpp GGUF Quantization and llama-server Launch Planner: Quant Choice by VRAM, Context Length and KV Cache Budget, GPU Layer Offload, Parallel Slots, Chat Template, and OpenAI Compatible Endpoint Tests

Plan a local model server on your own GPU with llama.cpp: choose a GGUF quantization that fits your VRAM, budget the KV cache for the context length and number of parallel users, set GPU offload, flash attention, and cache type flags, load the right chat template, and test the OpenAI compatible endpoint before pointing apps at it.

Act as a local inference engineer who deploys llama.cpp llama-server on workstations and small servers for teams, and who sizes every launch from the model architecture and the GPU's free memory instead of guessing until it stops crashing. Inputs: - The model and its architecture details from the model card or GGUF metadata (parameters, layers, attention heads, KV heads, head dimension, trained context): [ModelArch] - GPU model, total VRAM, and what else uses it (desktop, other apps); CPU and system RAM: [Hardware] - Context length needed per request and how many concurrent users or requests: [Workload] - Quality priority (coding, extraction, chat) and acceptable speed: [QualityNeeds] - llama.cpp build or release tag and how it was installed: [BuildVersion] - Who connects and from where (localhost only, LAN, an app using the OpenAI SDK): [ClientSetup] - Output format: [Format] Generate: 1. A quantization shortlist (for example Q8_0, Q6_K, Q5_K_M, Q4_K_M, IQ4_XS) with approximate file size for this parameter count, the expected quality tradeoff in plain words, and which one fits QualityNeeds. 2. KV cache math: bytes per token = 2 x layers x KV heads x head dimension x bytes per element, then total for the context in Workload, for f16 and for q8_0 cache types. Use the KV head count, not the attention head count, for grouped query attention models. 3. A VRAM budget table: weights plus KV cache plus a compute buffer allowance against free VRAM, with the recommended quant and context, and what to change first if it does not fit (cache type, context, quant, then partial offload). 4. Parallel slots: how the total context is divided across --parallel slots in most builds, so per user context is clear, and when to raise context instead. 5. The llama-server command with each flag explained: model path, --host and --port, context size, --parallel, GPU layers, flash attention, cache types, --jinja for the model's chat template, an API key, and threads for any CPU part. 6. Endpoint tests with curl: /health, /v1/models, and a /v1/chat/completions request, plus the base_url and key settings for an OpenAI SDK client. 7. Tuning notes: how to read the startup log for offloaded layers and buffer sizes, and the symptom of each common mistake (out of memory at load, gibberish from a wrong template, slow speed from partial offload). Constraints: - Flag names change between releases; tell the user to confirm against llama-server --help for their BuildVersion. - Do not publish benchmark speeds you cannot know; give sizes and math, and label estimates. - Never suggest exposing the server on a public interface without an API key and a firewall. No em dashes.