Running a local model on one server: a reproducible blueprint

Most businesses that ask me about "private AI" do not need a cluster. They need one model, running on one machine they already control, reachable from their own applications, with no customer data leaving the building. That is a weekend of work if you size it correctly and a month of frustration if you do not. This is the configuration I use, written so you can reproduce it.

1. Size the hardware before choosing the model

The constraint is almost always VRAM, and the arithmetic is simple. A model quantised to 4 bits needs roughly half a gigabyte of VRAM per billion parameters, plus a few gigabytes for the context window and the runtime. Work backwards from the card you have:

CPU-only inference works and costs nothing extra, but expect single-digit tokens per second on a mid-range server. That is acceptable for a nightly batch job. It is not acceptable behind a user typing in a form.

2. Serve it behind an OpenAI-compatible endpoint

Pick a runtime that exposes the OpenAI HTTP schema. That single decision is what keeps you free: every client library already speaks it, and swapping the model, the runtime, or later a hosted provider becomes a change of two environment variables rather than a rewrite.

Option A — Ollama (simplest to operate)

# install and pull a model
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:14b-instruct-q4_K_M

# bind to the loopback interface only; put a reverse proxy in front for anything else
export OLLAMA_HOST=127.0.0.1:11434
export OLLAMA_KEEP_ALIVE=30m      # keep the model resident between requests
export OLLAMA_NUM_PARALLEL=2      # concurrent requests; raise only if VRAM allows
ollama serve

The OpenAI-compatible endpoint is then http://127.0.0.1:11434/v1.

Option B — llama.cpp server (most control)

./llama-server \
  --model ./models/qwen2.5-14b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 8192 \
  --n-gpu-layers 999 \
  --parallel 2 \
  --api-key "$LOCAL_LLM_API_KEY"

--n-gpu-layers 999 offloads every layer it can fit; lower it if the process is killed on start. --ctx-size is the setting people over-provision most: a 32k context reserves memory you are probably not using, and it is the first thing to cut when the model will not load.

Wiring it into an application

# .env — never commit this file
LLM_BASE_URL=http://127.0.0.1:11434/v1
LLM_API_KEY=local-does-not-matter
LLM_MODEL=qwen2.5:14b-instruct-q4_K_M
LLM_TIMEOUT_MS=120000

Any OpenAI client library takes base_url and api_key directly. If your code reads those four variables and nothing else, you can move between local and hosted inference without touching the application.

3. The failure modes nobody documents

• The model is evicted between requests. First call takes forty seconds, the next takes two, the one after takes forty again. Fix the keep-alive setting rather than the prompt.
• Context length silently truncates. Exceed the configured window and the runtime drops the beginning of the conversation — including your system prompt. Measure token counts; do not assume.
• Quantisation degrades structured output first. Prose survives aggressive quantisation far better than strict JSON. If a 4-bit model produces malformed JSON, try a larger quantisation before concluding the model is unsuitable.
• Concurrency multiplies memory, not just load. Each parallel slot reserves its own context. Two slots at 8k cost roughly what one slot at 16k costs.
• The endpoint is open to the network. Binding to 0.0.0.0 without a proxy exposes an unauthenticated inference server. Bind to loopback, terminate TLS at a reverse proxy, and require a key.

4. What to measure once it runs

Three numbers tell you whether the deployment is healthy, and all three are cheap to log: time to first token, tokens per second, and the share of requests that hit the context ceiling. Log them per request from day one. Retrofitting measurement after the complaints start is how teams end up guessing.

5. When local is the wrong answer

Local inference wins on data residency, on predictable cost at steady volume, and on latency once the model is resident. It loses on peak quality, on multimodal work, and on anything where demand is spiky enough that the hardware sits idle most of the week. I say so when that is the case — a blueprint you cannot afford to run is not a solution.