# llama.cpp, vLLM or Ollama: which inference engine for a home server?

> The advice circulating is written from datacenter GPU benchmarks and rarely survives contact with a single box. What each engine is actually for, which one we run where and why, and the defaults question that matters more than the choice itself.

- Published: 2026-08-21
- Evidence level: lab_single_run
- Funding: Self-funded internal research
- Canonical: https://agmind.ai/reports/which-inference-engine-local-llm/

On a single box serving a handful of people, the engine is rarely your bottleneck, and the honest reason to switch is a change in
workload shape rather than a benchmark table. We serve llama.cpp on our
Strix Halo machines and vLLM on the DGX Spark pair, and the deciding factor
in both cases was what the workload needed, not which engine is "faster" in
the abstract.

A warning about the genre first. Most engine comparisons you will find were
run on datacenter GPUs, quote throughput multipliers from batch sizes a home
server never sees, and omit the model, quant and concurrency that produced
them. We have no cross-engine numbers on identical hardware either — and
rather than borrow someone else's, this page says what we know from
operating both.

## What each one is for

**llama.cpp** runs quantized GGUF weights on almost anything, including
integrated GPUs through Vulkan, which is exactly why it is our serving
runtime on unified-memory boxes. It is a server plus a set of backends, it
starts in seconds, and it is honest about what it is doing if you ask it —
the `/props` endpoint will tell you the effective configuration it resolved,
which turns out to matter more than most people expect.

**vLLM** is built for serving at scale: continuous batching, paged attention
memory management, tensor parallelism across devices. On our Spark pair,
tensor parallelism is not a nice-to-have — it is the only reason a
284B-parameter model runs at all, and the
[serving recipe](https://agmind.ai/reports/deepseek-v4-flash-0731-dgx-spark-recipe/) exists
because getting there involved real operational work. In exchange for the
capability, you get a heavier stack: longer startup, model-format
constraints, and kernel-selection behavior that
[can silently pick the slower path](https://agmind.ai/reports/dsv4-flash-moe-backend-dgx-spark/).

**Ollama** is a distribution and model-management layer that runs llama.cpp
underneath. "Ollama versus llama.cpp" is therefore usually not an engine
question at all — it is a question about which defaults you inherit and
whether you can see them.

## The question that matters more than the engine

Whatever you run, know what it resolved. The most expensive measurement
mistake in our own journal came from a runtime default for parallel slots
that changed between versions and quietly divided per-stream throughput —
days of tuning theories died to one query against the running server. The
[full story is in an essay](https://agmind.ai/essays/luddites-were-right/), and the general
form of the lesson is
[here](https://agmind.ai/essays/benchmark-number-config-in-disguise/): an unset flag is still configuration.

This is where wrappers cost you. A layer that manages models for you also
chooses context length, slot count, offload behavior and sampling defaults
on your behalf, and if you cannot inspect those choices you cannot explain
your own numbers — or fix them.

## When llama.cpp is enough

Everything we measure on a single 128 GB box runs on it: solo assistants,
[small-team concurrency](https://agmind.ai/reports/concurrency-capacity-strix-halo/),
[document sessions](https://agmind.ai/reports/llamacpp-prompt-cache-strix-halo/),
[strict-JSON automation](https://agmind.ai/reports/local-llm-json-reliability/), and
[three-hour sustained load](https://agmind.ai/reports/strix-halo-sustained-load-3-hours/).
On this class of hardware the practical ceiling is memory bandwidth, and no
engine argues with physics: the choices that moved our numbers most were
[quantization](https://agmind.ai/reports/q8-0-vs-q4-k-m-strix-halo/) and
[backend](https://agmind.ai/compare/llamacpp-vulkan-vs-rocm-strix-halo/), both inside
llama.cpp.

## When the workload asks for vLLM

Three signals, from our own operating experience rather than a benchmark:

**The model does not fit in one device.** Tensor parallelism across nodes is
the capability, and it is not optional for the frontier open-weights class.

**You are serving a real queue, not a few chats.** Continuous batching earns
its complexity when requests genuinely overlap all day.

**You need the vendor's fast path for a specific model.** Some kernels ship
only in this ecosystem — and, as we found, sometimes only if you ask for
them explicitly.

If none of those describe your box, the heavier stack buys you operational
surface rather than speed.

## What we have not measured

No same-hardware, same-model, same-workload comparison of these engines
exists in our archive, so this page contains no cross-engine numbers. Our
two engines run on two different hardware classes for two different reasons,
and putting their figures in adjacent columns would manufacture exactly the
kind of folklore we spend our time dismantling. If the comparison lands
later, it will land with raw runs.

Meanwhile, the protocol for settling it on your own box is
[written up here](https://agmind.ai/reports/how-to-benchmark-local-llm/) — freeze the
workload, pin both stacks, ask each server what it actually resolved, and
keep failures in the denominator. That measurement is worth more to you than
any general answer, including this one.

---

Machine-readable claim registry: https://agmind.ai/claims.json · llms.txt: https://agmind.ai/llms.txt