On a single box serving a handful of people, the engine is rarely your bottleneck, and the honest reason to switch is a change in workload shape rather than a benchmark table. We serve llama.cpp on our Strix Halo machines and vLLM on the DGX Spark pair, and the deciding factor in both cases was what the workload needed, not which engine is “faster” in the abstract.
A warning about the genre first. Most engine comparisons you will find were run on datacenter GPUs, quote throughput multipliers from batch sizes a home server never sees, and omit the model, quant and concurrency that produced them. We have no cross-engine numbers on identical hardware either — and rather than borrow someone else’s, this page says what we know from operating both.
What each one is for
llama.cpp runs quantized GGUF weights on almost anything, including
integrated GPUs through Vulkan, which is exactly why it is our serving
runtime on unified-memory boxes. It is a server plus a set of backends, it
starts in seconds, and it is honest about what it is doing if you ask it —
the /props endpoint will tell you the effective configuration it resolved,
which turns out to matter more than most people expect.
vLLM is built for serving at scale: continuous batching, paged attention memory management, tensor parallelism across devices. On our Spark pair, tensor parallelism is not a nice-to-have — it is the only reason a 284B-parameter model runs at all, and the serving recipe exists because getting there involved real operational work. In exchange for the capability, you get a heavier stack: longer startup, model-format constraints, and kernel-selection behavior that can silently pick the slower path.
Ollama is a distribution and model-management layer that runs llama.cpp underneath. “Ollama versus llama.cpp” is therefore usually not an engine question at all — it is a question about which defaults you inherit and whether you can see them.
The question that matters more than the engine
Whatever you run, know what it resolved. The most expensive measurement mistake in our own journal came from a runtime default for parallel slots that changed between versions and quietly divided per-stream throughput — days of tuning theories died to one query against the running server. The full story is in an essay, and the general form of the lesson is here: an unset flag is still configuration.
This is where wrappers cost you. A layer that manages models for you also chooses context length, slot count, offload behavior and sampling defaults on your behalf, and if you cannot inspect those choices you cannot explain your own numbers — or fix them.
When llama.cpp is enough
Everything we measure on a single 128 GB box runs on it: solo assistants, small-team concurrency, document sessions, strict-JSON automation, and three-hour sustained load. On this class of hardware the practical ceiling is memory bandwidth, and no engine argues with physics: the choices that moved our numbers most were quantization and backend, both inside llama.cpp.
When the workload asks for vLLM
Three signals, from our own operating experience rather than a benchmark:
The model does not fit in one device. Tensor parallelism across nodes is the capability, and it is not optional for the frontier open-weights class.
You are serving a real queue, not a few chats. Continuous batching earns its complexity when requests genuinely overlap all day.
You need the vendor’s fast path for a specific model. Some kernels ship only in this ecosystem — and, as we found, sometimes only if you ask for them explicitly.
If none of those describe your box, the heavier stack buys you operational surface rather than speed.
What we have not measured
No same-hardware, same-model, same-workload comparison of these engines exists in our archive, so this page contains no cross-engine numbers. Our two engines run on two different hardware classes for two different reasons, and putting their figures in adjacent columns would manufacture exactly the kind of folklore we spend our time dismantling. If the comparison lands later, it will land with raw runs.
Meanwhile, the protocol for settling it on your own box is written up here — freeze the workload, pin both stacks, ask each server what it actually resolved, and keep failures in the denominator. That measurement is worth more to you than any general answer, including this one.