Ollama vs vLLM vs llama.cpp: which serving engine should run your model

Here is the number that surprises most teams the first time they benchmark it for themselves: at one concurrent user, Ollama answers faster than vLLM. Time to first token, roughly 45 milliseconds against 82. The engine everyone files under “toy” beats the engine everyone files under “grown-up,” as long as only one person is talking to it at a time.

That single fact is where a lot of serving-engine decisions quietly go wrong. The story we tell about Ollama and vLLM is a story about seniority, and a GPU does not care about seniority. It cares about how many requests arrive at the same moment.

The myth that you graduate from Ollama to vLLM

There is a belief, and I have repeated a softer version of it myself, that Ollama is where a project starts and vLLM is where it goes once things get serious. Prototype on Ollama, then earn your way up to vLLM when you have real traffic. It gets phrased as maturity. As if the simpler engine is training wheels and the harder engine is what adults ship.

The framing is comfortable because it turns a technical decision into a status decision, and status decisions are easy. Pick the thing that sounds more production-grade and nobody in the architecture review will question it.

The problem is that the ladder does not exist. Ollama and vLLM solve two different shapes of problem, and neither one is a junior version of the other. One is tuned for a single stream and a fast start. The other is tuned for many streams sharing a GPU. Deciding between them by seniority is like choosing a vehicle by how impressive it looks in the parking lot instead of by what you need to haul.

Why the throughput charts make vLLM look like the only adult in the room

The reason the myth sticks is that the charts people share are real. They are just measured at one end of the range. Almost every comparison that goes viral is run at high concurrency, where vLLM is built to win, and the gap looks enormous.

As Particula Tech put it in a March benchmark writeup: “vLLM achieved 8,033 tokens per second compared to Ollama’s 484, a 16.6x throughput advantage.” That is not a marketing number, it is a fair result for the workload it describes: a batch of concurrent requests on a modern GPU, exactly the case continuous batching was designed for.

The architecture explains the whole thing. vLLM uses continuous batching and PagedAttention, so it packs many active sequences into shared GPU work and pages memory in and out as requests come and go. Ollama, which sits on top of llama.cpp, processes requests through a simpler queue and allocates memory statically per model. At one request that simplicity is an advantage. At a hundred requests it becomes the bottleneck. Same GPU, completely different behavior, entirely because of how each engine schedules work.

Where Ollama wins and where vLLM pulls ahead

The honest version of the vLLM vs Ollama comparison is not a ranking, it is a crossover. The two lines start close, touch, and then separate as load climbs.

Read the table top to bottom and the decision writes itself. If the model serves one person at a time, an internal tool a handful of colleagues poke at, a batch job that runs one prompt after another, Ollama is not the compromise. It is the correct answer, and it will have the team serving a model in about five minutes with model management built in.

Cross into sustained concurrency and the picture flips hard. By eight to ten simultaneous requests vLLM is already two to three times ahead on total throughput, and at fifty users Ollama flattens out near 155 tokens per second while vLLM holds close to 920 with far better tail latency. A peer-reviewed evaluation in Applied Sciences this spring found the same pattern across scenarios: the engines are close until concurrency rises, and then continuous batching decides the outcome. When independent benchmarks and an academic study agree on the direction, that is about as settled as this field gets.

SGLang and llama.cpp: the two engines the vs-posts skip

Framing the whole question as Ollama against vLLM also hides the other two names that belong in the room.

SGLang is not a step above vLLM, it is a sibling for a specific workload. On standard traffic it runs modestly ahead, roughly 16,200 tokens per second against vLLM’s 12,500 on an H100 in one 2026 comparison, thanks to prefix reuse. That edge nearly vanishes on unique-prompt batch jobs and can balloon on prefix-heavy work like retrieval pipelines and multi-step agents that share a long system prompt. SGLang earns its place when a team is already at vLLM scale and the workload reuses prefixes. Below that, it is complexity without a payoff.

llama.cpp is the layer underneath Ollama, which means the “vLLM vs llama.cpp” debate is mostly a hardware question wearing a software costume. Datacenter GPU with steady concurrency points to vLLM. CPU, Apple Silicon, a mixed or edge box points to the llama.cpp family, and Ollama is the friendly way to run it. That is a sizing decision about the metal on hand, not a verdict on which project is more serious.

The real trigger to switch is a concurrency number

Here is the reframe worth carrying into the next planning meeting. The signal to move off Ollama is not that the product got important. It is one of three measurable conditions: sustained concurrency above roughly four or five simultaneous requests, a rig with more than one GPU to fill, or a hard tail-latency promise you have to keep. Hit one of those and vLLM’s batching stops being overkill and starts being the reason the service stays responsive. Miss all three and the simpler engine is the right engineering call, not a placeholder.

The serving engine is not a seniority ladder. It is a match to how many people hit the model at the same instant.

Notice that every trigger is a number, not a feeling. A counter on concurrent requests is easy to add, and a tail-latency chart is easy to read. That turns a debate that usually runs on vibes into a decision an infra lead can defend in one sentence.

Ollama and llama.cpp are the same engine, so the speed argument settles little

The third engine in this decision is llama.cpp, and it deserves more than the footnote the head-to-head posts give it, because of one fact that reframes the whole comparison: Ollama runs llama.cpp inside itself. Ollama is a server, a web API, and a model registry wrapped around the same inference kernels. An inference kernel is the low-level routine that does the actual arithmetic of generating a token. At steady state the two are executing identical code on identical weights.

That is why the benchmark everyone reaches for decides so little. A July 6 benchmark from Markaicode ran both on an RTX 4090 and put llama.cpp at roughly 104 tokens per second against Ollama at 95 to 110, on the same model at the same quantization level, with time to first token under 200 milliseconds for both. Quantization is the practice of storing a model's weights at reduced numeric precision so it fits in less memory. The structural gap is 3 to 10 percent on NVIDIA hardware, and the cause is the wrapper, not the math.

A 3 to 10 percent gap is real and it almost never decides anything. If the margin between self-hosting working and not working is 7 percent of throughput, the deployment was undersized before the engine question came up. Set that against the concurrency gap with vLLM described above, which is measured in multiples rather than percent. That is the difference between a preference and a decision.

Where these two actually diverge: defaults and release cadence

Both projects ship a 4096-token context window by default. The context window is how much text the model can consider at once, counted in tokens, which are roughly word fragments. That default is fine for a first test and quietly ruinous for an agent workload that puts twenty thousand tokens of tool definitions and history into every request and then appears to forget its own instructions. In llama.cpp the fix is a visible startup flag. In Ollama it is an environment variable that many teams find only after truncation has been silently eating context for a month.

The sharper difference is cadence. Across 48 hours on July 16 and 17, Ollama shipped exactly one release, version 0.32.1, carrying improved Gemma 4 tool calling and a fix for a memory leak in its Apple Silicon model cache. In the same two days llama.cpp shipped thirteen build-tagged releases, covering virtual devices on NVIDIA hardware, saving and restoring server state, tensor-parallel fixes, and Vulkan synchronization. Same engine underneath, two different ideas of what shipping means.

Neither cadence is better. They are different bills. llama.cpp does not use semantic version numbers; it tags a build for nearly every merged change, so staying current means owning a version policy: pin a build, test against it, move deliberately. Ollama curates, which means less to track and less control over when a fix arrives. The Qwen 3.5 tool-call parser landed in Ollama roughly five months after that model's release, and until it did, tool calls were routed through the wrong template.

The convenience gap has narrowed from the llama.cpp side too. It now ships prebuilt binaries, a maintained container image, and a router mode that its maintainers described as bringing Ollama-style model management to llama.cpp: automatic discovery of downloaded model files, loading on demand, eviction of the least recently used model once a limit is passed, and each model in its own process so one crash does not take the others down.

This is the same reasoning that applies when a multi-agent workflow routes steps across more than one provider. The engine that serves a single interactive session well is rarely the one that serves a burst of parallel agents, and the honest answer is usually to run both and route by load rather than to crown a winner. Cerevisor's provider guide covers pointing local and hosted engines at different steps of the same workflow.

Pick by the load, and expect to run both engines

The tidy takeaway is that most teams do not choose one engine, they end up running two: Ollama for local development and low-concurrency internal tools, vLLM for the multi-user production path. That is not indecision, it is matching the tool to the load in each place. The average “we’ll just standardize on vLLM everywhere” plan has a slide for throughput and no slide for the developer who now waits on a heavier server to test a one-line change.

Nothing in the market this week forces the decision either way, which is a small gift. No open-weight model or serving engine shipped a release inside the last few days, and the GPU rental side barely moved: GetDeploying’s tracker put the average H100 around 3.37 dollars an hour across 47 providers on July 5, up about 4 percent on the year. When neither the models nor the hardware are lurching, the serving choice stops being a reaction and becomes what it should have been all along, a calm read of how many requests actually arrive at once.

So measure the concurrency you really have, not the concurrency the pitch deck imagines. Then let that number pick the engine, and let the other engine keep doing the job it is better at.