vLLM: High-Throughput Inference for Serving Many Users
vLLM solves a different problem from Ollama and llama.cpp. Those optimise for running a model on modest hardware. vLLM optimises for serving many requests at once from a GPU that could otherwise sit underused.
PagedAttention
The technique it is known for borrows from operating system memory management.
Naive inference allocates a contiguous block of GPU memory for each request’s key-value cache, sized for the longest possible output. Most requests are shorter, so most of that memory is wasted, and the waste caps how many requests fit concurrently.
PagedAttention allocates the cache in pages, as a kernel manages virtual memory. Memory is used as needed rather than reserved for a worst case, so far more requests fit on the same GPU. The throughput difference against a naive server is large.
Continuous batching
vLLM also batches continuously rather than in fixed groups. A finished request leaves the batch and a waiting one joins immediately, instead of the whole batch waiting for its slowest member.
For a server handling a stream of requests of varying length, which is what real usage looks like, this matters a great deal.
Running it
vllm serve meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000It exposes an OpenAI-compatible API, so anything that speaks that protocol works against it, including LibreChat and Open WebUI.
The hardware reality
vLLM wants a GPU with substantial VRAM. It does not target CPU inference and there is no useful quantized-on-a-laptop story here.
That is the honest dividing line. One person on a desktop should use Ollama. A team sharing one GPU server, or an application making many concurrent calls, should use vLLM.
License
vLLM is released under the Apache License 2.0.