vLLM vs. Ollama: When to use each framework

When integrating large language models (LLMs) into an AI application, vLLM works well for high-performance production, and Ollama is great for local development. Each tool caters to different ends of the LLM deployment spectrum: vLLM is ideal in enterprise settings, whereas Ollama works best for small-scale projects. 

When considering the differences between vLLM and Ollama, think of Ollama as a sports car and vLLM as a bullet train. With Ollama, you can move at a fast pace, but you can’t bring many people with you. With vLLM, you can both move fast and serve a lot of people at once. 

Ultimately, the choice between vLLM and Ollama depends on your expertise as a developer as well as the size and scope of your project. For developers experimenting locally, Ollama is a fantastic starting point. But for teams moving toward large-scale production, vLLM provides the foundation needed to reliably and efficiently serve LLMs at scale.

Ollama or vLLM? How to choose the right serving tool 

vLLM is a library of open source code that helps LLMs quickly and efficiently perform calculations at scale. The overall goal of vLLM is to maximize throughput (tokens processed per second) to serve many users at once. 

vLLM includes both an inference server (to manage network traffic) and an inference engine (to boost computational speed): 

• The inference server component manages the queue of users waiting for service and handles external network traffic. The inference server’s job isn't to perform intensive computation; it's to handle the communication protocol that gets data in and out of the system.
• The inference engine component speeds up generation by optimizing graphics processing unit (GPU) usage. It increases computational speed by applying algorithms like PagedAttention to manage the key value (KV) cache and continuous batching for optimized request scheduling.

Both the inference server and the inference engine are responsible for achieving low latency—or the time between a user hitting “send” and an output being delivered. The inference server is designed to not add unnecessary latency. It does this by accepting requests, passing them to the engine, and sending responses back across the network as quickly as possible. The inference engine is responsible for actively removing latency by organizing the GPU’s computations. By speeding up processing in this way, vLLM can serve hundreds of users simultaneously in 1 instance. 

vLLM allows organizations to do more with less in a market where the hardware for LLM-based applications comes with a hefty price tag. It can handle high traffic and is designed for large-scale scenarios. This means it’s ideal for latency-sensitive, multiuser deployments. Overall, vLLM outperforms Ollama when serving multiple requests simultaneously. 

Read more about vLLM

Ollama is an open source tool that lets users run LLMs locally and privately. This means you can download, update, and manage an LLM from your laptop without sending any private information to a cloud. 

Ollama is derived from the llama.cpp project, an open source library that performs inference on various LLMs. Ollama automates some of the more difficult steps involved in compiling, configuring, and managing the underlying components, thereby hiding these complexities from the end user. 

Built for simplicity, Ollama requires minimal setup and is generally considered to be intuitive and good for beginners. Experienced developers may use Ollama for experimenting with different LLMs and rapid prototyping.

vLLM and Ollama are both LLM serving frameworks that provide developers access to LLMs they can use to build AI applications. Specifically, an LLM serving framework is a software component that performs inference within the larger application architecture. 

While vLLM and Ollama cater to different types of users, they share several fundamental features:

• Both are open source tools.
• Both include an inference server component.
• Both let users run LLMs on their own hardware rather than relying on third-party APIs.
• Both are designed to make the most of users’ available hardware to accelerate inference speed.
• Both support multimodal models, meaning they can process more than just text.
• Both support retrieval-augmented generation (RAG), a technique that allows developers to supplement the existing data in an LLM with external knowledge/data of their choosing. 

The advanced features vLLM offers require deeper technical understanding. Because of this, vLLM caters more to seasoned developers and arguably has a steeper learning curve than Ollama. 

For example, vLLM can handle models of any size, including those with billions of parameters. To get the most out of the technology, developers should understand how to carry out concepts like distributed inference. 

It also has a lot of potential for fine-tuning for specific use cases. To get the best performance, developers should be familiar with methods like parameter-efficient fine-tuning (PEFT) and LoRA/QLoRA.

In summary, vLLM is for developers who need to squeeze every drop of performance potential from their servers and deploy reliable, scalable applications. Remember our bullet train analogy? vLLM is good at serving a lot of users in a short period of time. 

Ollama lets users download and run LLMs on personal computers and provides a simple way to test AI models. However, the primary goal of Ollama is accessibility, not scalability. If users make additional requests, they must wait in a queue. For this reason, developers choose Ollama when they want convenience and don’t need to serve a lot of users. 

Ollama operates offline by default. This means that once you’ve downloaded a model, it can function without an internet connection. While vLLM also offers maximum privacy, it requires setting up a private server or configuring a secured cloud environment. This extra step requires more expertise. 

Both vLLM and Ollama have an inference server component. This means both vLLM and Ollama take incoming requests, unpack the data, send it to the engine, and package the response to send back to the end users’ application. 

However, vLLM is an inference engine, and Ollama is not. This means vLLM can optimize inference in ways Ollama cannot. As an inference engine, vLLM is better at managing memory and handling multiple users at the same time (concurrency):

Managing memory: vLLM uses the PagedAttention algorithm to manipulate the structure of the GPU’s memory. It frees up space on the GPU, which creates potential to run more requests at once. This process enables high concurrency. 

Concurrency: vLLM uses continuous batching to manage data flow and make the best use of the GPU, so it can handle many users/requests at once. 

Compare performance for Ollama and vLLM

Red Hat® AI uses open source innovation to meet the challenges of wide-scale enterprise AI—and vLLM is a critical tool in our toolbox.

With Red Hat AI, you have access to Red Hat AI Inference Server to optimize model inference across the hybrid cloud for faster, cost-effective deployments. Powered by vLLM, the inference server makes the GPU run efficiently and supports faster response times.

Red Hat AI Inference Server includes the Red Hat AI repository, a collection of third-party validated and optimized models that allows model flexibility and encourages cross-team consistency. With access to the third-party model repository, enterprises can accelerate time to market and decrease financial barriers to AI success.