What is llm-d?

llm-d is a Kubernetes-native, open source framework that speeds up distributed large language model (LLM) inference at scale. 

This means when an AI model receives complicated queries with a lot of data, llm-d provides a framework that makes processing faster. 

llm-d was created by Google, NVIDIA, IBM Research, and CoreWeave. Its open source community contributes updates to improve the technology.

How Red Hat AI speeds up inference

LLM prompts can be complex and nonuniform. They typically require extensive computational resources and storage to process large amounts of data. 

llm-d has a modular architecture that can support the increasing resource demands of sophisticated and larger reasoning models like LLMs. 

A modular architecture allows all the different parts of the AI workload to work either together or separately, depending on the model's needs. This helps the model inference faster.

Imagine llm-d is like a marathon race: Each runner is in control of their own pace. You may cross the finish line at a different time than others, but everyone finishes when they’re ready. If everyone had to cross the finish line at the same time, you’d be tied to various unique needs of other runners, like endurance, water breaks, or time spent training. That would make things complicated. 

A modular architecture lets pieces of the inference process work at their own pace to reach the best result as quickly as possible. It makes it easier to fix or update specific processes independently, too.

This specific way of processing models allows llm-d to handle the demands of LLM inference at scale. It also empowers users to go beyond single-server deployments and use generative AI (gen AI) inference across the enterprise.

How does distributed inference work?  

The llm-d modular architecture is made up of: 

• Kubernetes: an open source container-orchestration platform that automates many of the manual processes involved in deploying, managing, and scaling containerized applications.
• vLLM: an open source inference server that speeds up the outputs of gen AI applications.
• Inference Gateway (IGW): a Kubernetes Gateway API extension that hosts features like model routing, serving priority, and “smart” load-balancing capabilities. 

This accessible, modular architecture makes llm-d an ideal platform for distributed LLM inference at scale.

What is operationalized AI?