How llm-d makes the most of the hardware you already have

As AI agents become more capable, they are placing new demands on inference infrastructure. Unlike traditional chatbots, agentic systems like coding assistants repeatedly process massive contexts, reuse information across many interactions, and recruit parallel sub-agents that generate unpredictable bursts of activity. These workloads are dominated by reading and managing context rather than generating text, creating challenges for latency, memory, throughput, and cost.

With the llm-d project, a collaborative effort led by IBM Research, Red Hat, and Google, and joined by other industry leaders, an open-source framework has been introduced to help serve LLMs at scale, addressing the challenges introduced by these new types of AI workloads. llm-d is filling a need that has arisen in the enterprise AI landscape: Amid rising per-token costs and the need to secure proprietary data, enterprises are increasingly looking for ways to deploy open models on their own infrastructure. Now, the…