llm-d 1.0: Red Hat, IBM, Google and Nvidia bet that AI’s real bill is inference, not training

For three years the AI conversation has been about who can build the biggest model. The less glamorous question — what it costs to keep that model answering, millions of times a day, without falling over — is where the money now leaks. llm-d, the open-source inference project started by Red Hat, IBM, Google Cloud, CoreWeave and Nvidia, has shipped its first stable release, and its pitch is aimed squarely at that leak.

The mechanism is worth stating plainly. An AI agent does not ask one question and leave. It keeps long sessions alive, rereads the same documents and spawns parallel helpers that hit the hardware in unpredictable bursts. In that pattern, handling the context can cost more than producing the answer. Every team serving models at scale, says Shadi Saba, CoreWeave’s vice president of engineering for AI/ML, runs into “context that often costs more to manage than the output”. llm-d’s answer is routing: send each request to the server that…